texts: Hierarchical Clustering: Extract observations from large heatmap

texts:
I'm currently trying to visualize a large data set as heat map. That in itself works smoothly but I struggle with gaining insights from interestingly looking clusters.

Specifically, I have two questions that are very related:

First, I find clusters of interesting features and am looking for a systematic way to extract the a flat clustering at a specific level (but the fcluster function seems to do something different and cut_tree doesn't work with those trees). I would like to have a slice of the hierarchical clustering at a specified depth of the dendrogram. This is probably encoded in the linkage matrix Z but I struggle to understand how exactly I can extract that information from Z.

Second, with the complicated heatmap pictured below, the row names on the right are gene names for every 100th data point (gene). I would now like to have a look at which genes are in some of the little clusters, for instance the little black square for feature MF: LIHC that is marked.

I know the IDs of the genes that are labelled on the right, so I would want to know something like:

Which genes are in the same cluster as CAPN7 at level 5?

Thank you for your help!

Roman


texts: LSTM classification with different sizes

texts:
I'm relatively new to the world of recurrent neural nets and I'm trying to build a classifier using an LSTM model to predict HIV activity from a given molecule (the original dataset can be found here ).
I have sequences of different lengths (from few dozen to almost 400 characters) but I'm not sure how to proceed. Let's say that I have a dataset structured like so:
import pandas as pd
import random
import string

random.seed(42)
seqs = [[random.choice(string.ascii_letters) for i in range(random.randint(1,10))] for i in range(5)]
classes = [random.randint(0,1) for i in range(5)]
df = pd.DataFrame({
    "seq": seqs,
    "class": classes
})
print(df)

    seq                             class
0   [b, V]                          0
1   [p, o, i, V, g]                 0
2   [f, L, B, c, b, f, n, o, G]     1
3   [b, J, m, T, P, S, I, A, o, C]  0
4   [r, Z, a, W, Z, k, S, B, v, r]  0

I know I should:

one hot encode the elements,
perform masking and padding

But I don't know how to perform it in Keras/TF2 and I can't find any resources online that explain how to code something similar.


texts:
I'm using neural network machine learning and would like to see the result of my confusion matrix for my model. However, there is an error that I've got and don't know how to solve it.
from sklearn.metrics import confusion_matrix
cm=confusion_matrix(testY, testPredict)
print(cm)

then it give an error stated: Found input variables with inconsistent numbers of samples: [30, 24]
actually I've check the shape of the test and the prediction value and it shows the different shape. How can I make the shape of the test and prediction become same ?
the


texts: Implementation of Stochastic Gradient Descent in Python

texts:
I am attempting to implement a basic Stochastic Gradient Descent algorithm for a 2-d linear regression in Python. I was given some boilerplate code for vanilla GD, and I have attempted to convert it to work for SGD.
Specifically -- I am a little unsure as to if I correctly implemented the loss function and partial derivatives, since I am new to regressions in general.
I do see that the errors tend to "zig zag" as expected. Does the following look like a correct implementation or have I made any mistakes?
    #sample data
    data  = [(1,1),(2,3),(4,3),(3,2),(5,5)]

    def compute_error_for_line_given_points(b, m, points):
        totalError = 0
        x = points[0]
        y = points[1]
        return float(totalError + (y - (m * x + b)) ** 2)

    def step_gradient(b_current, m_current, points, learningRate):
        N = float(1)
        for i in range(0, 1):
            x = points[0]
            y = points[1]
            b_gradient = -(2/N) * (y - ((m_current * x) + b_current)) #this is the part I am unsure
            m_gradient = -(2/N) * x * (y - ((m_current * x) + b_current)) #here as well
        new_b = b_current - (learningRate * b_gradient)
        new_m = m_current - (learningRate * m_gradient)
        return [new_b, new_m]

    err_log = []
    coef_log = []
    b = 0 #initial intercept
    m = 0 #initial slope

    iterations = 4
    for i in range(iterations): #epochs
        for point in data: #one point at a time for SGD
            err = compute_error_for_line_given_points(b,m, point)
            err_log.append(err)
            b,m = step_gradient(b,m,point,.01)
            coef_log.append((b,m))
```


texts: Use test data as train: does it make sense?

texts:
There is a classification problem(two classes). We have train data, for which we know class labels and we have test data.

Imagine, that you have created model that with good accuracy(~95%) make predictions and we know that we are not overfitted.

If we make prediction on test data, extract objects for which we sure in class label(for example, predict_proba higher than 90%) and add this objects to train data.

Does this tactic make any sense?


texts: What is better opensource alternative for identifying small face other than yolo?

texts:
I was trying to identify small face meaning that I want to know who that face belong to according to training dataset. I have previously use yolov4 to detect small object before and I know the accuracy is not very satisfying and I have read in internet that yolo is very terrible at detecting faces. Do anyone know what is the alternative opensource that is better than yolo v4 for identifying small face? Please share your knowledge. Thanks.
Edit 1: I mean like for identifying this face is for example angelina jolie face,etc(face recognition/ identification) not face detection(knowing existence of face)


texts: Learning Algorithm that decide which model gives better results for each testing instance

texts:
Is their any existing Ensemble technique which uses subset of training data to predict which algorithm is better for predicting each instance of testing data?

Let's say we have N sized training set and K sized testing set in which a particular attribute needed to be predicted using the training set. But there are hundreds of algorithms and ways we can use. We can divide training set into two parts and train each model with first half and decide test on second half. Based on characteristics, we can decide which algorithm to use for real test cases (K sized set). As an example lets say dataset have an attribute named "temperature". Particular algorithm may work well when temperature is higher than 100 Celsius. We can then classify all the 100 degree or above instances to particular class. Then final prediction will be done based on that by with that model class trained with all N sized data.

What I am asking is that is their any existing method similar to that?


texts:
I'm trying to train an LSTM for sentiment analysis on the IMDb review dataset.

As input to the word embedding layer, I transform each review to a list of indices (that corresponds to word index in the vocabulary set). I thought of converting the text into one-hot/count matrix, but I will end up with huge sparse matrix (should I worry about this?).

Here is how I am creating the network architecture:

model = Sequential()
model.add(Embedding(
    input_dim=vocab_size,
    output_dim=word_embed_vector_size,
    input_length=sentence_len_max)
         )
model.add(LSTM(units=1))
model.add(Dense(1, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', 'binary_accuracy'])
model.summary()

Here is the model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_2 (Embedding)      (None, 1422, 4)           201764
_________________________________________________________________
lstm_2 (LSTM)                (None, 1)                 24
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 2
=================================================================
Total params: 201,790
Trainable params: 201,790
Non-trainable params: 0
___________________________

Now when I try to train the model I see accuracy stuck at 50%

losses = model.fit(
    x                = term_idx_train,
    y                = y_train,
    epochs           = epochs,
    batch_size       = batch_size,
    validation_split = 0.01
)

Here is the epochs output:

Epoch 1/10
25000/25000 [==============================] - 1148s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 2/10
25000/25000 [==============================] - 1156s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 3/10
25000/25000 [==============================] - 1149s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 4/10
25000/25000 [==============================] - 1110s 44ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 5/10
16800/25000 [===================>..........] - ETA: 6:10 - loss: 7.9816 - acc: 0.4993 - binary_accuracy: 0.4993

Changing the activation function to a sigmoid and the LSTM blocks to 32 didn't help mush (with 1 epoch):

Train on 24750 samples, validate on 250 samples
Epoch 1/1
24750/24750 [==============================] - 1186s 48ms/step - loss: 0.6932 - acc: 0.5022 - binary_accuracy: 0.5022 - val_loss: 0.6951 - val_acc: 0.0000e+00 - val_binary_accuracy: 0.0000e+00

Epoch 00001: val_loss improved from inf to 0.69513, saving model to sentiment_model

Looking at what the LSTM is predicting, I see:

count   25000.000000
mean    0.499023
std 0.000013
min 0.499010
25% 0.499010
50% 0.499010
75% 0.499010
max 0.499443

Any idea why it's doing this? and how I could fix the issue?


texts: How to use the position factor in known data as a feature in recommendation surfacing?

texts:
The problem is recommending stories on a website, just below each story based on how similar the stories are and some historic data based on what recommended stories were clicked or not clicked. So basically the available data looks like
Referrer    Recommendation  Timestamp                Position   Clicked
url1          url2          2020-07-01 05:11:17        3           0
url3          url4          2020-07-02 15:11:17        5           1
For example, a user was reading url1 at the given timestamp when after reading he saw 12 urls as recommendation. Among them, url2 appeared at the third place, and the user did not click it. (That row in itself does not tell us anything about whether the user clicked any of the other 11 stories that appeared as recommendation).
The second row says a user was similarly reading story url3 and after reading he clicked url4, which appeared at the 5th position.
The problem is, I have to develop a model based on this information which will receive

A url which the user is reading now
200  other urls as potential recommendation candidates.

I have to assign some sort of score to rank them based on the propensity to click.
The URLs, timestamps have already been embedded into features. My core question is how do I use the position column in the historic learning data, which will not be a feature during deployment. Do I just ignore the column? Or is there some way to use the column to have a better model, even though the features will be the other columns? Also, any special performance metric to evaluate, other than the usual precision-recall metrics?
Any idea around this problem set up or any paper/reference will be greatly appreciated!


texts: Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

texts:
I've my data in a Pandas df with 25.000 rows and 1.500 columns without any NaNs. Of the columns about 30 contain numerical data which I standardized with StandardScaler(). The rest are cols with binary values which originated from cols with categorical data. (used pd.get_dummies() for this)

Now I'd like to reduce the dimensions. I'm already running

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
pca.fit(df)

for three hours and I asked my self if my approach was correct. I also saw two variants of PCA, one for sparse data. Does it mean that it doesn't make sense to run PCA in such a mixed scenario?

As I was up to now busy with cleaning and transforming my data, I'd like to understand what a good strategy would be to eliminate irrelevant columns.

I'd appreciate some hints to move forward.


texts:
I want to use the SQL table connector to link data in a PostGres database which is on my local machine running as 9.6. When I do this I get the following error;

could not open extension control file ".../postgressql/extension/quantile.control" No such file or directory

Looking so some similar issues I  tried:

find /usr -name quantile.control

in terminal and get:

find: /usr/sbin/authserver: Permission denied

What can I do to resolve this?

EDIT: From this answer I can see tat I need to install an extension, which I do according to the instructions here. But am not able to get the extension to install getting the following error:

INFO: best version: quantile 1.1.5
INFO: saving /var/folders/0c/g8tsv9jx3qgdwrdpqvjx45tr0000gn/T/tmppe1b4shp/quantile-1.1.5.zip
INFO: unpacking: /var/folders/0c/g8tsv9jx3qgdwrdpqvjx45tr0000gn/T/tmppe1b4shp/quantile-1.1.5.zip
INFO: building extension
Makefile:16: /Applications/anaconda/lib/pgxs/src/makefiles/pgxs.mk: No such file or directory
make: *** No rule to make target `/Applications/anaconda/lib/pgxs/src/makefiles/pgxs.mk'.  Stop.


texts:
I'm trying to run the code below in my Jupyter Notebook.
I get:

AttributeError: module 'tensorflow.python.keras.utils' has no
attribute 'to_categorical'

This is code from Kaggle tutorial. I have installed Keras and Tensorflow.
 import numpy as np
    import pandas as pd
    from sklearn.model_selection import train_test_split
    from tensorflow.python import keras
    from tensorflow.python.keras.models import Sequential
    from tensorflow.python.keras.layers import Dense, Flatten, Conv2D, Dropout

      img_rows, img_cols = 28, 28
    num_classes = 10

    def data_prep(raw):
        out_y = keras.utils.to_categorical(raw.label, num_classes)

        num_images = raw.shape[0]
        x_as_array = raw.values[:,1:]
        x_shaped_array = x_as_array.reshape(num_images, img_rows, img_cols, 1)
        out_x = x_shaped_array / 255
        return out_x, out_y

    raw_data = pd.read_csv('trainMNIST.csv')

    x, y = data_prep(raw_data)

    model = Sequential()
    model.add(Conv2D(20, kernel_size=(3, 3),
                     activation='relu',
                     input_shape=(img_rows, img_cols, 1)))
    model.add(Conv2D(20, kernel_size=(3, 3), activation='relu'))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))

    model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer='adam',
                  metrics=['accuracy'])
    model.fit(x, y,
              batch_size=128,
              epochs=2,
              validation_split = 0.2)


texts: Predicting when component will fail having its parameters data

texts:
I have a component and I need to predict when it will wear out and will need replacement. I monitor, let's say 5 parameters of this component, each one is monitored for every run cycle. So, the dataset can look like this:

No_of_runs    Para1    Para2    Para3    Para4    Para5
1              100      32        45      230       86
2              101      34        65      234       90
3              120      24        32      242       80
4              105      45        40      213       75
5              90       42        54      200       77
...            ...      ...       ...     ...       ...

In addition to the above, I have a dataset saying at which point this component needed replacement in the past. So, if the above dataset reaches up to 500 rows (run cycles of the component), the other dataset says that the component nedded replacement in cycles 50, 130, 340, 400.

Based on these data, the goal is to create a model able to predict when this component is going to fail in the future giving it the parameters' data up to this time.

I struggle to come up with a solution to the problem and as I have not much of experience in the field, I'm not sure of which approach to take.

It's a supervised problem, but I'm not sure of how to integrate the information of 'degradation' of the component (run cycles) or how to structure my features matrix.


texts: Understanding depthwise convolution vs convolution with group parameters in pytorch

texts:
So in the mobilenet-v1 network, depthwise conv layers are used. And I understand that as follows.

For a input feature map of (C_in, F_in, F_in), we take only 1 kernel with C_in channels, let's say, with size (C_in, K, K), and convolve each channel of the kernel to each channel of the input, to produce a (C_in, F_out, F_out) feature map. Then do pointwise conv to combine those feature maps, using C_out kernels with size (C_in, 1, 1), we get a result of (1, F_out, F_out). The kernel parameter reduce ratio comparing to normal conv is:

(K*K*C_in+C_in*C_out)/(K*K*C_in*C_out) = 1/C_out + 1/(K*K)

And I also checked Conv2d(doc) in pytorch, it is said one can achieve the depthwise convolution setting groups parameter equals to C_in. But as I read related articles, the logic behind setting groups looks different with the above depthwise convolution operation that mobilenet used. Let's say, we have C_in=6, and C_out=18, groups=6 means you divide both input and output channels to 6 groups. In each group, 3 kernels each having 1 channel  is used to conv with a input channel, so a total of 18 output channels can be produced.

But for a normal convolution, 18*6 total kernel-channels are used for 18 kernels, each having 6 channels. So the reduce ratio is 18/(18*6), thus the reduce ratio is  1/C_in=1/Groups . Leaving out the pointwise conv not considered, this number is different with the 1/C_out in above conclusion.

Can anyone explain where am I wrong? Is it bcause I missed something when C_out = factor * C_in (factor > 1) ?


texts: What is the difference between layer degradation and overfitting?

texts:
I've read the ResNet paper (https://arxiv.org/pdf/1512.03385.pdf)

They determined the layer degradation as that model with less layers learn quicker than with more. It can be visiable on plots below:

It looks like they say like overfitting is cause of it in subsection ("Exploring Over 1000 layers.")

  The testing result of this 1202-layer network is worse than that of
  our 110-layer network. We argue that this is because of overfitting

I think those are related like degradation causes overfitting, because it is like fitting too complex functions and is same as fitting too many layers. If that is correct than I do know know what type of overfitting is meant when the 1000+ layers are trained.

How to understand difference between those 2 concepts?


texts:
Please let me know if this question belongs elsewhere.
In my simple data set focused on sales pursuit opportunities, I have the following columns available.

Pursuit Name
Status(Open, Won, Lost)
Date Entered
Date Closed

I need to calculate the an average "cliff" date value of when a pursuit will likely either lose or never close. So if another opportunity has been open for x number of days, we can say (based on past opportunities) that this will either never close or will be lost.
My assumption is that this should be based on the difference between Date Entered and Date closed, but I'm unsure of the calculations needed to produce an average "Cliff date".
Is there another term aside from cliff date I should be using? Are there any ideas of how this can be calculated?
Thank you for your time.


texts:
I recently stumbled upon these layers, and it's an interesting idea, but I'm not sure what one would do with it.
What is the application of a Convolutional layer that doesn't share its weights? Where would you use a LocallyConnected layer that a Convolutional Layer wouldn't also be appropriate?


texts: What types of matrix multiplication are used in Machine Learning? When are they used?

texts:
I'm looking at equations for neural networks and backpropagation and I see this symbol in the equations, ⊙. I thought matrix multiplication of neural networks always involved matrices that matched dimensions on both sides, such as... [3, 3]@[3, 2]. (This is what is happening in the animated gif).

What part of a neural net uses a Hadamard product and which uses the Kronecker product? Because I see this notation for the Hadamard product (⊙) in papers and deep learning learning materials.


texts: why we need data labelling tool for computer vision?

texts:
Before start training images with tensorflow object detection api we need to use labelling tool to annotate our images and converted to XML format.
What happens when we convert our annoted image to XML file? Why we need that?


texts: Feature importance for random forest classification of a sample

texts:
Using a random forest is it possible to determine which features were the driving features to classify a specific sample as class A?

I know I can ask which features are more important to perform classification of ANY sample, but can I ask this for a specific sample?  E.g. Why was sample 1 classified as A?  Which of its features were much more like class A than class B?

Does it even make sense to ask this question of a random forest?

Bonus points on how to do it with sklearn in python :)

EDIT

Question has been answered in a crosspost here:  https://stats.stackexchange.com/questions/174229/feature-importance-for-random-forest-classification-of-a-sample

Python implementation here:  http://blog.datadive.net/random-forest-interpretation-with-scikit-learn/


texts: It may be unlikely that anyone knows this but I have a specific question about Freebase.  Here is the Freebase page from the Ford Taurus automotive model .  It has a property called "Related Models".  Does anyone know how this list of related models was compiled.  What is the similarity measure that they use?  I don't think it is only about other wikipedia pages that link to or from this page.  Alternatively, it may be that this is user generated.  Does anyone know for sure?


texts: How to use a dataset where attribute names are changed?

texts:
I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed.

My goal is to use dataset to decide whether new credit requesters request is approved or not, base on the data provided by them.

I see this dataset is widely use, but I do not know use It without knowing the attribute names?(I am aware that some attribute values can be changed to protect confidentiality)

What piece of knowledge am I missing here to use this dataset?


texts: Classification with millions of records, thousands of categories - keep memory use efficient?

texts:
I have about 20 categorical variables to predict another categorical variable. One of the variables can have as many as 12000 levels (O/P will be one of the 12000) and another one can have about 8000 levels. Rest will have less than 100 levels. Random Forest was the first thing that came to my mind, but Python’s Random Forest implementation doesn’t support categorical variables. If I one-hot encoded all these variables, then I will end up with thousands of variables for Millions of records which will throw me into memory issues.

What are my options?


texts: Passing variable length sentences to Tensorflow LSTM

texts:
I have a tensorflow LSTM model for predicting the sentiment. I build the model with the maximum sequence length 150. (Maximum number of words) While making predictions, i have written the code as below:

batchSize = 32
maxSeqLength = 150

def getSentenceMatrix(sentence):
    arr = np.zeros([batchSize, maxSeqLength])
    sentenceMatrix = np.zeros([batchSize,maxSeqLength], dtype='int32')
    cleanedSentence = cleanSentences(sentence)
    cleanedSentence = ' '.join(cleanedSentence.split()[:150])
    split = cleanedSentence.split()
    for indexCounter,word in enumerate(split):
        try:
            sentenceMatrix[0,indexCounter] = wordsList.index(word)
        except ValueError:
            sentenceMatrix[0,indexCounter] = 399999 #Vector for unkown words
    return sentenceMatrix

input_text = "example data"
inputMatrix = getSentenceMatrix(input_text)\

In the code i'm truncating my input text to 150 words and ignoring remaining data.Due to this my predictions are wrong.

cleanedSentence = ' '.join(cleanedSentence.split()[:150])

I know that if we have lesser length than sequence length we can pad with zero's. What we need to do if we have more length.
Can anyone suggest me the best way to do this. Thanks in advance.


texts: Understanding general approach to updating optimization function parameters

texts:
This question not related to a specific method or technique, rather there is a broader concept that I'm struggling to see clearly.
Introduction
In machine learning, we have loss functions that we're trying to minimize. Gradient descent is a general solution to minimizing the output of these loss functions. I understand the basic idea there, but it's the details that I'm getting stuck on.
Suppose that I have some loss function $J_\Theta(x)$, where I have some $x$ input, where $\Theta$ is just some matrix of arbitrary parameters.
In many examples, I see $\Theta$ written something like:
$\Theta =
\begin{bmatrix}
    \hat{x_0} \\
    \hat{x_1} \\
    \hat{x_2} \\
    \vdots \\
\end{bmatrix}
$
and each row of $\Theta$ is actually a vector.
Note that some concrete examples that I've run across include updating word embeddings in word2vec, or updating a softmax layer. I'm happy to elaborate if my explanation is too abstract.
In the examples of gradient descent that I've seen, typically the derivative of $J$ is taken w.r.t each row of $\Theta$, not individual elements.
So something like $\frac{dJ}{d\hat{x_0}}$, where each vector in $\Theta$ is updated according to the output of this gradient function.
Now for my point of confusion:
Suppose the parameters are initialized to zero, which is sometimes a thing. Wouldn't the updates be the same for each element of $\hat{x_i}$ in $\Theta$ during the update step? Wouldn't that lead to each vector having the same numbers for each element? I know that's the wrong conclusion, but I'm not able to see how each dimension would, in the end, become different values. I assume (hope) I'm missing something simple.


texts: difference btw data distribution and frquency distribution

texts: I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?


texts: Model accuracy changes when target attribute change

texts:
I have a dataset. For a target attribute for example:person, I get an accuracy which is different when I use for example:price as target attribute and so on.

Why this accuracy keep changing?


texts:
My problem is as follows:
I have a long segment [say, 100000 samples] as input.
Regardless of the method, I need to output a partition of this segment into k sub segments, which do not overlap, and whose union can but doesn't have to be the entire segment.
For instance, for the segment [0, 1000], a valid output can be the list
[[0, 10], [20, 50], [500, 900], [950, 970]].

I have a ground truth of the "correct" partition, for example
[[0, 15], [25, 40], [400, 650], [950, 975]].
Notice the number and length of the "truth" segments doesn't have to even resemble the number and length of the actual segments.
I need a measurement for the error.
I would like something "standard", although I know that is not well defined.
Currently, I don't have any special requirements from that error function, so I would just like one that makes sense, a trivial one is fine too, as I can't come up even with that.
If some error functions for such problem exist, please let me know of them, or even what that problem is called, so I can search for something.
Thanks

Also, if this belongs in another stack exchange, please let me know.


texts: How to evaluate multi label image retrieval model

texts:
I'm using a deep hashing model to search most similar images in a database (most similar to the image given as a query). I'm doing this on the coco dataset which has multiple labels per image. I'd like to evaluate the performance of the model but I'm not sure what type of metric should be used here.

If it was just a single label per image, I'd go for something like mean average precision (given a query image of a dog, check how many dog images the system retrieved, evaluate MAP). But this obviously cannot be used on the multi label task (given a query image of three classes, system retrieved image with just one of them, it's not totally correct but it's not incorrect either). So are there any commonly used metrics to evaluate this kind of tasks? If so, please refer me to them. Or do I have to come up with something own (maybe some kind of a weighted MAP)?


texts: Will the features in the image (edge, color, etc.. ) impacts on the performance of the spherical k-means?

texts:
I am very new in Machine learning, I recently implemented the spherical k-means, but finally I found a interesting point from the result. I used four datasets, they are minst, cifar-10, fashion-minst, and svhn. I was following the paper Learning Feature Representations with K-means (Coates & Ng, 2012) https://www-cs.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf. I did not finish the deep learning yet.

See the following image, I tried the max pooling and average pooling to see if there any difference, but I found an interesting point, the dataset with the clear feature (edge, color, etc. of the object in the image ) has the high performance, the dataset without clear feature has low performance.

See the four pictures on the bottom, I extracted one image from each dataset. I ordered them according the performance. I tried search online, but I did not get any useful information about my question.

I am not sure whether the point is correct or not. Could any one please explain this to me. Thanks!


texts: What are the tradeoffs between Bayesian Deep Learning and Deep Gaussain Processes?

texts:
I understand the differences between Deep Gaussian Processes(DGPs) and Bayesian Deep Learning(BDL): DGPs are essentially feed-forward neural networks where each node is a Gaussian Processes, which BDL places a prior belief on parameters of a normal(potentially convolutional) neural network.

But what are the trade-offs and relationships between these two models?


texts: Train a RNN with strings using positive and negative samples

texts:
I'm trying to recognize strings coming from a regular language, using positive and negative samples by Recurrent Neural Networks.

In particular, I tried to use the rnnlib by Alex Graves, but I had some troubles in changing the architecture of the network to adapt the network to my needs.

This is an example of my training set:

str1: w1 w2 ... wn --> label 1 (it is a valid string for the language)

str2: w1 w2 ... wi --> label 0 (it does not belong to the language)

where wi are drawn from the alphabet of the unknown language.

In other words, I'm trying to use RNN as a binary classifier.

Can you suggest a good tutorial or reference for this task?

Thanks in advance.


texts:
I have recently ported a stemmer from Java to Python for a highly inflectional language.
The stemmer learns how to change suffixes from the dictionary of words and their inflected forms. It basically builds a stemming table with learned stemming rules. As I was porting the algorithm I decided to train it on a larger dictionary. As a result, the learned stemming table got bigger, and stemming accuracy got higher as well.
Then I thought this actually make no sense as the stemming table size gets closer and closer to the size of the dictionary.
Why build or train stemming algorithms if you can simply lookup a dictionary?
I can understand that in old times storing large files could be a problem, but now? And for some languages there might be no proper dictionary resources. But is there any other reason?


texts:
I'm trying to read a csv file into a dataframe in Python but when examining the file I noticed they are many misaligned rows . Does anyone know how I can shift these rows so my Dataframe's size is consistent.
data= pd.read_csv('MyData.csv')


texts:
I would like to create some annotations on some texts using BRAT (brat rapid annotation tool). Can BRAT be installed on Microsoft Windows? If so, how?

The installation instructions do not rule out Microsoft Windows:

  The brat server is a Python (version 2.5+) program that runs by default as a CGI application, and the installation script assumes a UNIX-like environment. If you are setting up brat server in a compatible environment with an existing web server that supports CGI, the quick start instructions for using CGI should work. If you don't have a web server installed and want to try brat out locally, you may want try out the quick start instructions for the standalone server. (For security reasons, we strongly recommend serving brat via a full web server such as Apache in production environments.)

However they don't seem to give any instructions for Microsoft Windows.

I don't want to use some in emulation of Unix such as virtual machines or Cygwin.

If interested, to use BRAT with Cygwin: https://github.com/Franck-Dernoncourt/NeuroNER/blob/master/install_windows.md

  After installing Cygwin, run Cygwin.bat (by default located on
  C:\cygwin64\ if you used the 64-bit installation for Cygwin). Then,
  in the terminal that just opened, run:

mkdir brat
cd brat
wget http://weaver.nlplab.org/~brat/releases/brat-v1.3_Crunchy_Frog.tar.gz
tar xzf brat-v1.3_Crunchy_Frog.tar.gz
cd brat-v1.3_Crunchy_Frog
./install.sh -u
python standalone.py

  BRAT should now be accessible through the web browser at
  http://127.0.0.1:8001.


texts: How do data scientists test the claimed accuracy of a model?

texts:
For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go about verifying this claim?

I would test it on the data that I possess and see what the acc. is for detecting frauds in my data-set, and run a t-test with the sample acc., to see if it is close enough to 90% to make the call that it would detect 90% of all frauds in the entire population.

Let me know if this is the correct way to think about this question.

Thanks


texts: What's the purpose of padding with Maxpooling?

texts:
As mentioned in the question, i've noticed that sometimes there are pooling layers with padding.

More specifically, I found this Keras tutorial, where there's a net which contains MaxPooling layers with padding.

If padding=same in convolutional layers, our output size (at least height and width, depth can change based on the number of filters) is the same as the input.

I expected the same with the MaxPooling layer, but Keras model.summary() (as shown in the article) shows that the output size after the pooling layers is half of the input.

What's the point of adding padding to the Pooling layer if we still get an output which is half of the input?


texts: Tensorflow.Keras: How to get gradient for an output class w.r.t a given input?

texts:
I have implemented and trained a sequential model using tf.keras. Say I am given an input array of size 8X8 and an output [0,1,0,...(rest all 0)].
How to calculate the gradient of the input w.r.t to the given output?
model = ...
output = np.asarray([0, 1, 0, 0, 0, 0, 0, 0, 0, 0])
input = np.random.randn(1,64)
pred = model.predict(input)

gradient = somefunction(model,input,output,pred)

Is there any way to achieve that?
I am trying to implement a project similar to Google Deep Dreams, so that given a random picture, if I mention a digit, then with each iteration I will update the input with its gradient to make the picture more like the digit that was asked.
I tried to follow other StackOverflow answers and Keras documentation. But as they are not mentioning the input shape, when someone writes input[:,:,something] I am having a hard time translating it to my requirement.
Can you provide a simple generic answer so that given an array as input and an expected output array, one can calculate the gradient of loss w.r.t the input for that exact output.


texts:
Cross validation on a dataset with 22680 observations. Want the training set to contain 21420 entries. How many folds can you use for your cross validation?
How do you calculate the folds?


texts:
I have a dataset of shape edges, that I am trying to make a model for with sklearn. I'm new to the machine learning world, so I am struggling to create a good model. Using SVM, I was able to get a supposed 81% precision, but when I feed it an image outside the training or test set, it consistently returns the wrong prediction, almost every time.

Question: Is there is a better way of doing this than using SVM? Or are these shapes too similar? I have 90 images in the training set.

Here is a link to my ML code.


texts: global independence vs local independence in markov network

texts:
I could not understand the local independence and global independence of a markov network.
Please help me understand with a simple graph


texts: Represent time-series data in much compact form

texts:
I have time series data of one month plotted day-wise as

Notice every day follows a different pattern. Now, I want to show this "diversity in pattern" of each day in much compact form in a research paper. What are the different ways/options of representing this in compact form using R.


texts:
I'm trying to use Tensorflow to optimize a few variables to be used in a KNN algorithm, however, I'm running into an issue where I'm unable to have a layer work properly if it is not connected to an Input layer.

What I'm trying to do below is pass in [[1]] as a static tensor, and then using the Lambda layer to coerce the weights into a usable shape.  Once trained, I would just retrieve the weight value from the first Dense layer.

ones = tf.ones(shape=(1,1)) # trying to use this to take the place of dynamic inputs

theta_layer = layers.Dense(1, activation="linear", use_bias=False, trainable=True)(ones)
theta_layer = layers.Lambda(lambda x: tf.ones(shape=(self.batch_size,1)) * x)(theta_layer)
print(theta_layer)
# tf.Tensor(
# [[-1.151663]
# [-1.151663]
# [-1.151663]], shape=(3, 1), dtype=float32)

concat_layer = layers.Concatenate()([theta_layer, features_input])

model = models.Model(inputs=features_input, outputs=concat_layer)
model.compile(optimizer="adam", loss="mse") #, metrics=["mae"])#, bias_regularizer=None)

The problem with the above strategy is that theta_layer does not show up in model.summary() and the weights don't show up in model.get_weights().

Also, when I try to directly set theta_layer as the model output, I get this message:

AttributeError: Tensor.op is meaningless when eager execution is enabled.

This is probably the giveaway but frankly I'm just not knowledgeable enough about Tensorflow.

I know that I can fix this by passing in a separate tf.ones input to the model (not just into specific layers), but it would complicate the code seemingly unnecessarily (I have to do this for multiple layers/variables).  Is there a way that I can alter the tf.ones passed to the layer or the alter the layer itself to eliminate this issue?


texts: Linear regression incorrect prediction using Matlab

texts:
In the plot below the red crossed line is the actual curve and the crossed blue line is the predicted curve. I am using least squares for linear prediction. I have used 1:79 examples in training and the remaining for testing. The test data points are never seen during training. What is my mistake? Why am I getting such a weird prediction? I want to see the sine curve as the predicted output which should be very close to the original data.

%generate some data
x=linspace(0,2*pi,100)';
y=sin(x); %response

X=x;
y=y;
% Convert matrix values to double
X = double(X(1:79));
y = double(y(1:79));

% Plot data
plot(X, y, 'rx', 'MarkerSize', 10);

m = length(y);
% Add ones column
X = [ones(m, 1) X];

% Gradient Descent with Normal Equation
theta = (pinv(X'*X))*X'*y

% Predict  from 80 till last sample
test_samples = x(80:end);
test_samples_val = [ones(length(test_samples),1) test_samples];

% Calculate predicted value
pred_value = test_samples_val * theta;

X = vertcat(X, test_samples_val);
regressionline = X*theta;

% Plot predicted value with blue cross
plot(test_samples, pred_value, 'bx', 'MarkerSize', 10);


texts: Feature Scaling and normalization in cross-validation set

texts:
I have a question that normally, when we are making a training set and a final test set,

  we would compute the mean and standard deviation for preprocessing
  using the training data and use it to standardize (transform) the test
  data.

So, when we are making a training data, a cross-validation data and a final test data, shouldn't we compute mean and std for preprocessing using the training data alone and then standardize the validation set and test set using it? And, if I'm right how can we do it in keras, since in keras like in the below code, taken from kaggle( by Francois Chollet MNIST tutorial):

simple-deep-mlp-with-keras by Francois Chollet

# pre-processing: divide by max and substract mean
scale = np.max(X_train)
X_train /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_test -= mean

model.fit(X_train, y_train, nb_epoch=10, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=2)

The validation set is created using the fit method and thus, it is also used in computing the preprocessing mean and std. So, how to create validation set before and use only the training data for feature scaling and normalization? And which method is the right method?

Also, in the above code scale and mean are being computed for each column respectively and is being used for normalization , should we compute mean and scale(standard deviation or range) for each pixel or for the entire image? How does that affect?


texts: NLP: Getting the top 5 or top 10 predictions

texts:
I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books',  it should yield some names.

Now, I have used the Infersent algo (to begin with) in order for my model to be able to answer questions.

I am getting only the best output that my model could predict viz., 'Alchemist'.

I want at least 4 or 5 other outputs, other words, the top five predictions.

I know that Xgboost has the ability to do this activity in some sorts, but I am not sure how I should use that in my problem.

Any heads up?

My apologies, I cannot share any code but I would really appreciate ideas and suggestions.

Thank you,


texts: Mixture Density Network: determine the parameters of each Gaussian component

texts:
I am reading Bishop's Mixture Density Network paper at:
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-ncrg-94-004.pdf

This is a good paper, but I am still confused about a few small details. I am wondering if anyone could give me some help:

Basically the mixing coefficient alpha_i can be computed through a softmax function in eq (25) below. However, in eq (25), what's the upper alpha for each $(z_i)^{alpha}$? Is it a free parameter to be fitted?
Similarly, in eq (26), what's the upper sigma in $(z_i)^{sigma}$? Is it a free parameter to be fitted as well? Thanks!


texts:
In the paper ImageNet Classification with Deep Convolutional Neural Networks, the size of input image is 224x224. The following figure shows the input size.

From caffe, deploy.prototxt file from the directory of the bvlc_alexnet says that the input size is 227x227.

Why is the input size different?


texts: Do Convolution Layers in a CNN Treat the Previous Layer Outputs as Channels?

texts:
Lets say you have a max pooling layer that gives 10 downsampled feature maps. Do you stack those feature maps, treat them as channels and convolve that 'single image' of depth 10 with a 3d kernel of depth 10? That is how I have generally thought about it. Is that correct?

This visualization confused me:
http://scs.ryerson.ca/~aharley/vis/conv/flat.html

On the second convolution layer in the above visualization most of the feature maps only connect to 3 or 4 of the previous layers maps. Can anyone help me understand this better?

Related side question: If our input is a color image our first convolution kernel will be 3D. This means we learn different weights for each color channel (I assume we aren't learning a single 2D kernel that is duplicated on each channel, correct)?


texts: GPU performance is about 50% slower than benchmarks

texts:
Running this benchmark I get 50% slower performance than the author on practically all deep learning sub problems (SINGLE precision and on TRAINING only):

I tested this on a GeForce 1080 GTX Ti and am using PyTorch 1.0.1 with Cuda 10.0.13 (cudnn version 7402) running on an AMD Ryzen 7 1700 Eight-Core Processor (with 16 threads). OS is arch linux. The GPU temperature remains below 80 degrees at full volatile usage.

Would anyone have an idea what causes the lower performance? Can someone else with the same gfx run this and let me know what differences they see?


texts: What is the best way to read SQL dataset in to Tensorflow?

texts:
What is the best way to read SQL database in to Tensorflow?

Currently, I am using Postgres on server and developed DL algorithm on Tensorflow using Jupyter Lab. How can I import data into Jupyter Lab using tf.data API? I do not want to store the data in the disk and keep running the algorithm when the new data arrives.

It seems like tf.data.experimental.SqlDataset only support for sqlite.

(NOTE: I did not upgrade my Tensorflow, so, I am using  tf.contrib.data.SqlDataset() for the minimal working example.)

I migrated the data from PostgreSQL to SQLite3 and using

#Ignore the warnings
import warnings
warnings.filterwarnings("ignore")

import tensorflow as tf
#To start an input pipeline, you must define a source
dataset = tf.contrib.data.SqlDataset("sqlite", "/home/musara1/musara_dev.sqlite3",
                                 "SELECT * FROM basetable LIMIT 10",
                                 (tf.string, tf.int32, tf.int32, tf.int32, tf.int32, tf.int32, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.int32, tf.int32, tf.int32))

iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
# Prints the rows of the result set of the above query.
sess=tf.InteractiveSession()

print(sess.run(next_element))

I can print the next element. However, there are other transformations I need to do on the dataset. such as splitting into training/validation/testing and getting rid of some columns et cetera. However, the output of tf.contrib.data.SqlDataset() is for me
<SqlDataset shapes: ((), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), ()), types: (tf.string, tf.int32, tf.int32, tf.int32, tf.int32, tf.int32, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.int32, tf.int32, tf.int32)>

I have 25 columns and tf.contrib.data.SqlDataset() creates 25 different tensorflow.python.framework.ops.Tensor. How can I bring them together? Therefore, I can use tf.data.Dataset.from_tensor_slices()?


texts:
I want to create a new DataFrame based upon the columns of another DataFrame, for which I would like to do the following:

Data Retrieval: Take partial strings from column elements from DataFrame1
Renaming Column Elements: Combine these strings and place into 2 new columns in DataFrame2

An example of the DataFrames can be seen below, where DataFrame1 is my actual data and the DataFrame2 is the type of data frame that I am interested in obtaining:

import pandas as pd
df1 = pd.DataFrame({'BGC Name': ['Hm106_120.region001', 'MBT13_26.region001',
'MBT23_64.region001', 'MBT36_100.region001'], 'GCF No': [1813, 1813, 1883,
1887], 'Organism': ['Streptomyces sp', 'Streptomyces sp', 'Listeria sp',
'Streptomyces sp'], 'BGC Class': ['PKSI', 'NRPS', 'PSKI', 'NRPS']})

df2 = pd.DataFrame({'GCF': ['PKSI_GCF1813', 'NRPS_GCF1813', 'PKSI_GCF1883',
'NRPS_GCF1887'],'Genome': ['Streptomyces_sp_Hm106', 'Streptomyces_sp_MBT13',
'Listeria_sp_MBT23', 'Streptomyces_sp_MBT36']})

print("DataFrame1 (Original DataFrame)")
print(df1)
print("DataFrame2 (Desired DataFrame)")
print(df2)

Snapshot can be found: [

The Code That I am trying to use

#Dataframe Copies
df1 = df1.copy() #self copy
df1.columns = df1.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
df2 = df1.copy() #copy of original dataframe
#Change Column Names
df1.columns = ['BGC', 'GCF', 'Description', 'Product Prediction', 'BiG-SCAPE class', 'Organism']
df2.columns = ['BGC', 'GCF', 'Description', 'Product Prediction', 'BiG-SCAPE class', 'Organism']
#DataFrame Manipulation
df2['GCF'] = df1['BiG-SCAPE class'].astype(str) + df2['GCF'].astype(str)
df1[['BGC','BGC2']] = df1['BGC'].str.split('_',n=1,expand=True)
df2['Genome'] = df1['Organism'].astype(str) + df1['BGC'].astype(str)
df2.set_index('GCF')
df2 = df2[['GCF','Genome']]
df2['Genome'] = df2['Genome'].str.replace(' ', '_').str.replace('Unclassified','_').str.replace('__','_').str.replace('.','')
print(df1)
print("Updated table")
print(df2)


texts: Incremental Learning with sklearn: warm_start, partial_fit(), fit()

texts:
I have built an ML model with the goal of making predictions for targets of the following week. In general, new data will come in and be processed at the end of each week and be in the same data structure as before. In other words, the same number of features, same classes for classification, etc.
Instead of re-training the model from scratch for each week's predictions, I am considering applying an incremental learning approach so that past learning is not entirely discarded and the model would (presumably) increase in performance over time. I'm working with sklearn  on Python 3. There were only a handful of posts on StackOverflow regarding this, but many of the answers seem inconsistent (possibly due to updates with sklearn's API?).
The documentation here and here suggests that incremental/online learning is possible with certain ML implementations - implying that the new datasets could be thought of as "mini-batches" and incrementally trained by saving/loading the model and calling .partial_fit() with the same model parameters.

Although all algorithms cannot learn incrementally (i.e. without seeing all the instances at once), all estimators implementing the partial_fit API are candidates. 1
Unlike fit, repeatedly calling partial_fit does not clear the model, but updates it with respect to the data provided. The portion of data provided to partial_fit may be called a mini-batch. Each mini-batch must be of consistent shape, etc. In iterative estimators, partial_fit often only performs a single iteration. 2

However, the documentation here is throwing me off.

partial_fit also retains the model between calls, but differs: with warm_start the parameters change and the data is (more-or-less) constant across calls to fit; with partial_fit, the mini-batch of data changes and model parameters stay fixed. 3
There are cases where you want to use warm_start to fit on different, but closely related data. 3

For the problem I am tackling, ideally model parameters should be adjusted based on cross-validation and new datasets should  be weighted more heavily than old ones due to concept drift. However  ignoring this for now,

In (3), what exactly does "(more-or-less) constant...different but closely related data" mean? Since the data structure of new datasets are the same, should i be calling estimator(warm_start=True).fit(#new df) or estimator.partial_fit(#new df)?
For iterative estimators such as sklearn.linear_model.SGDClassifier, only one epoch is run when using .partial_fit(). If I want $k$ epochs, would calling it on the same dataset repeatedly be the same as calling .fit() with $k$ epochs to begin with?
Do dedicated libraries such as creme offer any advantage for incremental learniing?


texts: Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

texts:
Context:
My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.).
It is too costly for us to emulate agents. We decided therefore to concentrate on Offline RL techniques. For this, we are currently using the RL-Coach library by Intel, which offers support for Batch/Offline RL. More specifically, to evaluate policies in offline settings, we train a DDQN-BCQ model and evaluate the learned policies using Offline Policy Estimators (OPEs).
Tools:
RL-Coach library implements different OPEs:

For Contextual Bandits problems: Inverse Propensity Score (IPS), Direct Method Reward (DM), Doubly Robust (DR)
For Full-blown RL problems: Weighted Importance Sampling (WIS) and a sequential verison of Doubly Robust (Seq-DR)

The implementation of Seq-DR is based on the following paper https://arxiv.org/pdf/1511.03722.pdf and is defined as the following: $V_{DR}^{H+1-t} := \hat{V}(s_t) + \rho_t(r_t + \gamma V_{DR}^{H-t} - \hat{Q}(s_t, a_t))$, where $\rho_t = \frac{\pi_e(a_t|s_t)}{\pi_b(a_t|s_t)}$. The target policy probabilities $\pi_e(s|a)$ is computed from the softmax of the learned Q-table $\hat{Q}(s,a)$. The state value estimate $\hat{V}(s_t)$ is computed from the Q-table value and softmax probabilities too. The behavior policy probabilities $\pi_b(s|a)$ are directly estimated from the data.
Problem:
At each epoch, we evaluate the learned policy with Seq-DR. Systemtically, we get gigantic values in the orders from $10^{18}$ to $10^{38}$, while the WIS estimation gives values ranging from 100 to 500, which is far more sensible given the returns in our data.
Problem cause explorations:
I have looked at the different terms composing the above equation separately, too see where these estimation explosions come from. What I have discovered is that the Seq-DR value always explodes when evaluating the same specific episodes in our data. I focused on a specific one and looked at the estimated value after each transition. Basically, what happens is that at some transition $t$, during ~5 consecutive transitions, $\rho_t = ~7$ (while it generally takes values between 0 and 2) and these transitions are sufficient to start the explosion of Seq-DR estimates (as the definition is recursive). All other terms in the above formula have reasonable values and I am pretty sure that the explosions is not because of them.
We also tried to look at the length of our episodes. Our episodes actually have values of different lengths which can sometime be pretty long, where no clear horizon exists (infinite horizon case). We tried to cut those episodes into smaller episodes of length 20 to see if the problem would disappear, as long episodes might lead to unstable values, but this didn't change anything and the Seq-DR value still explose.
Questions:
Why would Doubly Robust estimators give such gigantic values? Doubly Robust estimators are combinations of both Model-Based and Importance Sampling estimators. They are supposed to still give good values when either one of the two following conditions are true: 1) The reward model is accurate, 2) The behavior policy is close the the reality. If the values we get with the WIS estimator are reasonable, why would we get completely off values for a DR estimator?
Is it possible that this happens just because of a distributional shift between the Q-values target policy probabilities and the behavior policy probabilities, i.e. the Q-values would like to go to take actions where we actually only have very few data? But in that case, shouldn't the WIS estimations be completely off as well?


texts: How do I forecast sales data down to the individual item?

texts:
I have a dataset that looks like,

order datetime, customer id, product name, type of product, quantity sold

I want a model to forecast sales for each individual item.  I'm thinking of using one of the following but would like some advice, literature, or other options.

One time series forecast (ARIMA) for each product
RNN
HMM

What's the best method to predict sales of each individual item into the future?


texts: Scheduler for activation layer parameter using Keras callback

texts:
I want to train a keras model with a custom activation layer.
The custom activation layer has one fixed non trainable parameter.
I want to change/set this non trainable parameter of all custom activation layers in model during training after few epochs.
How to achieve this using keras callback?


texts: Accelerate deep learning model training on several GPUs

texts:
I have a deep learning model that can be trained in one GPU, however, is very slow. Is there a way to accelerate the training by parallelizing it across several GPUs? How would be the training process?

I can use any framework (pytorch, tensorflow, etc). And I know they accept multiple GPU and how to do it.  I am more interested in how to spread, conceptually, a single model between GPUs to gain performance. I do mostly NLP models with RNN but I am also interested in CNN.


texts: What is the best way to visualize 10 Fold Cross Validation Scores?

texts: I have trained a CNN model and I have applied 10 Fold Cross Validation because I don't have much data to train the classifier. Now I am unsure about how to visulize fold wise results. Please suggest some visualization charts or techniques to display fold wise results.


texts: Meaning of axes in Linear Discrimination Analysis

texts:
Looking at the LDA of the Iris Dataset, which looks like this:

It's understandable that the 3 types of flowers have succesfuly been seperated into categories, with versicolor and virginica slightly overlapping. My question is, how does this help us in a practical way? What can we do with the information, now that we've seperated the 3 categories like this. It's great to know that Setosa is apparent for values between 5 and 10, but what do those values mean?


texts:
Problem
From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of the model when predicting on the observed data. This method requires the user to identify non-anomalous data beforehand.

What if it's not possible to label non-anomalous data to train the model?
Is there anything in literature that explain how to overcome this issue?

I have an idea, but I was wondering if someone has heard for something similar before, and could point me to the right direction (link papers/blogs, or explain existing methods).
Idea
I'd like to train a decision tree on a dataset $X$ with $N$ rows, $p$ columns, having a real valued target variable $Y$ (it's a regression problem). The dataset $X$ contains both anomalous and non-anomalous objects. The decision tree training process generates groups of objects, splitting the dataset iteratively along one dimension, at each iterations. The decision trees during prediction assigns an object to a specific leaf node. Each leaf node will have a certain distribution of values of the target variable Y. An anomalous object in this problem it's the one which doesn't perform well, more precisely an object for which Y is too low.

Can I use the distribution in a leaf node to perform anomaly detection?
Assuming that a bigger value for Y is preferred, can I say that the entities in the lowest 5th percentile of Y in a node are outliers?

Example
The decision tree assigns to the node j the objects following the rule: $2 <= x_1 < 4$ and $5 <= x_2 < 7$, where $x_1$ and $x_2$ are two columns of the dataset. If I run a prediction on the entire dataset, the values of Y in the node j have have a gaussian distribution (mean=25, std=3), and I consider the values with $Y < 20.5$ outliers. The idea is that objects with $x_1$ and $x_2$ within a range, should perform similarly, and never below a target value.
Considerations
*First of all, I can see that this problem can be generalized to other methods. However, I find decision trees easier to explain here, moreover, I find the hard clustering property of trees useful for my problem, as I also need to cluster objects together.
*Second, I can see the issue of training the model with both anomalous and non-anomalous data. Though, I'm wondering whether due to some property, with a big amount of data, the leaf node's distribution move's toward the optimal (value of Y of the non-anomalous objects). But I guess this depends on the ration between anomalous vs non-anomalous objects.
Any hint in the right direction would help.


texts: Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?

texts:
Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant?

When applying BP, the derivative of CEL is the difference between the output probability (SM) and the one hot encoded vector. For me the CEL output, which is very sophisticated, does not play any roll for learning.

I´m expecting a fallacy in my reasoning, so could somebody please help me out?


texts:
I have a dataset with 600 examples of target class yes and 300 of no.
After classification I get high accuracy(75%) but low kappa (0.4).
What can it mean? It seems that kappa is really low, am I right?
Other metrics like precission or AUR are about 0.7-0.8 what seems to be good.


texts: Outlier detection and removal using scatter-plots and histograms

texts:
Suppose, we use the following code to generate scatter plots,

function res = plot2features(tset, f1, f2)
% Plots tset samples on a 2-dimensional diagram
%   using features f1 and f2
% tset - training set; the first column contains class label
% f1 - index of the first feature (mapped to horizontal axis)
% f2 - index of the second feature (mapped to vertical axis)
%
% res - matrix containing values of f1 and f2 features

    % plotting parameters for different classes
    %   restriction to 8 classes seems reasonable
    pattern(1,:) = 'ks';
    pattern(2,:) = 'rd';
    pattern(3,:) = 'mv';
    pattern(4,:) = 'b^';
    pattern(5,:) = 'gs';
    pattern(6,:) = 'md';
    pattern(7,:) = 'mv';
    pattern(8,:) = 'g^';

    res = tset(:, [f1, f2]);

    % extraction of all unique labels used in tset
    labels = unique(tset(:,1));

    % create diagram and switch to content preserving mode
    figure;
    hold on;
    for i=1:size(labels,1)
        idx = tset(:,1) == labels(i);
        plot(res(idx,1), res(idx,2), pattern(i,:));
    end
    hold off;
end

The following is its usage,

>> plot2features(train, 3,4)

This code generates the following image before removing outliers,

and following image after removing outliers,

I have the following questions,

(1) What do the 1st image tell us about the existence of outliers? I can guess that the plot at a distant position is an outlier. But, how can I find which row or column is generating outliers? according to the 1st picture, the outlier is situated at (27,375) coordinates. But, in the actual data it is situated on the train(184:188,:) rows. So, why is that difference?

(2) What do the color codes in the second picture represent?

(3) Why has the two images that much different? Why only removing 4 rows bring so radical differnce?

(4) How can we analyze the existence of outliers using histograms? Please, supply me any study material about outlier detection using histograms.

Suppose we have the following training and test data in our hands to be used in testing Bayes Classifier algorithm,

Training data

train.txt

Test data

test.txt

First column represents class. Rest of the columns represent features.


texts: Splitting a geospatial data set with existing groupings

texts:
Data set:
I have a data set consisting of trip data for a large number of cars. The raw data consists of rows with (GPS) coordinates, timestamp, current fuel consumption, temperature, etc. It has been map matched to OpenStreetMap, leading to trip segments, each with a road segment from OSM, a number of associated rows in the raw data, plus aggregated fuel usage, average temperature, etc. Additionally, the trip segments have been grouped into trips, each of which is a continuous driving session, and have been assigned a trip ID.

Target:
I'm looking to predict fuel consumption for trips. For this, I'm planning to train a model that can predict fuel consumption for trip segments, then sum up the predictions for a trip. Following, I'd like to split up my data set of trip segments into training and test sets.

Challenges:
I want to keep segments for each trip together in one of the two sets, since this will allow me to calculate the error for whole trips during testing.

At the same time, I would like to hit a specific ratio of trip segments between the two sets, but my trips have different numbers of segments, so a simple split of the trips according to this ratio might skew the ratio of segments.

Third, I believe I would like to split the data in such a way as to keep the two sets representative of the full data set. That is, that both sets contain trip segments from across the area that I have data for.

However, I have read a couple of articles with other types of spatial data (for example electoral and agricultural data) that explicitly try to split the data in a way such that there is no overlap in order to prevent implicit contamination of training data with test data. I don't have much experience with spatial data sets, so this makes me unsure of the what my goal should be in this regard.

I don't believe it's possible to both keep the trips grouped and avoid overlapping trip segments, since most trips share segments on heavily travelled roads.

Question:
How should I split the data set to best satisfy the goals described above? Or are any of my assumptions incorrect, and should I approach the problem in a different way?


texts: What are the main distribution semantics based algorithms?

texts:
I am aware that LSI, RRI and word embeddings are distributional semantics models. However, I am not certain if the below mentioned are also distributional semantic models.

Non-Negative Tensor Factorisation
Singular Value Decomposition (SVD)
Vector Space Model (VSM)

Please let me know if the above mentioned algorithms are also distributional semantics models. Moreover, please also let me know the other distributional semantics based algorithms.


texts: Dealing with large data: selecting a sample

texts:
I'm given a data set to create a model that would predict whether a certain supply chain would be able to deliver the goods without delay or not. I'm doing this in Python.

The data set has 93000 data points has at least 100 features as given. What's worse, most of these features are categorical data and each feature has around 200 unique categories.

I want to develop the best model I can but I find the size of the data set is too large for this.

I tried different methods of feature reduction. But even for feature reduction methods, it takes infeasible amount of time in my laptop. So I thought the best thing I could do is to first select a sub sample of the data points and find the best model and best parameters and then train the whole data set with this model.

Is selecting a sample from the data set the best thing I could do in this situation?

If not, your kind suggestions are welcome.

If it is, what's the best method of selecting these samples?


texts: How to import statsmodels module to use OLS class?

texts:
I am trying multiple Regression

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Importing Dataset
dataset = pd.read_csv(
    'C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 2 - Regression/Section 5 - Multiple Linear Regression/50_Startups.csv')
print(dataset)
X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, 4].values

# Categorical Data

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

labelencoder = LabelEncoder()
X[:, 3] = labelencoder.fit_transform(X[:, 3])
onehotencoder = OneHotEncoder(categorical_features=[3])
X = onehotencoder.fit_transform(X).toarray()

# Splitting the dataset into training set and test set

from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)
print(Y_train)

# Fitting Multiple Linear Regression

from sklearn.linear_model import LinearRegression

regressor = LinearRegression()
regressor.fit(X_train, Y_train)

# predicting the test result
Y_pred = regressor.predict(X_test)

This is the part where error is occuring

# Building the optimal model with Backward Elimination
import statsmodels.formula.api as sm

X = np.append(arr=np.ones((50, 1)).astype(int), values=X, axis=1)
print(X)
X_opt = X[:, [0, 1, 2, 3, 4, 5]]
regressor_ols = sm.OLS(endog=Y, exog=X_opt).fit()
print(regressor_ols.summary())

This is the error message

Traceback (most recent call last):
  File "C:/Users/Rupali Singh/PycharmProjects/Machine_Learning/Muliple_Linear_Regression.py", line 39, in <module>
    import statsmodels.formula.api as sm
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\formula\api.py", line 15, in <module>
    from statsmodels.discrete.discrete_model import MNLogit
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\discrete\discrete_model.py", line 45, in <module>
    from statsmodels.distributions import genpoisson_p
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\distributions\__init__.py", line 2, in <module>
    from .edgeworth import ExpandedNormal
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\distributions\edgeworth.py", line 7, in <module>
    from scipy.misc import factorial
ImportError: cannot import name 'factorial'

Process finished with exit code 1


texts: How do neural networks account for outliers?

texts: How do neural networks account for outliers and overfitting?


texts: How to do this complicated data extrapolation, prediction modeling?

texts:
I have some very complicated data about some movie sales online, first for each data entry, I have a key which is a combination of five keys, which are territory, day, etc, and then, for each key I have the sales for a period of time, and other information, like the movie's box office and genre.
For each day, there is a delay for the data loading to the database, around ten hours, I try to fill the gap, do some data extrapolations.
For each movie we sell, there is some decay of selling since the new release of the movie, i.e. usually for each movie, it follows some sales decay pattern.
For a recent day, I pulled some data, and I found that some decay pattern:

And for that day, the sales for each key can range from around $150000 to $0. The pic is as follow:

In the picture, the 15000 means there are around 15000 keys for each day.
Found this article.
I am trying to predict for each key, the sales amount, like for a movie, territory, day etc combination, the sales amount, how much dollars, means for that movie, that territory, that day, how much money we get from selling online. I tried ARIMA time series model, but there is some concerns for that model, seen from the pics, there is some seasonal thing, and decay thing for the movie, so the sales prediction can not be always flat, there may be a pump after a going down, it may happens on a weekend, since there is seasonal thing, and the decay trend, etc, how to capture these things. Thank you for your reply!
I am not sure whether can be applied, and how to be applied here.
Thanks a lot in advance.


texts:
I have a set of clusters which each cluster contains a list of short documents. I want to compute how coherent and cohesive each cluster is and filter out the incoherent and in-cohesive ones.

I am aware of intra-cluster distance and within-cluster dispersion, which are part of computing clustering evaluation metrics Silhouette Coefficient and Calinski-Harabaz Index, respectively. My question is, is there any other metrics or ways to compute such intra-cluster coherence or cohesion? Is there any standard ways for this that I am not aware of? Thanks.


texts: why trainable parameters are not considered right?

texts:
I have tested the "ResNet" block and it works fine,but when i call it in the model class, in some how it does not work properly ?
Is it related to the model definition ?


texts: Using TensorFlow's preexistent Inception labeler

texts: I know I can retrain Inception to label images, but can I just provide an image to a non retrained Inception and get a label back?


texts: Paraphrase Generation - state-of-the-art?

texts:
I need to paraphrase short, but abstract sentences, such as:

  He prefers variety to routine.

or

  I am easily discouraged.

I've played around couple of online tools (such as https://quillbot.com/) just to get a feeling for how well this particular task is solved by available models and was quite disappointed. Hence my question:

What is current state-of-the-art in paraphrase generation?


texts:
In the MuZero paper pseudocode, they have the following line of code:

hidden_state = tf.scale_gradient(hidden_state, 0.5)

What does this do? Why is it there?

I've searched for tf.scale_gradient and it doesn't exist in tensorflow. And, unlike scalar_loss, they don't seem to have defined it in their own code.

For context, here's the entire function:

def update_weights(optimizer: tf.train.Optimizer, network: Network, batch,
                   weight_decay: float):
  loss = 0
  for image, actions, targets in batch:
    # Initial step, from the real observation.
    value, reward, policy_logits, hidden_state = network.initial_inference(
        image)
    predictions = [(1.0, value, reward, policy_logits)]

    # Recurrent steps, from action and previous hidden state.
    for action in actions:
      value, reward, policy_logits, hidden_state = network.recurrent_inference(
          hidden_state, action)
      predictions.append((1.0 / len(actions), value, reward, policy_logits))

      # THIS LINE HERE
      hidden_state = tf.scale_gradient(hidden_state, 0.5)

    for prediction, target in zip(predictions, targets):
      gradient_scale, value, reward, policy_logits = prediction
      target_value, target_reward, target_policy = target

      l = (
          scalar_loss(value, target_value) +
          scalar_loss(reward, target_reward) +
          tf.nn.softmax_cross_entropy_with_logits(
              logits=policy_logits, labels=target_policy))

      # AND AGAIN HERE
      loss += tf.scale_gradient(l, gradient_scale)

  for weights in network.get_weights():
    loss += weight_decay * tf.nn.l2_loss(weights)

  optimizer.minimize(loss)

What does scaling the gradient do, and why are they doing it there?


texts:
the hidden units have no offset parameter.

The  1  “classifier" with the normal  1=[1121]  is the line given by  1=⋅1=0 .
Similarly, the  2  “classifier" with the normal  2=[1222]  is the line given by  2=⋅2=0.
The arrows labeled  1  and  2  point in the positive directions of the respective normal vectors.
The regions labeled  I,II,III,IV  are the 4 regions defined by these two lines not including the boundaries.

i need to choose the region(s) in (x_1,x_2) space which are mapped into each of the following regions in (f_1,f_2) space,in the 2-dimensional space of hidden unit activations f(z_1) and f(z_2).

is{(f_1, f_2):f_1 > 0,f_2 > 0} in only  region marked 2.is  f_1 axis lies is in regions marked 1 and 2, and is f_2 axis lies is in regions marked 2 and 3, and is the origin (f_1,f_2)=(0,0) lies in regions marked 1,2,3,4


texts: Learn large, variable-size action space for Diplomacy game

texts:
I am making an environment using OpenAI gym for Diplomacy, and making an AI for it.

In Diplomacy, a player has many units, and each unit has a number of moves available to it.

Therefore, the player's action space is the product of each unit's moves, minus the combinations which make no sense.

What I am doing is constructing a list of all available actions for the agent, like so:

(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Burgundy, France: TROOP Paris -> Burgundy)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Burgundy, France: TROOP Paris -> Brest)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Burgundy, France: TROOP Paris -> Picardy)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Burgundy, France: TROOP Paris -> Gascony)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Burgundy, France: TROOP Paris -> Paris)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Burgundy, France: TROOP Paris Supports TROOP Marseilles -> Burgundy)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Gascony, France: TROOP Paris -> Burgundy)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Gascony, France: TROOP Paris -> Brest)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Gascony, France: TROOP Paris -> Picardy)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Gascony, France: TROOP Paris -> Gascony)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Gascony, France: TROOP Paris -> Paris)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Gascony, France: TROOP Paris Supports TROOP Marseilles -> Gascony)
(France: FLEET Brest Coast -> English Channel, France: TROOP Marseilles -> Spain, France: TROOP Paris -> Burgundy)
... (many hundred more)

And making the action space larger than the largest ones of these lists I've ever encountered - it is presently 2024.

Previously, I represented this entire list inside the observation space, but this slowed down the learning substantially - I get orders of magnitude higher fps when I don't represent this list in the observation space.

The agent selects an action, which is just an index of this list.

If the index is too large:

action = action % len(all_possible_actions)

So that the agent always selects a valid action.

My thought is that the agent will learn the structure of this list (the list is a function of the environment, so the agent should be able to reproduce some representation of it).

However, my impression is that forcing the agent to learn how this list is produced, the indexing, and the rules of the game will substantially slow down learning.

Has anybody dealt with a similar problem and solved it? What would be a good way to give "hints" to the network regarding the structure of this list without exploding the observation space?


texts: Validation set performance increased, test set performance decreased

texts:
I am training a CNN model for a three-class classification problem. To do this, I'm gradually unfreezing more convolutional blocks of a pre-trained Resnet-18 network. The thing is that after unfreezing a block (let's say block 3, and preceding blocks), the performance on the validation set did improve but the performance on the test dataset did not improve (as opposed to block 2, and preceding blocks).

I'm now wondering whether or not it is 'justified' (so to speak) to keep unfreezing blocks to see how this affects performance on the test dataset. I feel like I'm just capitalising on some kind of luck that the model fits the test dataset better just by chance.

In short, I guess my question is whether I should choose my models on validation performance or test performance?


texts: What is a discrimination threshold of binary classifier?

texts: With respect to ROC can anyone please tell me what the phrase "discrimination threshold of binary classifier system" means? I know what a binary classifier is.


texts: How to calculate date range constrained average Lifetime Value within Data Studio?

texts:
So within a data studio dashboard, I have a number of metrics e.g. Total Orders, Total Sales, Average order value etc. with YoY indicators as shown below:

And all the metrics are working fine with daterange control (as it would count distinct customer IDs which appeared in this date range as customers which is logically correct as well) except for LTV field. LTV is defined as "Total monetary value of customers averaged over all customers considered". So I calculated it over here as SUM(usd_shipped)/COUNTD(user_id).
So for a specified date range logical definition would be: All time $ Shipped of all customers who placed order(s) in this date range / Number of unique customers who placed orders in this date range.
However the way Data Studio filters work, Its calculating Total $ Shipped during this date range / Number of unique customers who placed orders in this date range which is logically a whole different metric usually called average customer value.
So how can I get LTV value to be calculated correctly using date range. Current work around is to have a separate query and have date range as parameters to calculate LTV via SQL. But how can it be done in Data Studio itself so I can have YoY metrics as well as a simple clean table connection?


texts:
I have a naive question about using the K Nearest Neighbor algorithm: is feature selection more important in KNN than in other algorithms?

If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right? If I'm predicting height based on weight and age and gender, my model will get worse if I now add house numbers, because people will similar house numbers to me will be closer to me.

In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?


texts:
While studying Independent Component Analysis, specifically SVD to be used for the task of image separation. The lines given in the textbook is like this:-
"The first assumption will be that the two images are statistically independent."
Now, Statistical independence means images must have some kind of distribution to start with. But I am unable to grasp the meaning behind this. What does it mean for an image to have a distribution? Like how is a image having Gaussian distribution or Uniform distribution look like. It would be great if some examples are presented.


texts: Why don't we use space filling curves for high-dimensional nearest neighbor search?

texts:
Some space filling curves like the Hilbert Curve are able to map an n-dimensional space to a one dimensional line whilst preserving locality. Does that mean that we could map a dataset of high dimensional points to a line and expect the order of the nearest neighbors to be preserved?

If so, wouldn't that be more efficient than building a Ball tree?


texts: Perceptron - Which step function to choose

texts:
I'm studying Perceptron algorithm. Some books use this step function

  1 if x>=0 else -1

where x is a dot product between the weights w and a sample x.

Other books use:

  1 if x>=0 else 0

What are the practical differences between these step functions?


texts: What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales?

texts:
I have a dataset composed of few features :

customerId, actionDay1, SalesDay1, actionDay20, SalesDay20, actionDay30, SalesDay30

action can be :

call
email
face 2 face
nothing

sales : amount of sales in $

my goal here is to predict the best action we have to take in any day (1, 20 or 30) to sale more.

some customer prefers email, some prefer to be called every time, some others prefer to not be contacted to buy.

the pattern is different for each customer. my first thought was to apply reinforcement learning for this problem, but I found some difficulty to do the right environment, but also the reward function. a reinforcement learning would work without data and the model will be different for each customer. I would like to have a model that can approximate the behavior of all customers using the dataset I have.

does anyone have an idea on how I can approach this problem using reinforcement learning or something different like Recurrent Neural Net?


texts: Neural Network Golf: smallest network for a certain level of performance

texts:
I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance.  By small I mean few parameters, not few arithmetic operations (=fast).

I am interested primarily in convolutional neural networks for vision applications, using something simple like CIFAR-10 without augmentation as the benchmark.  Top-performing networks on CIFAR in recent years have had anywhere between 100 million and 0.7 million parameters (!!), so clearly small size is not (always) a bad thing.

Small networks are also in general faster to train and overfit less.  Moreover, recent work on Knowledge Distillation, FitNets, etc show ways of making smaller networks from large networks while preserving most of the performance.

Another question is, what is the best performance achievable with a network no larger than a fixed size?

Examples of especially small networks that get good performance (100k parameters with 10% on CIFAR, anyone?) or systematic studies of the size vs performance tradeoff would be appreciated.


texts: What is the difference between CountVectorizer token counts and TfidfTransformer with use_idf set to False?

texts:
We can use CountVectorizer to count the number of times a word occurs in a corpus:

# Tokenizing text
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)

If we convert this to a data frame, we can see what the tokens look like:

For example, the 35,780th word of the 3rd document occurs twice.

We can use TfidfTransformer to count the number of times a word occurs in a corpus (only the term frequency and not the inverse) as follows:

from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)

Converting this to a data frame, we get:

We can see the representation is different. The TF is shown as 0.15523. Why is this different than the token count using CountVectorizer?


texts:
I am performing a classification task and was able to identify significant predictors (important features using Random Forest) that can help separate the classes or influence the outcome.

But I read online that prediction models are not causal models.

Let's say if my prediction model says that Age is one of the significant factors that influence outcome (death), how can I prove that Age is the cause of death.

I read that any intervention/change on strong predictors of your models, will not necessarily impact the outcome.

How can I find out the list of factors that really cause change in the outcome?

Currently what I do is run a RF model to identify the important features and communicate that these are the top 5 features that seem to influence the outcome.

How can I prove that it is causation and not just correlation?


texts:
I am trying to build the KNN algorithm for IRIS dataset. First, I've computed the distance and stored it in 1d array. However, I am really struggling to build the prediction function. Therefore two questions arise:

What does this code do? classes = y[np.argsort(dist)][:k]
How should I change the last loop of the function given certain requirements?

def KNN(k, X, y, x):
    from scipy.spatial.distance import cdist
    """K nearest neighbors
    k: number of nearest neighbors
    X: training input locations
    y: training labels
    x: test input
    """
    N, D = X.shape
    num_classes = len(np.unique(y))
    dist = np.zeros(X.shape[0])   # <-- EDIT THIS to compute the pairwise distance matrix
    for i in range(len(dist)):
        dist[i] = np.linalg.norm(X[i]-x)
    print(dist)

    # Next we make the predictions
    ypred = np.zeros(num_classes)
    classes = y[np.argsort(dist)][:k] # find the labels of the k nearest neighbors

    for c in np.unique(classes):
        ypred[c] = y[c]  # <-- EDIT THIS to compute the correct prediction
        print(ypred)
    return np.argmax(ypred)
```


texts:
I am having a time series prediction problem and the data set has 4 variables.
My data set is like below:
import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
df.columns=['var1','var2','var3','var4']

The timestamp is the index of my original data frame and I only take the values in the sorted order.
From this var1 is my target variable to predict.I also need to key in 3-time lags so I prepare the dataset by below function:
# convert series to supervised learning
def series_to_supervised(data, n_in=1, n_out=1, dropnan=True):
    n_vars = 1 if type(data) is list else data.shape[1]
    df = pd.DataFrame(data)
    cols, names = list(), list()
    # input sequence (t-n, ... t-1)
    for i in range(n_in, 0, -1):
        cols.append(df.shift(i))
        names += [('var%d(t-%d)' % (j+1, i)) for j in range(n_vars)]
    # forecast sequence (t, t+1, ... t+n)
    for i in range(0, n_out):
        cols.append(df.shift(-i))
        if i == 0:
            names += [('var%d(t)' % (j+1)) for j in range(n_vars)]
        else:
            names += [('var%d(t+%d)' % (j+1, i)) for j in range(n_vars)]
    # put it all together
    agg = pd.concat(cols, axis=1)
    agg.columns = names
    # drop rows with NaN values
    if dropnan:
        agg.dropna(inplace=True)
    return agg

#take the 3 lags for each row
reframed = series_to_supervised(df,3, 1)
train_X=reframed.drop(['var1(t)'],axis=1)
train_y=reframed[['var1(t)']]

# design network
model = Sequential()
model.add(LSTM(20, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

So here the shape will be (97, 16) where 97 is the row count since the first three timesteps will not have 3 lags and the 16 is the overall columns with 3 lags plus the current time stamp.
Now in my case the value of var1(t) is also influenced by the var2(t), var3(t),var4(t).
So my training shape of the data will be
   train_X shape  (97, 15)
   train_y shape  (97, 1)

Now the problem is how to reshape this into [samples, timesteps, features].Because I can only do this if I drop var2(t), var3(t), var4(t) then I can reshape as [97,3,4] but in my case, I also want to include the current timestamp's remaining variable.
So then I found out that I can use the keras functional API to overcome this .
I created the model with keras functional API but something is wrong in the shape of the tensors.
def createFunctionalKerasLSTM():
    visible = Input(shape=(train_X.shape[0],train_X.shape[1]))
    hidden1 = LSTM(20)(visible)
    hidden2 = Dense(10, activation='relu')(hidden1)
    output = Dense(1, activation='sigmoid')(hidden2)
    model = Model(inputs=visible, outputs=output)
    model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mean_squared_error'])
    return model

Error

ValueError: Input 0 of layer lstm_14 is incompatible with the layer:
expected ndim=3, found ndim=2. Full shape received: [None, 15]

I am not sure how to reshape in this case .because for normal LSTM I can drop the var2(t), var3(t), var4(t) and then reshape it (97,12) into (97,3,4) But here y shape is (97,15) so a bit confused how to reshape it properly to the input of the network.
Any help is appreciated!


texts: How can you make use of Json format data?

texts: I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an excel format. Thanks for your time.


texts: how to compare different sets of time series data

texts:
I am trying to do some anomaly detection between time#series using Python and sklearn (but other package suggestions are definitely welcome!).

I have a set of 10 time-series; each time-series consists of data collected from torque value of a tire (so 10 tires in total) and the sets may not contain same number of data points (set size differ). Each time-series data is pretty much just the tire_id, timestamp, and the sig_value (value from the signal, or the sensor). Sample data for one time-series looks like this:

tire_id        timestamp        sig_value
tire_1           23:06.1            12.75
tire_1           23:07.5                0
tire_1           23:09.0            -10.5

Now I have 10 of them, and 2 of them behave strangely. I understand that this is an anomaly detection problem, but most of the articles I read online are detecting anomaly points within the same time-series (aka if at some points the torque values are not normal for that tire).

To detect which 2 tires are behaving abnormally, I tried using clustering method, basically k-means clustering (since it's unsupervised).

To prepare the data to feed into the k-means clustering, for each time-series (aka for each tire), I calculated:

The top 3 sets of adjacent local maximum and local minimum with
highest amplitude (difference)
Mean of torque value
Standard Deviation of torque values

I also set the number of clusters to be only 2, so either cluster 1 or 2.

So my end result (after assigning clusters) looks like following:

        amplitude  local maxima  local minima  sig_value_std  \
tire_0     558.50        437.75       -120.75      77.538645
tire_0     532.75        433.75        -99.00      77.538645
tire_0     526.25        438.00        -88.25      77.538645
tire_1     552.50       -116.50        436.00      71.125912
tire_1     542.75        439.25       -103.50      71.125912

        sig_value_average  cluster
tire_0          12.816990        0
tire_0          12.816990        0
tire_0          12.816990        0
tire_1          11.588038        1
tire_1          11.588038        0

Now I have a question of what to do with this result ... so each tire has 3 rows of data, as I've picked the top 3 pairs of local max/min with 3 largest amplitudes, and that means each row can be assigned to a cluster, and sometimes they are assigned to different clusters for 1 tire even. Also the cluster size is normally larger than just 2.

My questions are:

How to do anomaly detection about "set of time-series" not just individual
data points?
Is my approach reasonable/logical? If it is, how can I clean up my result to get what I want? And if not, what can I do to improve?


texts: Numbers of participants in crowdsourcing tasks

texts: Within the recent years, leveraging crowdsourcing for performing, mainly intelligent-intensive, tasks has become prevalent. An important question in this regard, that I could not find any specific answer for, is how many participants are adequate for running such a project? Is there any standard for that?


texts:
I am doing a toy example with mushroom dataset to learn class embedding with keras:

I am trying to embed a single feature:

inputs = Input(shape=[1])
emb = Embedding(input_dim=cap_shape_dummy.shape[1], output_dim=3)(inputs)
output = Dense(units=1,activation='sigmoid')(emb)
model = Model(inputs=inputs,outputs=output)

However I always receive the same error when I try to fit the model:

Error when checking target: expected dense_2 to have 3 dimensions, but got array with shape (8124, 1)

Am I doing anything wrong? Did I miss anything with category embedding?
Thanks.


texts: Possible to correct an actual cell state in LSTM via gradient?

texts:
Why in LSTM we calculate gradient w.r.t weights, but not w.r.t the cell state?

Is it theoretically possible to correct the contents of the cell state, and what would it result in?

I understand that weights are like a "set of skills", so that network can respond correctly to the input, even gazillions of iterations later. The cell-state is an understanding of what's going on in the past, up to the start of the current minibatch.

So why not to correct the value stored in the cell state? It would be very useful if we carry the cell state forward, between minibatches.

https://stackoverflow.com/a/44183738/9007125]

  Generally it is recommended to reset state after each epoch, as the state may grow for too long and become unstable. However in my experience with small size datasets (20,000- 40,000 samples) resetting or not resetting the state after an epoch does not make much of a difference to the end result. For bigger datasets it may make a difference.


texts: Using SVM Classifier with C=0 and C=infinity ,what would be the effect on classifying this data?

texts: I want to know the effect of applying different C values on SVM Classifier margin


texts: the library 'transformers' works also with older version of Tensorflow?

texts:
i am working with Tensorflow version 1.14 and i would like to use the bert embedding.
In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/)  but i am not sure if that will work with my tensorflow version.