texts: How to calculate VIF in python 3.6 with scipy 1.0.0?

texts:
I am trying to import

from statsmodels.stats.outliers_influence import variance_inflation_factor

This is working fine upto Scipy 0.19.

But , with Python 3.6.3 ,it's failing due to unavailability of ss module in Scipy 1.0.0.

~\Anaconda3\lib\site-packages\statsmodels\regression\linear_model.py in <module>()
     41 from scipy.linalg import toeplitz
     42 from scipy import stats
---> 43 from scipy.stats.stats import ss
     44 from scipy import optimize
     45 from scipy.stats import chi2

ImportError: cannot import name 'ss'

Can anyone please guide me how to do VIF analysis using Scipy1.0.0 ?


texts: Why do convolutional neural networks work?

texts:
I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated features as we go up the layers?
What caused them create such a stack of features and would this also be true for other types of deep neural networks?


texts: Data augmentation: rotating images and zero values

texts: A lot of people rotate images to create a larger training set for neural networks. For most nets, all of the inputs have to be the same size so the image rotation function has to crop the newly rotated images to match the input size. So, say you have $32\times32$ resolution images and do 45 degree rotations. Some of the output images will look diamond shaped with black (zero values) in the corners. So, my question is: should you leave these zero values alone or change them in some way and if so, how?


texts: How effective is Genetic Algorithm for finding Attribute-Value relationships

texts:
I am implementing a module which finds based on user interactions on an online portal to find which attributes of a certain product and what values for those products influence the buyer to choose the said Item.

My employers have suggested me to use Genetic Algorithm to achieve this. I have started implementing and found even the initial population generation to be a bit ineffective. Before I go further with this I would like your thoughts whether 'Genetic Algorithm' is the right way or is there a better way to go about this.

For me each genome is a list of attributes and their corresponding values.

Eg. genome 1: Color: Black, Material: wool, thread count:5
     genome 2: Color: white, Handwash: yes, Dry clean only: No

Its a mixture of different attributes and their values. Am I going about this the wrong way?


texts: How to approach Peak picking with a wide range of peak shapes, sizes, varying noise level, and occasionally shifting baseline?

texts:
I am trying write a program that continuously tracks the location a peak. To do that I need a very good peak detection algorithm. It not only has to tell the location of the peak but also the absence of it.

My current approach for peak detection:

Baseline subtracted (account for different baseline signals)
Divided by standard deviation of the smoothed signal (account for different baseline signals)
I superimposed all the candidate peak data from all the spectra (local minima after gaussian smooth) and did a gaussian mixture in one dimension (y-axis; 2 clusters;100 spectra;3000 data points)

Yellow - "peaks" cluster; blue - "noise" cluster.

Trouble region: between 1 and 3 in the Y (cannot distinguish shallow peaks from spectra of with no peak).

How do I approach this problem to make the peak detection sensitive enough but still can tell if the peak is not there? Is there any better way to normalize the data? What unsupervised learning algorithm would be suitable for this(my data is unlabeled)? Please let me know if you have any suggestion. It will be greatly appreciated.

Examples:

Peak shapes and sizes, and noise level vary greatly:

Very shallow and broad

Very narrow and sharp

Frequency response went outside of the sweep window (need to identify that the peak is absent)

shifting baseline sometimes

How the data is obtained:

A peak can be thought of as a response of a harmonic oscillator in a frequency sweep.

One example of a harmonic oscillator is a swing. (or a ball on a spring; a ball in a well). They are all characterized by the periodic motion related to a natural position. In the swing example, the natural position is at the middle when the girl first got on the swing.

Imagine a person started periodically pushing her on the back as she swings, her swinging motion might increase. However, if the person pushes her at the wrong time, like pushing her back when her back is coming towards him, she will slow down and will not swing higher. This is because the frequency of the push (driving frequency) does not match the periodic movement of the swing (natural frequency). Imagine the person gradually increases the frequency of the push from really infrequent to really frequent (frequency sweep), the frequency of the push will at some point match the natural frequency of the swing (coming into resonance). In other words, every push of the person will be at the right time and will push her higher and higher.

Imagine you shine a very bright spot light at the natural position of the swing (the lowest position). The amount of time the girl spends inside of the spot light will depend on how high she swings. So the higher she swings, the less time she will spend in the light. Back to my plot, the signal (y axis) depends on how far the harmonic oscillator swings and the dip of the plot tells me at what driving frequency it comes into resonance.(driving frequency matches with natural frequency so it swings very far and decreases the signal)

A shifting baseline is due to a drift in the spot light brightness itself

I hope that would clarify things a bit.


texts: Change rate of cross validation data, after training

texts:
Say we have N of labeled data, and we need to take some part for the cross validation (we will skip test part for this case). We chose, 0.6 part for the training and 0.4 for validation.

After training neural Network with early stop, we have found 8 epochs, as optimal to stop, and have received good enough results.

Q. In case, we have very limited N training samples. May we use all samples in new model training, and just stop it's training after discovered epochs? Without separating it to train and cross validation, and testing it, at all (or even, change rate of separating, to 0.9 train, 0.1 cross validation).

Maybe there is known technologies for such cases? Thanks.


texts: Partial derviative of prediction (sigmoid applied) with respect to weight

texts:
I am very confused as to where a seemingly "extra" term is included in the above mentioned calculation in my Udacity course.

The above is taking the derivative of a sigmoid so why isn't it just

$$=\sigma(Wx+b)(1-\sigma(Wx+b)$$
but rather has $\frac{\partial}{\partial w_j}(Wx+b)$ tacked on the tail?


texts: Heuristics, methods to speed up searches over subsets of big set (combinatorially NP hard probably)

texts:
I have a reasonable-sized set of size N (say 10 000 objects) in which I am searching for groups of compatible elements. Meaning that I have a function y = f(x_1, x_2, x_3, ..., x_n) returning bool 0/1 answer whether n elements are compatible. We are interested in executing this search on each subset smaller than 8 elements of N.
Which is obviously either NP hard or close to this.
Even for pairwise search for n elements set we have n^2 pairs to explore, which is somehow doable yet still grows quite fast for big sets.
Yet then it goes nasty when we are looking for triples (identifying 3-element cliques) and quadriple or greater magnitudes.

Would you suggest any machine learning approach to help with this task? Maybe at efficiently sampling the search-space and identifying which subsets to search on, maybe fast heauristic evaluations of the function to direct the searches.


texts:
I have a question regarding the number of filters in a convolutional Autoencoder.
As far as I have understood, as the network gets deeper, the amount of filters in the convolutional layer increases.
A typical pattern would be to $16, 32, 64, 128, 256, 512 ...$

Right now I am looking into Autoencoders and on the Keras Blog I noticed that they do it the other way around.
They start with $16$ filters in the first layer, then the number of filters is decreased:

https://blog.keras.io/building-autoencoders-in-keras.html

Now I am wondering if this is usual when working with Autoencoders or does it depend on what kind of features the network should learn?

Thanks in advance,

Cheers,

Michael


texts: How to give name to topics created using LDA?

texts:
I have categorized 800,000 documents into 500 categories using the Mahout topic modelling.

Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name for the group using any existing algorithm.
For the time being, I have used the following algorithm to arrive at the name for the topic:

For each topic

Take all the documents belonging to the topic (using the document-topic distribution output)
Run python nltk to get the noun phrases
Create the TF file from the output
name for the topic is the phrase (limited towards max 5 words)

Please suggest a approach to arrive at more relevant name for the topics.


texts: How do Bayesian methods in machine learning help with the problem of limited data? Can this be used for image classification?

texts:
When reading about machine learning, I've often come across information stating that Bayesian methods in machine learning are effective when you only possess a limited amount of data. As someone who is planning to start learning machine learning, I was curious how it accomplishes this? Lastly, and more specifically, I was especially wondering how and/or if this can be applied to problems such as image classification, when there is a limited amount of data (limited number of images to train the algorithm on)?
Any textbook, research, or other references with regards to the latter would also be greatly appreciated.
Thanks.


texts: Gradient Boosted Trees or Neural Networks Using Model Averaging?

texts:
I am working on a certain insurance claims related data-set to classify newly acquired customers as either claim or non-claim.

The basic problem with the training set is the extremely large imbalance in claim and non-claim profiles, with the claims amounting to just ~ 0.26% of the training set. Also, most claims are concentrated largely towards the final few years (data is sorted by date).

On applying Logistic Regression or even Random Forests, to train on 70% of the data, the test results were well below satisfactory.

I've been looking at alternate models and I came across this blog post. A particular line that got my attention is:

  GBM is better than rf_t. In the paper, the best classifier for two-class data sets was avNNet_t, with 83.0% accuracy

Although, no real clarification was given as to why that was. Can someone help me open this "blackbox"? Which model really works (in the case described above) and why?


texts: Understanding Transformer's Self attention calculations

texts:
I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-framework#comment-160771

What is the value of Key, Value in the self attention calculation of Transformer model?
Query vector is embedding vector for the word that is queried, is that right?
Is attention calculated in RNN is different from self attention in Transformer?


texts: I've seen that in Kaggle competitions people are using lightgbms where they used to use xgboost. My question is: when would you rather use xgboost instead of lightgbm? What about catboost?


texts: How to use a one-hot encoded nominal feature in a classifier in Scikit Learn?

texts:
I'm working on a genre classification problem on a songs dataset. Since genre is a nominal feature, I used sklearn's LabelBinarizer to get the one-hot encoding for this feature for every row in the dataset. I'm then left with a dataframe(df_train_num) with two columns, both numeric in nature and a Series object for which every row value is a numpy array - the one-hot encoding of the genre. I now want to fit a classifier on this data. What I did was:

svm_classifier = LinearSVC()
svm_classifier.fit(df_train_num,df_train_genre)

This gives me:

  ValueError: Unknown label type: 'unknown'

What exactly is causing this error? Am I not allowed to use a Series object with a DataFrame object in the to fit a classifier? Although replacing df_train_genre with df_train_genre.values so as to pass the numpy array directly to the fit method also doesn't change anything. Same error.

Here is a view of the two pandas objects:

df_train_num.head(5)

Unique_Word_Count   Sentiment Polarity
157277                  126   0.027766
90109                   114  -0.199545
106224                  16    0.000000
221087                  103  -0.058025
247082                  409  -0.170143

df_train_genre.head(5)

157277    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
90109     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, ...
106224    [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
221087    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
247082    [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
Name: Genre_Encoded, dtype: object


texts: Types of maps in Interpretable Machine Learning

texts:
I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused.
For example, saliency maps/heat maps. Are they same?
Are activation maps visualizations of activation values of a unit(s) in Neural networks? Do you know any other types of maps?
Thank you.


texts:
My current tensorflow version is 2.1 and I'm using low level(core) tensorflow api's.
Please provide a substitute of "tf.contrib.layers.embed_sequence".
I have explored a lot, but could not find any relevant api.
Also tf2.3 API "tf.keras.layers.Embedding" can be used for sequential model only and not with core API's.


texts:
We are training the BERT model on masked language modeling task for the Russian Language. Our dataset consists of 60 mln texts with (128 tokens for each text) from online social networks, predominantly in the Russian language. We have not performed any text preprocessing. We use AdamW optimizer and RuBERT as a pre-trained model. The training parameters are the following:
Learning rate: 1**e-5
Warmup: 30 000
Seq. len: 128

The charts below show the cross-entropy(CE) loss function for test evaluation as iterations go on. One iteration contains 100 batches, One epoch contains ~ 3500 iterations. The blue chart shows loss function when we mask only Cyrillic tokens (Cyrillic BERT). The orange chart shows loss function when any token (including punctuation, URLS, numerals, English letters) can be masked and it is the only difference among the two charts (Default BERT).

My questions are the following:

Why "Cyrillic BERT" overfits dramatically after the ~2200 iteration while the "Default BERT" oscillates near 1.67 at the same time? What Bert config parameters may change it?
Why The "Default BERT" has no improvements does not improve after the ~3000 iterations? How can I improve it? (already trying to change hidden_dropout_prob to 0.2 from 0.1)
Does "Cyrillic BERT" shows worse loss function because it is easier to predict punctuation and several English names?


texts: Encoding categorical data with pre-determined dictionary

texts:
in case feature encoding, if I'd like to encode my values based on my pre-determined dictionary, how do I do that?
For instance, say, I've values as [Red, Green, and Blue] and I want to encode them as [-1,0,1]  -1 for red, 0 for Green, 1 for Blue... I'll apply it to my feature. I believe I can do it by mapping, apply method, not sure. But is there any better way to do that?
Column     expectedEncoding
Red             -1
Red             -1
Blue             1
Green            0
Red             -1
Blue             1

```


texts: Can a Conditional GAN be used for a regression task, where the condition can take any continuous value?

texts: I came across conditional GAN, where the classes can be provided as input to the GAN. I would like to know, if the same network can be used for classes which can take any continuous value.ie, the generator once trained should be able to produce output for any condition, even if it was not a part of training set.


texts: How to Extract Information from the Image(PNG)

texts:
I'm trying to extract some particular information from the image(png).

I tried to extract the text using the below code

import cv2
import pytesseract
import os
from PIL import Image
import sys

def get_string(img_path):
    # Read image with opencv
    img = cv2.imread(img_path)

    # Convert to gray
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    # Apply dilation and erosion to remove some noise
    kernel = np.ones((1, 1), np.uint8)
    img = cv2.dilate(img, kernel, iterations=1)
    img = cv2.erode(img, kernel, iterations=1)

    # Write the image after apply opencv to do some ...
    cv2.imwrite("thres.png", img)
    # Recognize text with tesseract for python
    result = pytesseract.image_to_string(Image.open("invoice.png"))
    os.remove("invoice.png")

    return result

if __name__ == '__main__':
    from sys import argv

    if len(argv)<2:
        print("Usage: python image-to-text.py relative-filepath")
    else:
        print('--- Start recognize text from image ---')
        for i in range(1,len(argv)):
            print(argv[i])
            print(get_string(argv[i]))
            print()
            print()

        print('------ Done -------')

But I want to extract data from particular fields.

Such as

 a) INVOICE NO.
 b) CUSTOMER NO.
 c) SUBTOTAL
 d) TOTAL
 e) DATE

How can I extract the required information from the below image "invoice"?

PFB


texts: Offline evaluation of recommender systems

texts:
Let's say I want to compare whether one recommender system (A) is better than the other (B).

One approach is to let people rate recommendations returned by both systems.

However, there situations when I would like to evaluate my recommender system offline. One approach I considered is to collect user ratings for recommendations from system A and turn them into a test dataset. However, if the system B returns recommendations that do not appear in test dataset, that does not mean they are bad. It just means I have no rating about them.

Some alternatives:

try to increase test dataset volume as much as possible to increase overlap between recommenders.
limit the size of items space, let recommender select items from 500 instead of 5*10^6 items during tests.

What other approaches would you recommend suggest?


texts: Train building classifier for imagerial data

texts:
I am using the https://www.arcgis.com/ api for accessing imagery of aerial data.

I would like to train a model that can caputure on the imagery if the provided street address has a house, pool, garage etc.?

Any suggestions how to start such a project? Any recommendations for similar implementations?

Looking forward to your replies!


texts: Is my data appropriate for Hierarchical Clustering?

texts:
I am a newbie in clustering and trying to check whether there are differences in Symptoms (example: cough, sneezing, shortness of breath, etc) reported across different comorbidity groups ( obesity, asthma, etc).
My data is in two formats:

original one - with 53.000 rows
and aggregate one obtained from the 53.000 k rows data - frequency and percentages of symptoms across comorbidities - which leads to an aggregate data of 9 rows of comorbidities, and 17 rows of symptoms (I have chosen the percentage instead of counts).

I have already done some hierarchical clustering on the aggregate level - yet I am wondering whether this is okay? I have chosen to do hierarchical clustering on 5000 rows only, but very computational expensive.
Thus in summary my questions are:

is aggregate level data (2nd type of data outlined above) okay for clustering? I have actually 4 clusters of comorbidities given by aglomerative one.  And hope the choice here is good one.

as I have done some clustering on original data, not sure why this is so slow when run on 10.000 let alone 53.000. I have therefore decreased it to 5,000 as this was the best choice for me. is this behaviour on 53.000 rows okay?

Thank you in advance for your help.


texts: Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model

texts:
I am currently using transaction amount as a feature in an XGBoost classification model designed to identify fraudulent transactions. Furthermore, transaction amount is bounded for this problem between 0 and 500. Using transaction amount as a feature does improve target class separability. However, I can't help but wonder if there is a better way to use this variable. To explain, I care more about getting the high transaction amount values correct than I do the low ones. However, the model does not currently comprehend that. I have taught the XGBoost algorithm that the positive class is in effect more important by adjusting scale_pos_weight, but I haven't thought of a way to teach the algorithm that high transaction amount values are more important.
EDIT: I wanted to provide a bit more detail. After some additional reading, I think what I may be looking for is some kind of custom objective function. Possibly something like what is being discussed here.


texts:
I am trying to do a multi output regression using tensorflow. I have got a dataset in Excel which includes a column of input points and 2 columns of output.

I coverted all numbers to numpy objects. And I am trying to do a basic regression but accuracy is always 1.0, I also want to draw graph but dunno where to start. Could Anyone please help? My code is here:

import os
import numpy as np
import tensorflow as tf
import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from keras.models import Sequential
from keras.layers import Dense
from keras.activations import relu

training_data = pd.read_excel(r'C:\Users\lenovo\Desktop\Yeni klasör\training_data.xlsx',sheet_name="i1-o2")
training_data_X = training_data['i1']
traindataX = np.array(training_data_X)
training_data_Y = training_data[['o1','o2']]
traindataY = np.array(training_data_Y)
testing_data = data = pd.read_excel(r'C:\Users\lenovo\Desktop\Yeni klasör\testing_data.xlsx',sheet_name="i1-o2")
testing_data_X = testing_data['i1']
testing_data_Y = testing_data[['o1','o2']]
testdataX = np.array(testing_data_X)
testdataY = np.array(testing_data_Y)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128,activation=tf.nn.tanh))
model.add(tf.keras.layers.Dense(128,activation=tf.nn.tanh))
model.add(tf.keras.layers.Dense(2))

model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])
model.fit(traindataX,traindataY,epochs=500)

val_loss,val_acc = model.evaluate(testdataX,testdataY)
print(val_loss,val_acc)


texts: How is y=mx+b different from hθ(x)=θ0+θ1x?

texts:
I could not quite comprehend the hypothesis represented by  hθ(x)=θ0+θ1x
To find out good values for the parameters θ0 and θ1 we want to minimize the difference between the calculated result and the actual result of our test data. So we subtract hθ(x(i))−y(i) for all i from 1 to m. Hence we calculate the sum over this difference and then calculate the average by multiplying the sum by 1/2m.  This would result in:

1/2m∑mi=1(hθ(x(i))−y(i))2

So, I googled further and landed on a youtube video that talked about
y=mx+b, where m is the slope and b is y intercept. This is called a Linear Model.

Also, in the linear model, the following formula is used to determine m and b.

Now my questions are:

1) Is the hypotheses and the linear model the same?
2) Is there a cost function for the linear model?
3) Why would anyone want to guess, choose and arrive at θ0 and θ1 when there is a straight forward formula (linear model)?


texts: Stata: Trying to merge two data sets with common year observations

texts:
I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "cpilevel". The other is our main dataset, and we are trying to merge the CPI-year dataset in order to create a new "cpilevel" variable in the main dataset associated with the same years. I am attaching photos of the CPI-specific dataset and a cross-section of the main dataset (there are far too many observations to capture the whole set). Please let me know the easiest way to merge these datasets; we have tried using m:1 codes but can't seem to get the right combination.

::EDIT::

Here are the results of the merge command. It did not accomplish what we need in terms of merging the "cpilvl" variable with the "year" variable in each of the main observations, but it seems to be on the right track?


texts: how to retrain model with periodic new features?

texts:
I've trained a gradient boosting classification model. But, suppose i've a set of fixed features F1,F2....Fn and new features which are added weekly (no. of actions done in that week). So, after 2 weeks dataset to be trained on is :

   Fixed           Dynamic
F1 ,F2 .....Fn    W1 ,W2

After 3 weeks

  Fixed           Dynamic
F1 ,F2 .....Fn    W1 ,W2, W3

How do we approach this problem on production server, is there any approach available which allow model to be retrained on new features and not only on new observations ?


texts:
I would like to label character data with BIO tags as part of an active learning process on unlabelled data. I am assuming there are open source GUI tools available which I can use to make this easier - i.e. present the string to be labeled and some way of tagging characters from a predefined set of tags (and probably allow new tags to be added).
I have not been able to find anything though - ideally cross-platform (Linux and macOS at a minimum). It could be a cloud service or AWS product but I'd prefer something which is locally installed.
Does anyone know of such a thing? Even something as simple as search term would help.


texts:
I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word from the context given. The loss function will be the lowest for output embeddings which are closest to the original masked word embedding. Wouldn't it be that using this loss funtion does not guarantee that BERT will output word embeddings with context and instead could just output an embedding closest to the original masked word embedding without the relevant context?
Thanks.


texts: How to convert trained data (feature extracted) into a prediction model?

texts:
Background:

I am analyzing and labeling some log data. (parsed already, sample data below)
I have extracted the major features of data.

For examples, the classification results ("1 - normal" or "0 - anomaly") largely depends on columns "duration", "mean", "std". For example,

records/sec usually <= 10
std usually < 0.5   , etc

Then I want to convert current result into a prediction model, in order to classify the future log data.

Sample data:

| ID |        datetime       | No. of Records | duration(s) |  mean | std  |  labels |
| 1  |  26/7/2019 8:06:00 PM |       5        |     1.0     |  0.33 | 0.47 |  normal |
| 2  |  ...                                                                | anomaly |
...
| 1,000,000 |  ...                                                         |  normal |

Question:

How to convert such parsed & feature extracted & labelled data into a prediction mode (or prediction function)?

I am NOT asking the theory. I have read some articles or projects of machine learning models. But many of them introduces too high-level. I still don't understand how to implement those machine learning models. I need to convert them into a real prediction function running in the computer.

I am asking the detailed / hands-on implementation steps. Maybe it becomes a call function f(x1, x2, x3, ...) or API in the code? The new data may be the input of a function call? The output will be normal or anomaly in this case.
In other words, how to present a prediction model in the form of code?


texts: About the relevance and interprertability of convolutional filters?

texts:
Convolution filters are known to perform very well in tasks, concerning some work with the image or video data, due to their ability to preserve some spatial information and equivariance property under the translations.
Interpretability of CNN filters
The idea of convolutional filter on a high level, that it is in some sense a switch, which reacts on a certain pattern, present on the image. For example this filter
$$
\begin{pmatrix}
1 & -2 & 1 \\
1 & -2 & 1 \\
1 & -2 & 1 \\
\end{pmatrix}
$$
can select horizontal borders on the image. And lower level filters activate on more sophisticated features and concepts like the presence of mouth or ears on the image.
However, initially these convolutions are initialized randomly, and only during the training procedure they capture properties of the images, that are transmitted through the network, and in case the network is trained well enough, these filters select the useful features on the image.
However, it is common, that for large and deep enough networks there are a lot of channels (hundreds or even thousands), and I wonder - how much are all of these filters relevant? The training process seems to be a bit arbitrary - for example the permutation of filters order won't in fact change the result, provided the subsequent layers action is permuted in the same way.
During the training, can one figure out, that some filters are useless and drop them? And is there a point in insertion a new filters, hoping that they can be more useful, that those dropped?
The pass through the convolutional layers can be thought as some transformation of the initial image, which makes a richer and more useful representation, than the naive pixel map, because each pixel is hard to interpret, and the meaning is carried by the image as whole.
Output of the convolutional layers as richer representation
Is it the case, that in the initial formulation it is rather difficult to give a precise mathematical meaning, whether we have a cat or a dog on the image, but after the pass through the deep network, dogs, for instance, map to one cluster, and cats map to another cluster, and the data is linearly separable (or not linearly, but in some simple way) in this representation?
And if this representation is simple to split enough, may it be the case, that instead of fully connected neural network classifier on the bottom layer, the gradient boosting or SVM classifier may perform better?


texts: Salesman problem with additional conditions and features

texts:
I'd like to specify the kind of a problem I encountered. I need to compose the best route for a car driver who goes to different cities.
His aim to check in the most proper car park in a city taking into account arrival and departure dates and time, distance to a parking lot, traffic, a car type and cars density at a parking lot. The driver can travel through cities in random order.
I know it's an optimization problem and it also resembles the salesman and the transportation problems. But also it looks like a recommendation task. I'm confused about it. I want to use the gradient boosting or NN's to solve it, but I'm stumped by the strict mathematical statement of the problem in the mathematical language. Any clarifications, references to similar works or blogs are desired.


texts: How to read specific time for specific value row by row using python


texts:
I am not sure whether I formulated the question correctly. Basically, what I want to do is:

Let's suppose I have a list of 1000 strings which look like this:

cvzxcvzxstringcvzcxvz

otortorotrstringgrptprt

vmvmvmeopstring2vmrprp

vccermpqpstring2rowerm

proorororstring3potrprt

mprto2435string3famerpaer

etc.

I'd like to extract these reoccuring strings that occur on the list. What solution should I use? Does anyone know about algorithm that could do this?


texts: Is there a clustering algorithm which accepts some clusters as input and outputs some more clusters?

texts:
Heres the task: I have data I don't know much about. The final task is to build a classifier to classify the samples into a few categories. Some of the categories are pretty clear, we can easily use these as labels for a classifier. But I guess there are more useful categories possible, because right now most of my samples don't belong to any category. As I am no expert in the specific field, I would like to use some clustering algorithm to show possible label ideas.
When using traditional clustering algorithms, they find all sorts of patterns in the data I am not interested in.
So I am looking for a way to tell the algorithm: "Hey, find some clusters in my data, but please take the existing clusters (or labeled data) into account." This should tell the clustering algorithm what I am interested in, and in what not.
Does something like this exists? Or any other idea how to solve the problem of finding additional labels?
BTW: in my case, I am doing NLP.


texts: how to determine hashing bit length for multiple categorical features?

texts: Say I have $N$ categorical features $f_i$ $i\in(1,N)$ each of which of different alphabet size $n_i$. How can I efficiently optimize the hashing trick on that feature vector? Should I enumerate hash bit length for each feature independently (greedy)? Should i assume no hashing is necessary for features with small alphabet ($n_i$ is small)? What is the best practice strategy?


texts: Can we learn f(x)=1/x using a neural network exactly?

texts: Is there a way to train a neural network as $f(x) = {1 \over x}$ precisely?


texts: Could you generate search queries to poison data analysis by a search engine?

texts:
A simple problem with search engines is that you have to trust that they will not build a profile of search queries you submit. (Without Tor or e.g. homomorphic encryption, that is.)
Suppose we put together a search engine server with a use policy that permits constant queries being sent by paid customers.
The search engine's client transmits, at some frequency, generated search queries (e.g. markov, ML-generated, random dictionary words, sourced from news, whatever; up to you) in order to intentionally obscure the real search queries performed by customers. In other words it pretends to be a thousand contradictory personalities, nationalities, genders, races, hobbies, etc.
How difficult would it be to generate enough queries to hide yourself in the data?


texts: Explaining what a neural network is learning

texts:
In more traditional statistical learning methods, such as logistic regression, the coefficients on the pre-selected features can be interpreted as increasing or decreasing the log-odds of success. For example, say we want to use logistic regression with a single predictor variable studying_hours to predict whether a student will pass the exam. We can estimate the parameters in the model:

$log(Y) = \beta_0 + \beta_1$studying_hours

where $Y = 1$ if the student passes the exam and
$Y = 0$ otherwise, and

studying_hours is a variable with values 1, 2, ..., 100 hours.

We can say that a one-unit increase (a one hour increase) in the number of hours studying for the exam increases the log-odds of passing the exam by $\beta_1$.

It stands to reason that we should be able to make a similar argument when the features are learned instead of pre-selected. But then we would need to understand exactly what the learned feature is. In my example above, instead of pre-selecting studying_hours to be the feature we want to use to predict whether a student passes the course, we would learn some informative feature instead.

I've heard of Hinton diagrams, and of visualizing the feature map for CNNs, but I don't think either of those techniques really works here. A Hinton diagram would just tell me the magnitude and sign of the weights. A feature map would not really be informative. I don't expect to understand what all of the features are, because the whole point is that the algorithm can do a better job of learning features than I can do designing features. But even so, at least some of the features should be human-intepretable, like a 'straight line detector' in a CNN.

How can we interpret the features being learned by a multilayer perceptron for logistic regression?

Not really sure what to tag here, so please, if there's an edit you would recommend, go ahead!

EDIT

This is my attempt at starting an answer, but clarification/other opinions would be appreciated.

Let's take a case of binary classification, and start from the simplest possible case: one scalar input, one hidden unit with a relu activation, and one output unit. Assume the hidden unit has a bias of 0. We can express the function that describes this network as:

$log(Y) = \beta_0 + \beta_1 max\{wx, 0\}$,

where

$\beta_0$ is the bias on the output unit
$\beta_1$ is the weight on the output unit
$w$ is the weight from the input to the hidden unit

Then we can expand that function to:

$log(Y) = \beta_0 + \beta_1wx$, if $|wx| > 0$

$log(Y) = \beta_0$ otherwise.

Which means that if $|wx| > 0$, then the hidden unit is giving us a factor by which to multiply the slope of the line separating the two classes. We can say that a one-unit increase in $\beta_1 w$ implies a one-unit increase in the log-odds of success.

If $|wx| <= 0$, then the best separating line between the two classes has 0 slope, regardless of the magnitude of the weight (I think this is related to the dying relu problem).

If we expand this to include a bias on the output, we have:

$log(Y) = \beta_0 + \beta_1(wx + b)$, if $|wx + b| > 0$

$log(Y) = \beta_0$ otherwise.

Through arithmetic, that first part can be expressed as:

$log(Y) = \beta_0 + \beta_1b + \beta_1wx$, if $|wx + b| > 0$

which means to me that the weight $w$ from the input to the hidden unit tells us both how to modify the y-intercept (which is now $\beta_0 + \beta_1b$) and how to modify the slope.

Does that make sense?


texts:
I have a response variable series which will be generated randomly in a fixed interval [0-100] base on every second,
and I want to detect the event when the new generated data is significantly greater than data of last second, and send alarm message to me.

So, I calculate the difference of response variable by 1 lag and divided by difference of time (slope),
than use bootstrapping to construct the 95% confidence interval of response's 90% percentile, if the new data is greater than Upper Limit, I define it as unusual.

In fact, the quantity of data I have is extremely large, and these data update frequently.
So, it will be horrible to resampling and calculate the CI, even it could be a way to solve problem.
And it's looks like not necessarily to sampling because I have large sample.

Are there any better way to do? Thanks!


texts:
I am trying to learn scikit-learn neuralnetwork and am coming up against the same problem in regression where no matter the dataset I getting a horizontal straight line for my fit.

here is an example using the Linear regression example from scikit-learn and then using the SKNN regressor , simple example code from the docs.

# -*- coding: utf-8 -*-

# Code source: Jaques Grobler
# http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html
# License: BSD 3 clause

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model

# Load the diabetes dataset
diabetes = datasets.load_diabetes()

# Use only one feature
diabetes_X = diabetes.data[:, np.newaxis]
diabetes_X_temp = diabetes_X[:, :, 2]

# Split the data into training/testing sets
diabetes_X_train = diabetes_X_temp[:-20]
diabetes_X_test = diabetes_X_temp[-20:]

# Split the targets into training/testing sets
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test = diabetes.target[-20:]

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)

print "Results of Linear Regression...."
print "================================\n"
# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((regr.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, regr.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

# Now using the sknn regressor
# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html
#

from sknn.mlp import Regressor, Layer

nn = Regressor(
    layers=[
        Layer("Rectifier", units=200),
        Layer("Linear")],
    learning_rate=0.02,
    n_iter=10)

nn.fit(diabetes_X_train, diabetes_y_train)
print "Results of SKNN Regression...."
print "==============================\n"

# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

Results of Linear Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 2548.07
Variance score: 0.47

￼
Results of SKNN Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 5737.52
Variance score: -0.19

Changing the number of iterations to 1000 results in a score of -0.15


texts: Why is my predicted vs observed plot worse for training than validation. Running an overfitted GBM on a binomial outcome

texts:
I have a binomial outcome that I am trying to predict using a gbm in h2o.

I have set quite a low min_rows value for each node and it appears to be overfitting.  See plots below.

When I group the results into percentiles and plot Observed vs Predicted I get good results for Validation but Training shows a higher proportion of observed outcomes in the upper percentiles and lower observed outcomes in the lower percentiles.  See images below.

I would have thought training would have predicted more accurately and the Validation and Hold Out plots would have been more inaccurate.  Is this the expected result and why does it occur?


texts: Support Vector Machines with soft margin: solving the dual form

texts:
I am currently struggling with finding an analytical solution for the $\alpha_k$.
I have derived the following constrained optimization problem:
$$
L = \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i \alpha_j y_i y_j (\textbf{x}_j^T \textbf{x})
$$
$$
s.t. \quad 0 \leq \alpha_i \leq C \quad \forall i, \quad \sum_{i=1}^{N} \alpha_i y_i = 0
$$
I had, at first, not taken the constraints into account which, after taking the derivative w.r.t. $\alpha_k$, gave me:
$$
y_k \sum_{i=1}^{N} \alpha_i y_i (\textbf{x}_j^T \textbf{x}) = 1
$$
This system of linear equations I could easily solve in Python using numpy.
But as the alpha values were way too high (as could have been expected), I went back and found that I had forgotten about the constraints.
Now, I don't know how to find an analytical solution to that.
I have tried writing down the problem using Lagrange Multipliers but that doesn't seem to get me anywhere.
I have also looked around the Internet a lot and couldn't find a single lecture/slides/etc. that actually went on from that point.
Now is my question, is there a way to find an analytical solution to that constrained optimization problem or do I have to just put all of that into a solver?
And if there is a solution, I would appreciate some hints on how to get there.
Thank you for your Time!


texts: Training a classifier with text and numerical features - what is the state of the art?

texts:
I'm trying to build a binary classifier where the features are mostly numerical (about 20) and there are a couple of unstructured short text fields as well. What is currently considered the state of the art for combining these types of features?

I've tried building a separate classifier (logistic regression, TFIDF) using the text data alone, and then including that output score of that classifier as an additional when training using the rest of the numerical features (random forest, taking care to train each classifier on different folds of the data to prevent signal leakage). It works alright, but I think a better performance is possible.

A variant is to simply train two separate classifiers, one using text and the other using numerican features and then stacking those two. And finally, another idea is to use neural networks, have two input networks, one CNN/LSTM for the text, and another dense for the numerical features, and then combining them and having a single output.

Are there other approaches I haven't thought of that are worth a try?


texts: What is the best approach to perform information extraction from tourist reviews using NLP, DL?

texts:
I am interested in performing some information extraction from tourist reviews about different places.
I have data of 50 different places and around 300-400 reviews about each of them and I would like to get information like: weather, experience(happy/not happy), costly/cheap, places around, best time to visit, etc.
Reviews are like:

Mahabaleshwar is a hill station situated nearby Pune. You can reach
there by road via Pune. Air connectivity is available up to Pune and
from there you can travel by road. Mahabaleshwar is having so many
mountain range. Weather is remaining very beautiful throughout the
year.
There are so many points to be visited in Mahabaleshwar like Parsi
point, Elphinstone point, Wilson Point, Arthur seat point. There is
Vennna Lake where you can have boating and horse riding. Strawberry is
available at each and every place in Mahabaleshwar which is its main
attraction. There are so many hotels available but they are somewhat
costly. By the way it is the nice place to visit.

Mahabaleshwar is a best place for tourists. It is located in Maharashtra. I also visit Mahabaleshwar when I'm free time. It is
famous for their's natural beauty. Here is a one river ans when I
visit to Mahabaleshwar then I definitely go to ride on small boat, it
Is very joyful.

Need some suggestions on what algorithm can be used and about data processing approach. Links or tutorials are would be helpful too.
Thanks!


texts:
On this page of Google's Machine Learning Crash Course, we find the following statement:

"Test sets and validation sets "wear out" with repeated use. That is, the more you use the same data to make decisions about hyperparameter settings or other model improvements, the less confidence you'll have that these results actually generalize to new, unseen data. Note that validation sets typically wear out more slowly than test sets."

I don't get the last sentence. Why would a validation set wear out slower than a test set? To me it would wear out faster as the validation set is used a lot more frequently than the test set. What am I missing?


texts: Kronecker product using sparse matrices / tensors in tensorflow

texts:
I am currently trying to perform a kronecker product on a pair of sparse tensors in tensorflow. I have found some code on another thread to perform the kronecker product using normal tensors.

def tf_kron(a,b):
    a_shape = [a.shape[0].value,a.shape[1].value]
    b_shape = [b.shape[0].value,b.shape[1].value]
    return tf.reshape(tf.reshape(a,[a_shape[0],1,a_shape[1],1])*tf.reshape(b,[1,b_shape[0],1,b_shape[1]]),[a_shape[0]*b_shape[0],a_shape[1]*b_shape[1]])

However this is not the case with sparse tensors. I'm not fully aware of how to reshape sparse tensors.


texts: I have photos of people's faces, and I want to detect each person's skin color for later use. I don't want to use predetermined color ranges, instead I want to determine each person's color and then use it to detect occlusions in this person's later photos. The training face images are taken under the same conditions, but testing will be with photos of different conditions. My first thought is to use color histograms of these images, and then the peak will be this person's skin color. Is there a smarter way to do this? I'm using Python and OpenCV, but I can use other tools if needed.


texts: Clustering Customers on transactional behavior

texts:
Objective:

Segment the accounts on their transactional behavior and find the accounts which are more likely to subscribe for loans.

Dataset:

1)      Account_Number

2-91)   Transaction amount aggregated on days for last 90 days

92)     is_subscriber? (0/1)

My Approach:

1) Normalized the data (x-min(x)) / max(x) - min(x)

2) Clustering on column 2-91

3) Analyze the clusters to check which cluster has the maximum percentage of subscribers (is_subscriber? == 1)

4) Sort the clusters on the basis of maximum percentage of subscribers

5) The remaining non-subscribers (is_subscriber? == 0) of cluster with highest percentage are most likely to get subscribed.

But the results I am getting doesn't look promising.

The data is already prepared. No other variable can be added.

Questions

1) which data normalization to use?

2) Which distance to use in clustering?

3) Which clustering algorithm to use?

4) Considering the data if there is any other approach you can suggest?

Data can be found here: https://drive.google.com/open?id=1ydL1AQmozBlfiqz0nsM3f_E2JK2KB8tD


texts: Is machine learning a viable tool to map accent from speech onto text/syllables?

texts:
I'm quite new to coding and even newer to machine learning, so please excuse if this is a stupid question to ask.
The project I am thinking about building is a program that helps correct pitch accent (the changes in high and low in between syllables) by analyzing speech from user input.
So let's say we have a random word I want to analyze, what I give to the Program is the representation of the audio (see further down) and a count for how many syllables the audio contains. As an output I want to know for each Syllable if it is either High or Low.
Let's do an example for the English word Animal . Pitch is the blue line
As an input I give the number of syllables (3) and a representation of the audio. As an output I want to see the Syllables mapped onto either High or Low, for the example "animal" [High, Low, Low/ 1,0,0] for "malaria" (4 syllables) I'd want [LLLH/0,0,0,1].
.

In order to do this I plan to train a Neural Network (Since with those the difference in speaking pace apperently doesn't need to be controlled for) and preprocess my audio with mel frequency cepstral coefficients in order to represent the features present in human speech. (I also thought about preprocessing to represent just the pitch but i am not sure if I might take away too many features then).
As mentioned earlier as an output I want the representation of where in the word the changes in hight occur.
My training data consists of 50000 recordings of various pitch-accent-patterns over words, the recordings themselves are labeled with the pattern and the word they represent.

Does this seem doable with a Neural Network or machine learning in general?
How do I get the Syllable Count to be weighted so much that the output is directly related to the high and low of each syllable?
Do you have any ideas/tips/suggestions in general?


texts: Variance as criteria for feature selection

texts:
I'm working on an unsupervised clustering problem. I read multiple times that a variable with higher variance can be chosen over a variable with a lower variance. For example, scikit-learn implements a function that removes features with a variance lower than a threshold. (sklearn.feature_selection.VarianceThreshold)

However, isn't the variance entirely dependent on scale/measurement unit? If I standardize my features, the variance is 1 for all of them.


texts: Hyper tuning reduce the accuracy score, why?

texts:
I have performed hyper tuning grid CV search on KNN model. The actual accuracy score for my KNN was accuracy of 42.31 % without performing hyper tuning. However, after performing hyper tuning, the accuracy score has gone down to 27.97 %?
I would like to know why this happened? I performed Near Miss (under sampling) to balance my dataset.


texts: how to build word2vec content based recommendation?

texts:
I am building a content-based recommendation system for hotel accommodation. I have a hotel name, hotel description and location. I combined hotel name, description and location. Then, applied NLP and converted into vectors using TF-IDF.

I am taking hotel name and location as an input and recommend the hotel based on cosine similarity. I wanted to use word2vec instead of TF-IDF and make a recommendation. Anyone can suggest how to do this?


texts: Difference between classification problem and concept learning

texts:
"To ground our discussion of concept learning, consider the example
task of learning the target concept: days on which my friend Aldo
enjoys his favorite water sport"

This is from the book Machine Learning by Tom Mitchell. This sure sounds like a classification problem to me, can anyone say why is a concept learning task?


texts: sine, cosine transformed cyclical features - am I losing information?

texts:
If I use sine, cosine transformation for cyclical features (e.g. weekday or hour of the day), do I lose information if the first ordinal value was 0 respectively?
Assume hours of the day are encoded as follows: 0, 1, ..., 23
If I apply the sine, cosine transformation, I get 23 data points instead of 24 (cf. plot)

Am I losing information on the first ordinal value (0)?
Many thanks in advance!


texts: Probability calibration is worsening my model performance

texts:
I'm using RandomForest and XGBoost for binary classification, and my task is to predict probabilities for each class. Since tree-based models are bad with outputting usable probabilities, i imported the sklearn.calibration CalibratedClassifierCV, trained RF on 40k, then trained CCV with a separate 10k samples ( with cv="prefit" option ), my metric ( Area Under ROC ) is showing a huge drop in performance. Is it normal for probability calibration to alter the base estimator's behavior?

Edit : Since i'm minimizing logloss on my XGBClassifier, the output probabilities aren't that bad compared to RF's outputs.


texts:
I have some data reduced by TSNE into a 2D representation, which shows clear spatial features.

However, I'd like to format this into a grid – not just snapping data to the nearest grid square but spreading everything out to fill up a grid, preserving (as much as possible) the existing spatial relationships.

So far, I've only found this article which might close to what I need? This process might already have a name and I'm just one step from an easy Google solution, but at the moment I'm stuck!


texts: is it possible to output more than 2 nodes away from a node in a decision tree? if yes, how to do that with sklearn?

texts:
usually a decision tree has one root node, some nodes, and some leaves.

lots tutorial illustrate this as something like binary tree.

is it possible more than 2 nodes away from a node in a decision tree?

this image comes from this post

by "more than 2 nodes", i mean there are more than 3 splits (in this case, 3, Low, Med, High) away from the root node.

if it is reasonable in real life application, plz provide an open dataset on which a decision tree would spit more than 2 nodes, and a piece of sklearn code.


texts: Importance/intuition behind stacking RNNs

texts:
Nowadays there's a trend towards using architectures of "deep"  RNNs i.e. vertically stacked RNNs. RNN chapter from Bengio's bookThese networks seem to work well in practice.

What's the intuition around using vertically stacked layers of RNNs (beyond the obvious fact that they increase the capacity by increasing the # parameters)?


texts: Recurrent Neural Networks Over Multiple Documents Over Time

texts:
So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on how to actually implement it let me know. Or if I am over-thinking a solution.

I am trying to classify accounts into one of two groups, good and bad. I have multiple text documents per account. What I want to do is take the text documents for a single account and order them chronologically. Then use a recurrent neural network to essentially learn an embedding for each document (traditional text recurrent architecture should work fine), and then plug those embeddings into a document-level recurrent neural net, to predict when an account switches from good to bad. That way, as new documents come in per account I get updated "badness" scores for an account, and potentially flag an account above some threshold. I would like some sort of end-to-end architecture too. Like I said, not sure how to get what is my head into the computer.

Here's my question, is anyone aware of an implementation like that?


texts: Automatic Semantic Clustering and Tagging of sentences using NLP

texts:
NLP Analysis for keyword clustering

I have a set of keywords for search engines and I would like to create a python script to classify and tag them under unknown categories.

To make it clear I should have an output like this one, without knowing the categories (Product, Colour, Accessory, Brand...):
+----------------------------+------------+----------+--------------+-----------+
|.......Keywords............|.Product...|.Colour.|.Accessory.|.Brand...|
+----------------------------+------------+----------+--------------+-----------+
|.red shoes with heels.|.shoes......|.red......|.heels.........|..............|
|.Apple computer.........|.computer.|............|..................|.Apple....|
|.Armani blue shoes....|.shoes......|.blue.....|..................|.Armani..|
|.black mouse..............|.mouse.....|.black...|..................|..............|
|.gaming laptop...........|.computer.|.............|..................|..............|
+----------------------------+------------+----------+--------------+-----------+

Any suggestions on how I could be able to do it?

I am currently using Word2Vec to find similarities between words and some APIs to recognize Brands and entities in the keywords

    model2 = models.Word2Vec.load('semantic_clustering/datasets/it/it.bin')
    with open('tmp/kw_msm.txt') as f:
        kwlist = f.readlines()
    kwlist = [x.strip() for x in kwlist

    # unique_words = list(set([word for word in kw.split(' ') for kw in kwlist]))
    freq_words = defaultdict(int)
    words = set()
    for kw in kwlist:
        for word in kw.split(' '):
            freq_words[word] += 1
            words.add(word)
    sorted_freq_words = sorted(freq_words.items(), key=operator.itemgetter(1), reverse=True)

    # Creating Dataframe with words as columns
    df = pd.DataFrame(columns=words)
    df['keyword'] = kwlist
    for i, row in df.iterrows():
        for w in row['keyword'].split():
            df.loc[i, w] = 1

    good_words = [w for w in words if kw_in_vocab(w, model2)]

    KW_MODEL = model2[good_words]
    NUM_CLUSTERS = 10
    kclusterer = KMeansClusterer(NUM_CLUSTERS, distance=nltk.cluster.util.cosine_distance, repeats=25)
    assigned_clusters = kclusterer.cluster(KW_MODEL, assign_clusters=True)
    clusters = {}
    for i, w in enumerate(good_words):
        clusters[w] = assigned_clusters[i]
    sorted_clusters = sorted(clusters.items(), key=operator.itemgetter(1))
    for k in sorted_clusters:
        print(k)

This is a snippet of code I am using, creating a sparse matrix of words and clustering the columns with a fixed number of clusters, it's only a first test


texts: About the Licence of Shared Codes on the Kaggle Site

texts:
I have one job offer and there are many codes on the Kaggle site based of that job offer which could be used for the company usage like the result of this google search, and Salomon Tetelepta codes:

So if possible I like to find out have all the shared codes via the Kaggle site the APACHE 2.0 license and if not how much change on one code needs to change the main license?


texts: How do I calculate metrics like precision @ 99% recall using sklearn?

texts:
I'm trying to calculate metrics like precision @ 99% recall for a classifier using sklearn.

sklearn.metrics has precision_score but it has no way to calculate it for specific values of recall. One way I can think of is to use precision_recall_curve and hope that for some threshold, recall value happens to be 99%, so that I can use the corresponding precision.

Is there a better way to do this? If not, is there a way I can specify the range/number of thresholds to be used by precision_recall_curve?


texts: How to find a 'similarity' measure between 2 pandas dataframes?

texts:
I have 2 pandas dataframes: one big (300.000+ rows) and one little (50- rows) with the same columns.
Assuming that the entries of the little one form a cluster I want to identify the entries of the big one that belong to this cluster.
Which algorithm can I use?
Any suggestion is welcome.
I have access to all the python ecosystem and I plan to use this: https://pypi.org/project/gower/.
(I have an approach in mind somehow described here: scikit-learn one class svm results: how to improve?)


texts: What values should I choose for P and Q?

texts:
I am trying to forecast energy data generated by LoadProfileGenerater. This data is generated every half hour for 2 years. I am following this tutorial.

I have checked it for stationarity using Dickey Fuller Test

Results of Dickey-Fuller Test:
Test Statistic                -1.684705e+01
p-value                        1.126331e-29
#Lags Used                     5.200000e+01
Number of Observations Used    3.503500e+04
Critical Value (5%)           -2.861623e+00
Critical Value (1%)           -3.430537e+00
Critical Value (10%)          -2.566814e+00

Results indicate that the time series is very close to stationarity (Test Statistic < Critical Value (1%)).

I have therefore used ARMA for forecasting.

However I am not being able to determine the values of p and q. Both, PACF and ACF charts indicate intersection with the upper confidence line at 0.01.

What values for p and q should I use?

EDIT 1:

I tried differencing as suggested in the comments. Rolling mean has dropped to nearly 0 but rolling std has increased to 0.75. ACF and PACF have generated similar charts as without differencing.

Source used for Differencing:

# Differencing
ts_log = np.log(ts)
ts_log_diff = ts_log - ts_log.shift()
ts_log_diff.dropna(inplace=True)
stationarity.test_stationarity(ts_log_diff)

Results of Dickey-Fuller Test:
Test Statistic                   -64.965417
p-value                            0.000000
#Lags Used                        50.000000
Number of Observations Used    35036.000000
Critical Value (5%)               -2.861622
Critical Value (1%)               -3.430537
Critical Value (10%)              -2.566814


texts: Implementation and interpretation of recursive convolution neural net(RCNN)

texts:
I am working on to implement the approach in the paper https://arxiv.org/pdf/1603.03101v1.pdf on detecting text in wild using recursive convolution neural net and attention modelling. Being a bit new to deep nets and have developed the following interpretation about the recursive CNN given in the paper:

It is given in the paper that while training could be done by " reusing the same convolutional weight matrix multiple times at each layer." Does this means the following thing in the pseudo code below

layer1:

iterating multiple times over { train convolution_layer(weights) with labels
}

layer2:

iterating multiple times over { train convolution_layer(weights) with labels }

Here in the consecutive iteration in a layer we use weights of previous iteration. We use different weight initialisation for different layers here

While using the network for test set, we will input the weights we got from the last iterations.
Please comment on this interpretation and I would be grateful if anyone could suggest some links and literature for the same. Thank you!


texts: Help on designing data science curriculum from "traditional" offerings

texts:
So I'm hoping to ask professionals for their advice here. My university doesn't offer a "data science" major and I'm cobbling one together from different departments. So far I have a math major with stat emphasis but I'm having trouble finding technology use in this program so I was thinking computer science for science credits. I also have a choice of an Information Management minor or a Business Analytics minor, both from the business college.

So basically, I'm wondering:

Which areas of computer science should I focus on? (I can only take 2 aside from the intro CSI and CSII course).
Which minor should I go for? Analytics consists of applied econ analytics, marketing research, and quantitative analysis while the info minor is management, programming, busn intelligence (advanced excel), and "big data". Right now I'm leaning towards the info one because it seems to have more software in it, including SQL.

(I should also mention that I'm located in Las Vegas and my university even offers probability and gambling statistics but that's in the hospitality college and not sure that would be worth the trouble. It sounds interesting though, and might be good experience?).

My advisors have no idea what I'm going for, partly b/c they're all in different departments, so I hoped someone from the real world job market might know a thing or two :-)


texts:
I am looking for a good book about unsupervised learning that goes beyond the typical k-means and hierarchical clustering algorithms.

Practical implementations in R or Python will be a plus.

Currently I am working in retail, so the typical use cases I am interested are customer segmentation, products segmentation.

Any ideas?


texts: Facing a difficult regular expression issue in cleaning text data

texts:
I am trying to substitute a sequence of words with some symbols from a long string appearing in multiple documents. As an example, suppose I want to remove:

Decision and analysis and comments

from a long string. Let the string be:

s = Management's decision and analysis and comments is to be removed.

I want to remove Decision and analysis and comments from s. The catch is, between Decision, and, analysis, and, comments, in s there could be 0, 1 or multiple spaces and newline characters (\n) showing up with no pattern in different documents, for example, one document shows:

Management's decision  \n \n and analysis\n and \n comments is to be removed

while another has a different pattern. How do I account for this and still remove it from the string?

I tried the following, of course unsuccessfully:

st = 'Management's decision  \n \n and analysis\n and  \n comments is to be removed'
re.sub(r'Decision[\s\n]and[\s\n]analysis[\s\n]and[\s\n]comments','',s)


texts: Using python and machine learning to extract information from an invoice? Inital dataset?

texts:
DISCLAIMER: I have absolutely no background with machine learning/data science, and am unfamiliar with the general lingo of data science, so please bear with me.

I'm trying to make a machine learning application with Python to extract invoice information (invoice number, vendor information, total amount, date, tax, etc.). As of right now, I'm using the Microsoft Vision API to extract the text from a given invoice image, and organizing the response into a top-down, line-by-line text document in hopes that it might increase the accuracy of my eventual machine learning model. My current situation is strictly using string parsing, and this method works pretty well for invoice number, data, and total amount.

However, when it comes to vendor information (name, address, city, province, etc.) string parsing hardly ever works due to my script relying on the detection of an address (because addresses have a pretty distinct format compared to the rest of the information on an invoice). Tax information is difficult to parse because of the amount of numeric values that appear on an invoice.

So (and here's where I get lost)
I envision a machine learning model that will have an input of a single invoice image from a user, and it's output will be the extracted invoice information. I am currently looking into Azure Machine Learning Studio because the plug 'n play aspect of it appeals to me and it seems easy enough to use and experiment with. BUT I have no clue what the requirements are for an initial dataset! Should I just fill my dataset (in a CSV format btw) with the necessary information (invoice number, total amount, date, ...) from a bunch of sample invoices? If not, what other information should I include in my dataset? I was thinking x-y coordinate pairs of where the important information occurs on the image.

One last question related to this problem scope, which algorithm (regression, classification, clustering) could even "extract" (or help with it) information from the input text? As far as I know, regression predicts numeric values (i.e. 2 + x = 10 so 8 + x could be 16) and classification "labels" things (i.e. this image contains a tree, this one does not). I'm not too familiar with clustering, although I think it could be useful to identify the structure of the input text.

To summarize: what might be some features of an invoice that I can fill an initial dataset with to initialize a model? How could a clustering algorithm be used to identify the structure of an invoice? How could I utilize a regression/classification algorithm in this problem, and what inputs would these algorithms need to deliver meaningful results?

Any help/guidance/feedback would be appreciated to the moon and back. Sorry for my lack of knowledge, but this field is very interesting and need some help wrapping this all around my head. Thanks:)


texts: How to compare two clustering solutions when their labelling differs

texts:
I am planning to test the reliability of a clustering approach for some data. My plan is to repeatedly (with replacement) draw a number of random subsample pairs (e.g. 2x 10% of the total data), run the clustering on both individually, and then compare the results. The issue is that I am using HDBSCAN, which not only creates a non-fixed number of clusters (for different sets of data but same params), but it also therefore labels clusters differently since k is not defined, and the input data will always have slightly different structure due to variability.
I tested this by using the same HDBSCAN parameters on two subsamples (A, B) of my data, and my issue is quite easy to see. The cluster labels with corresponding samples for A were:
{-1: 4306, 0: 1737, 1: 2999, 2: 72068, 3: 20628, 4: 3120}
while for B they were:
{-1: 4478, 0: 1711, 1: 3048, 2: 72089, 3: 3123, 4: 20408}.
From this, it seems that the solution is very close until we compare label 3. It looks like label 3 of A corresponds to label 4 of B.
My initial thought was that I could just relabel them both in order of each cluster's sample size. But this assumes that the two solutions will be similar across many tests (which is ultimately the whole point of the testing in the first place). So my next thought is I could set the constraints that (1) there should be a "similar" number of samples in the noise group, and (2) there should be the same number of clusters found. If these two conditions are met then I could relabel the clusters by order of their sample size, and then make my comparison using ARI or AMI.
I am doubtful that this is good, because I don't believe it is necessarily true that (even given the two constraints) two clusters labelled the same on the basis of their sample should necessarily correspond to the same "global" cluster. It therefore seems problematic to me but I can't think of an alternative.
Is the above approach generally reasonable? If not, is there something else I could do to assess the reliability/stability of HDBSCAN solutions? As an alternative, would it be better to just compute the DBCV score, %noise, and the number of clusters, and then use this as an indication of the quality of the clustering?


texts: How to deal with a sparse matrix when using a perceptron based recommender system?

texts:
I'm constrained to use a perceptron based method. I have a user-item matrix filled with rating data on scale of 1 to 5 like this, with around 50% of the matrix with no data:

r<- matrix(c(2,4, NA,5,NA,3, NA,5,NA,1,NA,3,NA,5,NA,4,4,NA,NA,NA,1,1,2,NA,1,1,1,1,NA,NA,NA,NA,2,3,4,2,NA,NA,NA,NA,3,4,5,1,NA,NA,2,3,NA), nrow=7).
#one row represents one user, hile one column represents one item
r
    [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    2    5   NA    1   NA    2    5
[2,]    4   NA    4    2   NA   NA    1
[3,]   NA    1    4   NA   NA   NA   NA
[4,]    5   NA   NA    1   NA   NA   NA
[5,]   NA    3   NA    1    2   NA    2
[6,]    3   NA   NA    1    3    3    3
[7,]   NA    5    1    1    4    4   NA

I'm recoding the NA's to 0's, which makes the perceptron give out only 2 classes (one containing all the 1's and 0's , and the other containing all the 2,3,4 and 5. Which is understandable I guess.) How do I deal with this? I tried mean imputation but the results are not good (20% accuracy)

(I can't give exact code due to this being a proprietary code of my company, but any perceptron based method should work for this example)


texts: What's the difference between multiclass categorical crossentropy, mlogloss and multi:softprob?

texts:
As far as I understand, an objective is something I'm trying to optimize and an evaluation statistic is something I use to look for overfitting. I stumbled upon 4 losses that seem to be the same, but I'm not quite sure.

In XGBoost, one of the objectives for multi class classification is multi:softprob (https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters). I couldn't find any formulas in the documentation, but apparently this objective returns "predicted probability of each data point belonging to each class", so an "ndata * nclass matrix".
One of XGBoost's eval_metrics is mlogloss. The documentation (same link as above) links to sklearn.metrics.log_loss, which is "log loss, aka logistic loss or cross-entropy loss". sklearn's User Guide about log loss provides this formula:
$$
L(Y, P) = -\frac1N \sum_i^N \sum_k^K y_{i,k} \log p_{i,k}
$$
So apparently, mlogloss and (multiclass categorical) cross-entropy loss are the same.
The Otto Group Product Classification Challenge talks about "the multi-class logarithmic loss" and gives the same formula as above, so looks like mlogloss, cross-entropy loss and multi-class logarithmic loss are all the same.

Now, I'm not particularly sure what multi:softprob is. multi:softmax is softmax - the activation function used in neural networks, for example. However, it's not a measure of error, like cross-entropy loss, right? Why is it one of the objectives in XGBoost, then? Unfortunately, there doesn't seem to be any useful information about multi:softprob, except that it's not the same as softmax because softprob outputs a vector of probabilities and softmax - "a class output" (so the ID of a class, I presume?).

Am I correct that mlogloss, cross-entropy loss and multi-class logarithmic loss are the same thing? How are they different from multi:softprob? What is multi:softprob anyway?


texts: What kind of clustering would work better on such data? Would k-means work on such data?

texts:
I have a dataset where datapoints are more or less spread like this:

What if I want to split the data in 2 data clusters, what would be a good choice? Would k-means work here?
Thanks.


texts: Random forest minimum number of observations

texts:
I have around 5000-6000 observations of nearly 8-10 variables (of which 2 are discrete, categorical) and a single numerical target parameter. As per initial evaluation, random forest regression might be a good algorithm for the current case.

Is the current observations/variables count adequate for the proposed method? If other regression algorithms are recommended as things are described currently, kindly let me know.


texts: Siamese vs matching network for correct image category matching

texts:
I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese or matching network. In Siamese, I have to match my image with all existing images in the folder to find the correct match. So do you think if I can use a matching network and produce a better result? What is the parameter based on which developer decides where to use matching/prototypical network and where to use Siamese network.
Note- At the time of the test new unseen images class can be added on which model is not trained. Do you think a matching network will still work for this case?


texts: Help improving my "read_excel" execution time in python. My code reads slowly

texts:
My first question here so please bare with me.

I'm trying to feed my neural network with training data read in from an excel file. It works perfectly fine when i have less than 50 rows in the sheet. But when i try with the real excel file containing almost 4.000 rows it suddenly takes forever. Although 4.000 is a lot i'm pretty sure my way of doing it is still very inefficient.

as you can see in the code below i'm using the read_excel over and over again in the loop.
I feel like there should be a way to only read the whole column 1 time and then work with it from there.

My goal is to read in 5 rows as the 1st input starting from row 0. then reading 5 rows in again but starting from row 1 and 5 rows again starting from row 3
So it's like a window of 5 rows that is read and then moving the window by 1.
The output should allways be the 1 row after the window.

**Example:** if row 1-20 contains numbers 1-20 then:
input1 = [1,2,3,4,5] and output1 = 6
input2 = [2,3,4,5,6] and output2 = 7
...
input15 = [15,16,17,18,19] and output15 = 20

notice how inputs are lists and outputs are just numbers. So when i append those to the final input & output lists i end up with inputs being a list of lists and out being list of outputs

My code

from pandas import read_excel

# initialize final input & output lists. The contents of the temporary input & output lists
# are gonna be appended to these final lists
training_input = []
training_output = []

# excel relevant info
my_sheet = 'Junaid'
file_name = '../Documents/Junaid1.xlsx'

# initialize counters
loop_count = 0
row_counter = 0

for x in range(25):

    # load the excel file containing inputs & outputs
    # using parameters skiprows, nrows (number of rows) and index col
    df = read_excel(file_name, sheet_name = my_sheet, skiprows=row_counter, nrows=6, index_col=0)

    # initialize temporary input & output lists
    input_temp = []
    output_temp = []

    for y in df.index:
        # append the first 5 rows of the 6 to input list
        if loop_count < 5:
            input_temp.append(df.index[loop_count])
            loop_count += 1
        else:
            # append the 6th data to output list
            training_output.append(df.index[loop_count])

    training_input.append(input_temp)
    row_counter += 1
    loop_count = 0


texts: How can I get my trained model ready for production

texts:
I successfully trained my model using the sklearn's multiple linear regression. This is the code I used:

import pandas as pd

dataset = pd.read_csv('C:\\mylocation\\myfile.csv')
dataset2 = pd.get_dummies(dataset)
y = dataset.iloc[:, 31:32].values
dataset2.pop('Target')
X = dataset2.iloc[:, :180].values

#Split the dataset
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 0)

#Feature Scaling
#from sklearn.preprocessing import StandardScaler

#sc_X = StandardScaler()
#X_train = sc_X.fit_transform(X_train)
#X_test = sc_X.transform(X_test)

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

#Predicting the Test set results
y_pred = regressor.predict(X_test)

According to validation results my y_pred is a reasonable predictor. Now, I would like to take into production this model and I am wondering what are reasonable steps to apply this model to the whole dataset I have stored and future datasets if needed.


texts: Low prediction/classification accuracy due to imbalance in data feeding

texts:
I am building the neural network for image analysis to do Chest Xray classification (Abnormal/Pass).

The classification accuracy for abnormal Xray is low, I guess it is due to the lack of abnormal samples to train.

Attached image is the schema I built. Is there any ways to improve the classification accuracy for abnormal Xray?


texts: On the offline evalution of recommender system

texts:
There are mainly three ways to evaluate a recommender system: offline, online and user study.
For most academic papers, offline evaluation is used to show the improvements:

They split the offline dataset into training and testing sets
They train models on the training dataset
They evaluate them on the testing dataset.

However, it seems that for most nonsequential recommendation works, the dataset splitting is random, without considering the temporal sequential order of the records. I think this may cause a problem of overestimating.

For example, consider evaluating itemCF on MovieLens: if the dataset is random split, the order of the samples will be shuffled. It means that we may use future data to train a model and make a prediction about the past:

  The training dataset contains that Bob bought a Harry Porter book in April 2019 (which is the second purchase of Harry Porter), and the
  testing dataset contains that Bob bought a Harry Porter book in March
  2019 (which is the first purchase of Harry Porter). An item-based CF
  method may recommend Harry Porter for the first purchase due to the
  second purchase, which is against the law of causation with a result
  of overestimating the performance of item-based CF.

It is kind of cheating. And I fail to see this problem being discussed somewhere.
So, I doubt the real performance of the offline evaluation for recommender system and would like to hear more voices about this.


texts: What are the state-of-the-art models for identifying objects in photos?

texts:
From my observations and little experience it appears that most of the ML project are about classifying stuff. Is there cancer signs on the photo? Does the picture show car, whale or banana? Etc.

I need to implement a model for face identification. Not detection/recognition, but identification: having two different photos of the same person, my model should determine if in the pictures is depicted the same person.

I want to achieve that using Tensorflow with convolutional nets. I've read this paper: http://ydwen.github.io/papers/WenECCV16.pdf and center loss looks promising. What do you think about that? Are there any new ideas/papers/implementations regarding that problem that are worth attention?

I asked this question also on MachineLearning reddit (https://www.reddit.com/r/MachineLearning/comments/8cysrx/d_what_are_the_stateoftheart_models_for/) and got an useful link with FaceNet implementation, trying here also :)


texts: Perform collaborative recommender analysis without rating in a transactional data

texts:
I'm exploring options for recommender systems optimized for the online retail dataset, which would take into account that no product or user rating is available. Can collaborative recommender analysis performed on these dataset ?

Is there any way or suggestion on how can we create our own ranking or rating based on user purchase history ?


texts: How do you get an RNN to learn in real time?

texts:
In order to train a recurrent neural network, you have to unfold it say 50 times and treat it like a chain of RNN cells.

Now if you are using this RNN to train in realtime, then it seems you may as well just have a chain of 50 cells. Maybe fix their weights to all be the same as in a convolutional neural network.

So are RNN's any use in learning in realtime for temporal data?


texts:
I'd like to classify the data on coordinate.

Here are 2 example data.

data1 = [(1,1), (2,2), (3, 3), (4, 2), (5, 3), (6, 0)]

data2 = [(1,1), (2,2), (3, 10), (4, 9), (5, 10), (6, 0)]

The bold part have the same wave in above data.

The length are all different in my data set.

Is there any way that I can find the similar wave in many data like this?

Thank you.


texts: My network optimises for mean_squared_error, but the predictions are useless

texts:
After going through the tutorials of tensorflow I decided to do my own project. For this I did the following:

I have scraped movie reviews from a polish site called filmweb. I got a dataset o 5400 movie reviews with a rating in range 1-10
I used the “Morfeusz” python bindings to tokenize and lemmatize the text. I have stripped most basic stop words.
I mapped most used 10000 lemmas to numbers and put the data in lists.
Randomly split the data into two equal sets.
My training data is the movie review lemmas padded with zeroes (like in the tutorial). The expected result is the rating.
My model is

    vocab_size = 10000

    model = keras.Sequential()
    model.add(keras.layers.Embedding(vocab_size, 16))
    model.add(keras.layers.GlobalAveragePooling1D())
    model.add(
        keras.layers.Dense(16, activation=tf.nn.relu)
    )
    model.add(keras.layers.Dense(1))

    optimizer = tf.keras.optimizers.RMSprop(0.001)

    model.compile(optimizer=optimizer,
              loss='mean_squared_error',
              metrics=['accuracy'])

After running the optimization process i see the loss metric dropping quite fast. However the accuracy is well below 0.1 and after I check the predictions of this network, they are useless. The model predicts all the ratings to be in range of 6-7. So I understand that the network “optimized” by simply always guessing the mean value.

I wonder what should be my next step to improve. Is my model wrong for this task? Should I somehow normalize or augment my data? Or is the dataset size simply too small to get any meaningful result?

EDIT: this is the link to my dataset and the script that generated it:
https://www.dropbox.com/s/csyo934hhtbxxu9/fw-data.tar.bz2?dl=0


texts: How to group identical values and count their frequency in Python?

texts:
Newbie to analytics with Python so please be gentle :-) I couldn't find the answer to this question - apologies if it is already answered elsewhere in a different format.

I have a dataset of transaction data for a retail outlet. Variables along with explanation are:

section: the section of the store, a str;
prod_name: name of the product, a str;
receipt: the number of the invoice, an int;
cashier, the number of the cashier, an int;
cost: the cost of the item, a float;
date, in format MM/DD/YY, a str;
time, in format HH:MM:SS, a str;

Receipt has the same value for all the products purchased in a single transaction, thus it can be used to determine the average number of purchases made in a single transaction.

What is the best way to go about this? I essentially want to use groupby() to group the receipt variable by its own identical occurrences so that I can create a histogram.

Working with the data in a pandas DataFrame.

EDIT:

Here is some sample data with header (prod_name is actually a hex number):

 section,prod_name,receipt,cashier,cost,date,time
 electronics,b46f23e7,102856,5,70.50,05/20/15,9:08:20
 womenswear,74558d0d,102857,8,20.00,05/20/15,9:12:46
 womenswear,031f36b7,102857,8,30.00,05/20/15,9:12:47
 menswear,1d52cd9d,102858,3,65.00,05/20/15,9:08:20

From this sample set I would expect a histogram of receipt that shows two occurrences of receipt 102857 (since that person bought two items in one transaction) and one occurrence respectively of receipt 102856 and of receipt 102858. Note: my dataset is not huge, about 1 million rows.


texts: Large variation in cross-validation scores

texts: I'm training a CNN with 5-fold stratified cross-validation. On the first fold, my accuracy is ~80%, on each subsequent fold the accuracy is ~50%. Finally, upon fitting the entire training set my accuracy jumps back to ~80%. I can see that on the folds with bad performance the accuracy and loss never really change. Is it simply that one fold received all/most of the signal from the dataset or is there something else going on here?


texts: How to manage survivorship bias in conversion metrics?

texts:
I am trying to understand diminishing returns on additional advertising. Basically, does the 2nd ad convert worse than the 1st, etc. I'm struggling with thinking about how to deal with the survivorship bias, because if a user converts, then they will stop seeing additional ads.
Are there ways to manage these biases? Ideally, I'm interested in modeling the likelihood of conversion given some demo factors and how many ads they've received so far.


texts: Linear Regression + KFold cross validation

texts:
I have a prepossessed data set ready and the corresponding labels (8 classes).
I've already done KFold cross validation with  K=10 with some classifiers such as DT,KNN,NB and SVM and now I want to do a linear regression model, but not sure how it goes with the KFold , is it even possible or for the regression I should just divide the set on my own to a training and testing sets ?

The 8 classes are ages (for NLP problem) , so I want to check both options of classification and regression.


texts: I'm wondering if there's a way to automatically generate a list of tags for live chat transcripts without domain knowledge.  I've tried applying NLP chunking to the chat transcripts and keep only the noun phrases as tag candidates.  However, this approach would generate too many useless noun phrases.  I could use some rules to prune out some of them, but it would be hard to generalize the rules.


texts:
I am looking for a benchmark result or any kaggle competition held using MovieLens(20M or latest) dataset.
Similar question has been asked here but, provided links are dead so re-raising the question.


texts: orientation detection - images of govt. issued documents

texts: Trying to figure out a macro approach to detecting orientation of documents that works across document types (maybe a pipe dream). We successfully used SIFT to detect orientation in documents where we had a logo/emblem but ran into issues with documents issued at the state level (separate from documents issued at the national/federal level). For instance certificates of registration of vehicles. There is no standard pattern across different states and definitely no common feature/logo/emblem that we can use to triangulate the orientation of the document. Its just a bunch of evenly spread out text. We have tried thresholding techniques but it doesn't cut it for our use case ..sample image below ..any pointers would be truly and deeply appreciated


texts: The Yelp Dataset Challenge (https://www.yelp.com/dataset_challenge) releases data for a handful of cities each year. I'd like to analyze some cities from past years. Is this data archived anywhere?


texts: Clustering bitcoin addresses with k-means - how would one prepare input

texts:
I am looking to perform clustering on bitcoin addresses within the blockchain. I have generated a graph structure of the blockchain with a source address, destination address, value of transaction, in-degree, out-degree.

I think misunderstanding lies in the fact that this is directed graph.

How could this information be represented as input to the k-means clustering algorithm? Should nodes and their corresponding features be the input to k-means alone and should the directed (source/destination) element be removed from the input data?

Example data:

source address | destination address | transaction value | timestamp | in-degree | out-degree | pagerank | tags

source and destination addresses are just a public key of a sender and receiver of the bitcoin.

transaction value would be the number of bitcoins you would want to send.

in-degree would show the number of links (transactions in this case) pointing in to a given node

out-degree would show the number of links (transactions in this case) pointing out of a given node

pagerank is a number showing the importance of a node by the number of other important nodes linking to it.

tags will be string based descriptions of an addresses known activity. For example, some gambling companies publish their addresses so we can associate transactions involving this address as a gambling transaction.


texts: Is it suitable to change a feature by itself to generate an another feature?

texts:
Hi all I would love to hear your answers on this. Lets say I have two variables, voltage and current, in my data set. I could add another feature by squaring current (so as to calculate power).

Is this an example of feature engineering?

Recently I tried to predict on a diameter prediction on asteroids and I took the natural log of some features which worked well.

Can someone provide some insight as to why this may have improved the model's performance?


texts: How to fetch text from pdf to further proceed with question answer based model from the same document?

texts:
To illustrate the above title.

Suppose you have a pdf document, which is basically scanned from hardcopy, now there are set of fixed questions to answer from the document itself.
For an example a document contains a contract of land, now the set of fixed questions be "who is the seller?" "what is price of the asset? ", document has referred to this answers probably 2-3 times, as a human it's a simple task.

How to automate this?


texts: Get bounding boxes for adjacent instances of a single class in image

texts:
I have a dataset with thousands of music score pages and manually annotated bounding boxes for the individual bars:

My objective is now to train a DNN that should ultimately be able to get these bounding boxes on its own. First idea was to use something like the Region Proposal Network (RPN) from Faster R-CNN on top of ResNet or VGG, but I am unsure if this still works because the "objectness" is rather high for almost each section of the page. Plus the regions are mostly touching each other but rarely overlap. Number of bars is roughly somewhere between 1 and 250 per page.

Additionally, the number of systems (=rows of bars) per page is oftentimes not changing between subsequent pages. This might be a very helpful info that RPN would miss. Maybe introduce some sort of recurrency?

Is there anything out there that would be more tailored to my specific problem? Any advise on a better fitting architecture or further tweaks would be highly appreciated.

EDIT:
Some more extreme examples:


texts: Predict the salaries of customers in 2019 from 2016 data

texts:
I have a spreadsheet of data with ages and salaries for 2016. For example, in 2016 there were 12988 customers who were 18 years old and earned 20k.
I need to predict the salaries of customers for 2019. All I have are their ages but not their actually salary.

Since I do not have a lot of data (just 2016), I was thinking of simply making an estimate based on percentages. For instance, if 1% of all customers earned 20k, then I assume that 1% of the current customers also earn 20k
Should that be enough or is there a better way?


texts: Multiclass classification with high number of classes, high number of features and small sample size

texts:
I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age such as 10 or 35 will be individual classes. So roughly 80 classes (range from 10 to 90) are needed for this problem.

I immediately know that regularization is needed to shrink the number of features to prevent overfitting. I just don't know whether such a dataset can be treated as a multiclass classification problem with many number of classes. If I need more data, how many data will be enough for the model to learn? Or are there any clever ways I can do for this problem?


texts:
I want to know how are the image files accessed from the given path to load the tensors? I have attached the image to illustrate better.
In a previous cell, I have used this,
seed =42
random.seed = seed
np.random.seed = seed

Does the seed value has anything to do how files are accessed from the given path? If yes, how? If no, how does google colab decide the order in which to access files?


texts: Normalizing the final weights vector in the upper bound on the Perceptron's convergence

texts:
The convergence of the "simple" perceptron says that:

$$k\leqslant \left ( \frac{R\left \| \bar{\theta} \right \|}{\gamma } \right )^{2}$$

where $k$ is the number of iterations (in which the weights are updated), $R$ is the maximum distance of a sample from the origin, $\bar{\theta}$ is the final weights vector, and  $\gamma$ is the smallest distance from $\bar{\theta}$ to a sample (= the margin of hyperplane).

Many books implicitly say that  $\left \| \bar{\theta} \right \|$ is equal to 1. But why do they normalize it ?


texts: Need sugestion to accuire a job in data mining

texts:
I interested in data mining specialization and have been enrolled in data mining specialization course at Coursera by University of Illinois. This specialization course contains four parts. The very first course is "Data Visualization" which contains more conceptual knowledge rather than technical skills that not as expected.

Anybody can give me suggestion what can I expect after completed that course in the case to find a job in data mining specialization?

If I am intended to acquire engineering job, is this course right for me?