texts: Spark 1.5.1: Train many binary classifiers, save them, then use them on new data

texts:
I have a DataFrame representing an annotated dataset with 300 labels.

The DataFrame looks like follow (the first row is just to explain the columns):

binary Vector column for labeling, feature column
[0, 1, 1, 0, 0 , 0.... ], featureVec
[0, 0, 1, 0, 1 , 0.... ], featureVec
......

The labelling column, represent the presence or the absence of each label as an annotation of a feature vector (which we can see in the second column).

Please correct me if I am wrong: I can't train one multi-class classifier because the labels are not exclusives, so I would like to train one binary classifier per label.

As I have 300 labels, then I need to train and optimise 300 classifier (say logistic regression classifier for example).

What is the best way and best practices to train my classifiers with N-cross validation for parameter optimisation? any example code or reference is highly appreciated.

Once the classifiers are optimised, what is the best way and best practices to save the best models and use them to classify the new data?


texts: Python writing to Excel file: writerow() takes no keyword arguments

texts:
I have this script:
    import requests
    from requests import get
    from bs4 import BeautifulSoup
    import csv
    import pandas as pd

    f = open('olanda.csv', 'wb')
    writer = csv.writer(f)

    url = ('https://www......')
    response = get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    type(soup)

    table = soup.find('table', id='tablepress-94').text.strip()

    print(table)

    writer.writerow(table.split(), delimiter = ',')

    f.close()

When it writes to a CSV file it writes everything in a single cell like that:
Sno.,CompanyLocation,1KarifyNetherlands,2Umenz,Benelux,BVNetherlands,3TovertafelNetherlands,4Behandeling,BegrepenNetherlands,5MEXTRANetherlands,6Sleep.aiNetherlands,7OWiseNetherlands,8Healthy,WorkersNetherlands,9&thijs,|,thuis,in,jouw,situatieNetherlands,10HerculesNetherlands, etc.
I wanted to have the output in a single column and each value (separated by comma) in a single row.
I tried to use delimiter = ',' but I got:
TypeError: a bytes-like object is required, not 'str'.
Screesnhot of table:

How can I do this? Thanks!


texts: Can I apply Dropout In layers other than Fully Connected layers in CNN

texts:
I have read and seen that in CNN we apply DROPOUT layer between the FULLY CONNECTED layers to reduce overfitting.
Can we also apply the dropout layer between the CONV layers and the POOL layers. I have not seen models with this method applied. Will it help in overfitting when applied between these layers or are there any disadvantages to it?


texts:
I'm trying to run some analysis with some big datasets (eg 400k rows vs. 400 columns) with R (e.g. using neural networks and recommendation systems).
But, it's taking too long to process the data (with huge matrices, e.g. 400k rows vs. 400k columns).
What are some free/cheap ways to improve R performance?

I'm accepting packages or web services suggestions (other options are welcome).


texts: How to plot mean_test score and mean_train score of GridSearchCV

texts: How to plot mean_train_score and mean_test_score values in GridSearchCV for C and gamma values of SVM?


texts:
I wanted to check the definition of Discounted Cumulative Gain (DCG) measure in the original paper Jarvelin and it seems it differs from the one given in the later literature Wang. Originally, for $n$ documents ranked from $r = 1, \ldots, p$, the $\text{DCG}_p$ is defined as
$$\text{DCG}_p = \sum\limits_{r=1}^{b} G_r + \sum\limits_{r=b}^{p}\frac{G_r}{\log_br},$$
where $G_i$ is the relevance (or gain) of the $i$-th document. So the measure depends on the logarithm base $b$. For ranks below $b$, i.e. $r<b$, gains are not penalized. If $b=2$, then we can write:
$$\text{DCG}_p = G_1 + \sum\limits_{r=2}^{p}\frac{G_r}{\log_2 r}.$$
It does not look the same as the one given on wikipedia, where the argument of the logarithm is shifted by $1$:
$$\text{DCG}_p = G_1 + \sum\limits_{r=2}^{p}\frac{G_r}{\log_2(r+1)}.$$

Where does this change come from? Why others use different metric?


texts:
In our organization, there are many people who are into analytics and data science who are OK in their work with in the sense that they know what packages in R/Python etc to use, what algorithms to call for a particular type of problem etc. The problem is that they have very little knowledge of underlying mathematics and are not able to think beyond calling ready made functions i.e. they can never build a custom solution to specific problem. Let us call them as Function Callers as opposed a genuine Data Scientist who has a fair knowledge of the underlying mathematics.

We want to conduction a training course to teach the mathematical foundations of machine learning to the Function Callers and help them become Data Scientists. For this we have shortlisted some ten topics namely.

• Probability
• Various Probability distributions and data
• Maximum likelihood methods
• Linear Algebra (advanced)
• Single variable calculus, Multi variable calculus
• Vector calculus
• Graphical models
• Bayesian networks
• Optimization techniques
• Statistical models

This is going to be an 80 hour course so it is not possible to cover everything in detail like a college/university course.

Question: Assume that the course will have enough time to teach and implement only 3 or 4 things in each of the topics mentioned above. So given this background, what are the best 3 or 4 things that can be covered under each topic?

Note: The target audience have either science or college level mathematics background but they do not have deep knowledge about the mathematics that is used in machine learning.


texts:
Given a value $y$ and some values $x_1, ..., x_n$.
How do I find weights $w_1, ... w_n$ so that the $error = y - w_1*x_1 + ... w_n*x_n$
is minimal, where the weights have to sum up to 1 or a different value?


texts:
What is the Square-law based RBF kernel (SQ-RBF)? The definition in the table at the Wikipedia article Activation Function looks wrong, since it says

y =
     1 - x^2/2     for |x| <= 1
     2 - (2-x^2)/2 for 1 < |x| <= 2
     0             for |x| > 2

but this makes it discontinuous at x = 1


texts: I have a non linear data set, and I am using SVM (RBF kernel) to build a classification model, but not sure how to set the best hyperparameters of the SVM, C and gamma in Matlab fitcsvm.  And is it acceptable if it is trial and error approach setting arbitrary values, until finding the best performance?


texts:
I use python 3.7.3 and I run this code on Jupyter Notebook .

I have a column named 'image' in my Dataframe named df that contains Url of images I wanna visualize inside the df.

I saw some answers and I tried to use this code:

from IPython.display import Image, HTML

def path_to_image_html(path):

    return '<img src="'+ path + '" style=max-height:124px;"/>'

HTML(df.to_html(escape=False ,formatters=dict(image=path_to_image_html)))

However, I get this result instead:

Can anyone tell me how I get the images using this code or any other code.


texts: Why use convolutional NNs for a visual inspection task over classic CV template matching?

texts:
I had an interesting discussion come up based on a project we were working on: why use a CNN visual inspection system over a template matching algorithm?

Background: I had shown a demo of a simple CNN vision system (webcam + laptop) that detected if a particular type of object was "broken"/defective or not - in this case, a PCB circuit board. My CNN model was shown examples of the proper and broken circuit boards (about 100 images of each) on a static background. Our model used the first few conv/maxpool layers of pre-trained VGG16 (on imagenet), and then we added a few more trainable convs/pools, with a few denses, leading to a dim-3 one hot encoded vectored output for classification: (is_empty, has_good_product, has_defective_product).

The model trained pretty easily and reached 99% validation acc no problems; we also trained with various data augmentation since we know our dataset was small. In practice, it worked about 9 times out of 10, but a few random translations/rotations of the same circuit board would occasionally put it in the opposite class. Perhaps more aggressive data augmentation would have helped. Anyways, for a prototype concept project we were happy.

Now we were presenting to another engineer and his colleague, and he brought up the argument that NNs are overkill for this, should just use template matching, why would one want to do CNNs?

We didn't have a great answer for why our approach could be better in certain applications (e.g. other parts to inspect). Some points we brought up:

1) More robust to invariances (through e.g. data augmentation)

2) Can do online learning to improve the system (e.g. human can tell the software which examples it got wrong)

3) No need to set thresholds like in classical computer vision algorithms
What do you guys think, are there more advantages for a CNN system for this type of inspection task? In what cases would it be better than template matching?

A few more random ideas for when deep NNs could be the tech for the job: for systems that require 3D depth sensing as part of the input, or any type of object that can be deformed/stretched/squished but still be "good" and not defective (e.g. a stuffed animal, wires, etc). Curious to hear your thoughts :)


texts: Model inclines because of imbalanced data

texts:
In life, some events are rare and most cases are normal. So I am wondering, to detect the rare cases, shall we use an imbalanced dataset with more historical rare cases?
Taking the German Credit Data, as an example. It contains data on 20 variables and the classification whether an applicant is considered a Good or a Bad credit risk for 1000 loan applicants. 70% are Good. 30% Bad.
With this original data set, I assume the model would incline to better recognize the normal cases (because more normal cases in the data)
If a balanced dataset is used, i.e. the number of good credits equals to that of bad credits, a final model will be good at predicting both ‘Good’ and ‘Bad’.
But if we want to use machine learning to recognize the rare events, e.g. in this case, the bad credit customers. Shall we use an imbalanced dataset (for example 70% Bad credits, 30% Good in total 1000 records), that contains much more bad credit customers, than good ones, so the final model is good at recognizing the bad customers? Or a balanced dataset is always necessary (also the only right way).
Can anybody please shed some light on this?
Thank you.
Link to data: https://online.stat.psu.edu/stat857/sites/onlinecourses.science.psu.edu.stat857/files/german_credit/index.csv


texts: Clustering with cosine similarity with specific threshold (in Python)

texts:
Given :

1, either given the origin data matrix X , such as X = [[1,1,1,1],[1,1,-1,-1],[1,1,1,-1]]

or given the cosine similarity matrix of the original data X , which is sth like
sklearn.metrics.pairwise_distance(X, metric='cosine'),

2, a specific threshold Delta = 0.1 (If the cosine similarity between two vectors is smaller than 0.1, then put them into the same cluster)

Problem: cluster these data, so that in the same cluster the cosine similarity (of the data in this cluster) < Delta

Thanks


texts:
So I've seen this:
Keras LSTM with 1D time series
And this: Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have many, many, many accountIDs, and 40 or more features associated with them for the start of each week since 2017.
I'm attempting to predict expected profit next week (next time step) for a given accountID with a given history (the idea is to modify strategy features, pass them both to the RNN, and see which one has the higher expected profit once the model is trained).

The suggestion provided in the second one I linked is that I make a sub-model for each accountID it seems, but I have many of them and new ones come in all the time.

Should I strip out accountID, but still pass its whole history in as a sequence, and then just append it back after prediction?

What if I only have a few weeks of data for some accounts, and years for others?

The dataset would look roughly like this: (weekstart, accountID, x, y, z, p, q, ...), and each accountID would have many tuples like that, but they wouldn't all have the same number of tuples.

For reference, I'm familiar with CNNs and standard neural networks, but this is my first attempt at using an RNN.


texts: What kind of images/objects are the most easy for neural networks to detect?

texts:
I need to design a marker image that should be detected by the neural network. I am aware that it is not a complex task just to detect an image and this can be done with OpenCV alone. However the detector but this should also work also in pure visibility conditions (low light, out of focus, worn out/partially damaged, makes tiny part of the image only, etc). The background around the marker image is just the real world, difficult to predict what it could be there.

What kind of images are the most easy for neural networks to work with? Most important

Should it have sharp contrast like letters or road signs or color gradients may work better?
Would repeating pattern (like chevron frame around) as part of the object help or better to have every piece of the tag unique?

I know that neural networks are mostly used to the "real world" images like humans and horses, but I am not sure if they are better objects or just it is more impressive to get them recognized.

From the other side, objects that humans make on purpose just to recognize them (like road signs) mostly use sharp contrast. I do not remember any road or other sign with any kind of color gradient but repeating pattern like chevron frame around is sometimes used. From the other side, a road sigh with color gradients is more complex to manufacture, and maybe the technology simply did not exist when they were first invented.

Is there any consensus which kind of objects are the most easy to recognize by neural networks under "noisy input" conditions?


texts: Stochastic gradient descent based on vector operations?

texts:
let's assume that I want to train a stochastic gradient descent regression algorithm using a dataset that has N samples. Since the size of the dataset is fixed, I will reuse the data T times. At each iteration or "epoch", I use each training sample exactly once after randomly reordering the whole training set.

My implementation is based on Python and Numpy. Therefore, using vector operations can remarkably decrease computation time. Coming up with a vectorized implementation of batch gradient descent is quite straightforward. However, in the case of stochastic gradient descent I can not figure out how to avoid the outer loop that iterates through all the samples at each epoch.

Does anybody know any vectorized implementation of stochastic gradient descent?

EDIT: I've been asked why would I like to use online gradient descent if the size of my dataset is fixed.

From [1], one can see that online gradient descent converges slower than batch gradient descent to the minimum of the empirical cost. However, it converges faster to the minimum of the expected cost, which measures generalization performance. I'd like to test the impact of these theoretical results in my particular problem, by means of cross validation. Without a vectorized implementation, my online gradient descent code is much slower than the batch gradient descent one. That remarkably increases the time it takes to the cross validation process to be completed.

EDIT: I include here the pseudocode of my on-line gradient descent implementation, as requested by ffriend. I am solving a regression problem.

Method: on-line gradient descent (regression)
Input: X (nxp matrix; each line contains a training sample, represented as a length-p vector), Y (length-n vector; output of the training samples)
Output: A (length-p+1 vector of coefficients)

Initialize coefficients (assign value 0 to all coefficients)
Calculate outputs F
prev_error = inf
error = sum((F-Y)^2)/n
it = 0
while abs(error - prev_error)>ERROR_THRESHOLD and it<=MAX_ITERATIONS:
    Randomly shuffle training samples
    for each training sample i:
        Compute error for training sample i
        Update coefficients based on the error above
    prev_error = error
    Calculate outputs F
    error = sum((F-Y)^2)/n
    it = it + 1

[1] "Large Scale Online Learning", L. Bottou, Y. Le Cunn, NIPS 2003.


texts: Positive/negative training sample imbalance in multi-label image classifiers

texts:
I'm trying to train VGG-16 on the Pascal VOC 2012 dataset, which has images with 20 labels (and a given image can have multiple classes present). The examples are highly imbalanced, so I've "balanced" them such that each label is represented roughly equal in the training set.
But this means that for each label, 5% of the total images are positive examples and 95% are negative samples. There is no way to achieve a 50/50 split for all classes.
I'm using binary cross entropy loss and a sigmoid activation at the final VGG layer, since this is a multi-label problem. Binary accuracy looks great but in fact, the results for any given class are pretty dismal (~15% recall). The classifier is not fitting to positive examples and is biased toward reporting a negative result because that matches the data distribution (very few positive samples).
What is typically done in this scenario? The original paper appears to train on mutually-independent classes. Should I be using a custom loss function?


texts: How to use "sann" function in R to solve graph problems?

texts:
I came across a package in R which has a function called sann for simulated annealing.

sann uses parameters fn and gr to optimize and to select new points, respectively.

For something like the max-clique or max-stable set problems, fn would be a summing function, but it's less clear how one would formulate gr to fix these graph computations. In these cases, how would gr "select"?


texts:
Why do I have a non-zero result in the following calculation:

  0.11-0.1-0.01
  [1] -5.20417e-18

Seems to be an easy and akward question, but I will be very greatful for any help!)


texts:
I am building a disease classifier. I have Dicom scans for many patients.
The scans have different slice thickness, and different scans have different number of slices. However, the slice thickness for a single patient's scan has the same thickness for all slices.

For example:

Patient1's scan : 100 slices of 2mm thickness

Patient2's scan : 500 slices of 1mm thichness

The number of pixels for each slice is 512 x 512,
so currently the shape of nd array containing the information for

Patient1 is 100*512*512

Patient2 is 500*512*512

I want to pass all patients information into a CNN.
Do I need to resample the slices and make it uniform for all patients (like 512*512*n). If yes, how to do it, and what should be the value of 'n'?


texts: Why is my TensorFlow 2.1 tensor returned as "dtype=resource"?

texts:
I have created a tensor:
tensorX= tf.compat.v1.get_variable("tensorX", dtype=tf.float32,
                                        initializer = tf.compat.v1.constant(100.0))

#.....and when I read this tensor from the graph i.e.

graph = tf.compat.v1.get_default_graph()
tensorXX = graph.get_tensor_by_name("tensorX:0")
print(tensorXX)

#Output
tensorX:0' shape=() dtype=resource

I have few related questions:

What is dtype=resource in tensorflow (version 2.1)?
While reading this tensor_by_name, why is it returning a "resource"?
Due to its "dtype=resource", I'm not able to assign some value to it. How should I read this tensor where "dtype=tf.float32" or how do I convert "dtype=resource" back to "dtype=tf.float32"?


texts: What is the formula for calculating "feature importance"?

texts:
I have a graph of the feature importance. What is the formula for calculating these values?

I got these values after training and testing the model using
xgb.plot_importance (model)

I need a formula to describe mathematically how the calculation is done


texts: K in the Naming of Models and Techniques

texts: Why is k chosen for the names k-nearest neighbors and k-fold cross validation? Is it arbitrary or just another mysterious naming?


texts: Binary classification based on pairwise relationships

texts:
I have a seemingly esoteric machine learning problem and I'm not sure where to start looking at literature.

I am trying to train a binary classifier that accepts/rejects each object in a collection of N objects. As input, it is given feature vectors describing each object, as well as feature vectors describing pairwise relationships between each object. The list of pairwise relationships is not complete, but not very sparse: approx 1/3 of the possible pairs are described.

The ground truth is the accept/reject label of each object, and I have a set of collections to train on. From domain knowledge I know that the global properties of the entire collection are important to determining the classification of the object, which is why I would like to use the pairwise relationships in classification.

Currently to classify the objects we are using a greedy algorithm that uses a rule-based score based on the relationship measure. (We start with an object that we trust to be very good, then we greedily add the other objects until no more objects have a positive total relationship score in relation to the already-accepted subset) So we know that the relationship measure is important and will be present in the test data. I am attempting to improve on this by using a machine learning approach.

My first attempt was to treat each pairwise relationship I have as a data point, and concatenate the two self-descriptor vectors to the relationship vector. Then, I labeled this as "1" if they were an accept-accept pair, or "-1" if it was an accept-reject pair. Then, during testing, I could put these predictions in one affinity matrix for each collection (if the pairwise relationship is unknown, it that matrix entry is left as 0) and use spectral clustering ("accept" and "reject" objects would have opposite signs in the first eigenvector). However, this performed worse than using only the self-descriptors for classification. I suspect this is because there is a considerable deficiency in this training protocol -- unlike the rule-based score we currently use, there is no indicator of how strong the agreement or disagreement between the two objects is.

Therefore, I would like to create a classifier that looks at the entire collection of self-descriptors and pairwise relationships, and uses the accept/reject labels directly as ground truth somehow.


texts: Differentiating between normal and attack clusters

texts:
In the below figure, we plotted some data of sensors in normal condition and under attack(outlier):
1.  Green points are normal samples in the training dataset.
2.  Cyan and red points are normal samples in the test dataset.
3.  Black and blue points are attack samples in the test dataset.
We wish to classify attack from normal samples in the test phase by using only the information we extract from the training dataset (the green points). I highly appreciate any suggestions for this purpose.

Thanks.


texts: Predictive modeling based on RFM scoring indicators

texts:
RFM - is a ranking model when all customers are ranked according to their purchasing F requency, R recency and M monetary value. This indicator is highly used by marketing departments of various organizations to segment customers into groups according to customer value.

The question is following: are there any substantial models based on RFM scoring (or related to) which have solid predictive power?

Update:

predicting which customer will most likely spend more
who is going to upgrade/renew subscribtion/refund etc

Update2:

I understand, this is simple problem with three independent variable and one classifier. My guess and experience say these pure three factors do not predict future customer value. But they can be used together with another data or can be an additional input into some model.
Please share which methodologies worked for you personally and are likely to have high predictive ability. What kind of data you used together with RFM indicators and it worked well?


texts:
I am solving regression problem now and unfortunately I'm stuck.

I have a dataset containing train, test dataframes (~5000 rows each) and images with common ID with dataframes records.

First approach was to use different machine learning regression algorithms (sklearn) to build model that uses only tabular data, but it had poor accuracy. Now, I'm 100% sure that the only way to boost accuracy is using stacked deep leaning model.

I've used this tutorial and built a model that includes CNN for image inputs, MLP for tabular data and Dense layer with linear activation function for branches outputs.

It's structure looks something like this:

----- Images ---- Dataframes ----
--------- | ----------------- | ------------
------ CNN ------------- MLP ---------
------------ \ ------------ / ---------------
--------------- Dense ------------------
------------------- | ----------------------
--------------- Output -----------------

It is implied that this model should have inputs for each branch, but there are only 20 images per dataframe (40 in total; 20 for training; 20 for testing). The problem is that I can improve accuracy (using this model) only for 20 records out of 5000 (0,4 %) what is quite useless.
Another point is that ML algs have better scores in comparison to MLP working with tabular data.

What we have:

ML algs have better accuracy than MLP
Stacked keras model has better accuracy than ML algs, but there're not enough images

The only way of solving that problem (that I see) is to use stacked model for IDs that have images and Ml algs for other IDs and then just concatenate the results, but it won't give me decent boost.

Is it somehow possible to fit this stacked model with images and tabular data and use it to predict targets with tabular data only?
Is there any approach that I just don't see?
Can you recommend different models or solutions?


texts: Which Format is Faster, Matlab (.mat) or NumPy (.npy)?

texts:
I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.

I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").

It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.


texts:
Can I make the following statement about a binary classification, please?

Precision 1: 0.10
Recall 1: 0.83

Statement: "We can expect 90% false alarms (1 - 0.10). But for the remaining 10%, we can be around 83% certain (Recall 1: 0.83), that we caught a label 1."

Thanks in advance!


texts: Looking for an advice based on an output of a LSTM model

texts:
I have fitted a LSTM model first using keras.
Data:I have a time series with 560 observations. From that I trained the model using first 500 observations and then evaluated the model using last 60 observations.
I have only one predictor(X) where X is the lag price of the time series. So basically I am trying to predict the current price from the previous days price by fitting a LSTM model.
I used the following setup to fit the model:
model = Sequential()
model.add(LSTM(32,  activation = 'relu', return_sequences = True,input_shape=(dataXtrain.shape[1], dataXtrain.shape[2])))
model.add(Dense(1))
model.add(Dropout(0.3))

model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(dataXtrain, dataYtrain, epochs=1000, batch_size=16)

Then I evaluated the model for test data. The comparison between the actual price and the predicted price for test data looks like this:

Based on the above plot, It seems that the model is kind of capturing the shape of the actual price. But it is underestimating the price all the time.
Can anyone suggest anything to improve this model?
I am beginner to model fitting in deep learning. So any help will be highly useful.
Thank you.
Update
I tried following setup and the results improved drastically:
model = Sequential()
model.add(LSTM(50,  activation = 'relu', return_sequences = True,input_shape=(dataXtrain.shape[1], dataXtrain.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(dataXtrain, dataYtrain, epochs=1000, batch_size=16)

However it seems that the predicted price capturing the actual trend after some delay. Will it be possible to fix this?
Any advice will be really helpful.
*Edit
I am updating the question with the code that I used to obtain the fitted values and make the plot.
First I Stored the predictions based on test data:
 y_pred = model.predict(dataXtest)
    y_pred.shape

When I am fitting the model, I reshaped the data so that it is a subsequence of 2 observations. (For an example 500 training observations are reshaped as (250,2,1). Also the 60 test data reshaped as (30,2,1) )
Then I reshaped the predictions to the original form.
y_pred=y_pred.reshape(60,1)
dataYtest=dataYtest.reshape(60,1)
pred_data=pd.concat([dataYtest,y_pred],axis=1)
pred_data.head(10)

0   5933    5837.184570
1   5991    5851.091797
2   6023    5947.239258
3   5972    5939.657227
4   6028    5928.400391
5   5429    5944.728516
6   5697    5390.026367
7   5841    5619.818359
8   5980    5798.516602
9   6548    5897.926758
10  6928    6499.494141

This is how I plotted the data:
plt.figure(figsize=(14,5))
plt.plot(dataYtest, color = 'red', label = 'Actual Price')
plt.plot(y_pred, color = 'blue', label = 'Predicted Price')
plt.title('Test data')
plt.xlabel('Time')
plt.ylabel('Price')
plt.legend()
plt.show()


texts: Reduce drug spend using medical claims analytics

texts:
I have access to medical claims data from a US health insurance company.

I believe there's an opportunity to find some cost savings by switching the site of service from high cost outpatient facilities to possibly home infusion or physician for some procedures.

An example would be that Botox costs(in dollars) the company 10000 per claim at the outpatient facilities compared to 3000 at the physicians office.

I am interested in learning about how a data scientist would approach this problem of reducing costs from medical claims.

Could I run a cluster analysis on the population of claims to see who is most similar to the existing Botox patients who go to their physician and target these to move them to the lower cost sites of service?


texts:
I was thinking of creating a CNN. Now it is known CNN takes long times to train so it is advisable to stick to known architectures and hyper-parameters.

My question is: I want to tinker with  the CNN architecture (since it is a specialised task). One approach would be to create a CNN and check on small data-sets, but then I would have no way of knowing whether the Fully Connected layer at the end is over-fitting the data while the convolutional layers do nothing (since large FC layers can easily over-fit data). Cross Validation is a good way to check it, but it might not be satisfactory (since my opinion is that a CNN can be replaced with a Fully Connected NN if the data-set is small enough and there is little variation in the future data-sets).

So what are some ways to actually tinker with CNN and get a good estimate for future datasets in a reasonable training time? Am I wrong in my previous assumptions? A detailed answer would be nice!


texts: useful method for analyzing the real-time (online learning) time series? forecasting and decomposition?

texts:
Which method is useful for analyzing the real-time time series? even for forecasting or decomposition?
I have a large dataset (sequential data) that need for predication analysis and decomposition analysis of this data Realtime. I need to make a decision in less than 0.1ms (response time<0.1ms). The size of data is 20 MB/hour.
What is the ML method you suggest to solve this type of data?


texts: Using a custom R generator function with fit_generator (Keras, R)

texts:
I'd like to train a convolutional network to solve a multi-class, multi-label problem on image data. Due to the nature of the data, and for reasons I'll spare you, it would be best if I could use a custom R generator function to feed to the fit_generator command, instead of its built-in image_data_generator and flow_images_from_directory commands (which I was successfully able to get working, just not for this particular problem).

Here (https://www.rdocumentation.org/packages/keras/versions/2.2.0/topics/fit_generator) it says that I can do just that, without giving any examples. So I tried the following. Here is an extremely stripped down example of what I'm trying to do (this code is entirely self contained):

library(keras)
library(reticulate)      #for py_iterator function

play.network = keras_model_sequential() %>%
  layer_dense(units = 10, activation = "relu", input_shape = c(10)) %>%
  layer_dense(units = 1, activation = "relu")

play.network %>% compile(
  optimizer = "rmsprop",
  loss = "mse"
)

mikes.custom.generator.function = function()     #generates a 2-list of a random 1 x 10 array, and a scalar
{
  new.func = function()
 {
    arr = array(dim = c(1,10))
    arr[,] = sample(1:10, 10, replace = TRUE)/10
    return(list(arr,runif(1)))
  }
}

mikes.custom.iterator = py_iterator(mikes.custom.generator.function())              #creates a python iterator object

generator_next(mikes.custom.iterator)                 #correctly returns a 2-member list consisting of a 1 x 10 array, and a scalar
generator_next(mikes.custom.iterator)[[1]]            #a 1 x 10 array
generator_next(mikes.custom.iterator)[[2]]            #a scalar

#try to fit with "fit_generator":

play.network %>% fit_generator(                       #FREEZES.
  mikes.custom.iterator,
  steps_per_epoch = 1,
  epochs = 1
)

The thing freezes at training time, without giving me an error message or anything. I also tried it with a custom image data generator for my original problem, same result.

Note that this network trains just fine if I just use fit and input the training data manually:

play.network %>% fit(generator_next(mikes.custom.iterator)[[1]],generator_next(mikes.custom.iterator)[[2]], epochs = 1, batch_size = 1)

#trains just fine

I think I know the problem, but I don't know the solution. If you ask it for the class of my custom iterator, it gives

class(mikes.custom.iterator)

[1] "python.builtin.iterator"  "rpytools.generator.RGenerator" "python.builtin.object"

whereas if I build an iterator using the builtin image_data_generator and flow_images_from_directory commands, it gives

train_datagen <- image_data_generator(rescale = 1/255)

class(train_datagen)

[1] "keras.preprocessing.image.ImageDataGenerator" "keras_preprocessing.image.ImageDataGenerator" "python.builtin.object"

train_generator <- flow_images_from_directory(
  train_dir,
  train_datagen,
  ....
)

class(train_generator)

[1] "python.builtin.iterator" "keras_preprocessing.image.DirectoryIterator" "keras_preprocessing.image.Iterator"        "tensorflow.python.keras.utils.data_utils.Sequence" "python.builtin.object"

So my guess is that train_datagen and/or train_generator have attributes that mikes.custom.iterator does not, and fit_generator is trying to call upon mikes.custom.iterator using functions other than the basic generator_next (which is in theory all it should really need). But I don't know what they may be, or how to build mikes.custom.iterator correctly, even after searching for two hours online.

Help anyone?


texts:
In the setting of Variational AutoEncoders, i.e. when we want to find the posterior distribution

over the data generating, latent variable z, given some observations x, what exactly (which part of the equation) makes this posterior distribution intractable and why?
Cheers


texts: What is the meaning of Information with respect to interpretability approaches in machine learning?

texts:
I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - LIME and COVAR was done based on their ITR(Information Transfer Rate) defined as

and continued in

I am having difficulty in understanding how this information is defined for humans and for the machine learning algorithm and hence the mutual information in the formula provided. I'm not even sure if I've asked correctly but it's not clear to me how are these 2 algorithms being compared using the formula mentioned above.


texts: Which is the most appropiate algorithm to use with Mlib for predicting prices

texts:
I am reading the docs of MLlib, but I am new with it and I don't know which would be the best algorithm to predict prices.

I have seen that with RandomRDDs I can use a generator for random numbers that follow a normal distribution.

In my case, this is not good enought, since the prices I want to simulate doesn't follow a normal distribution. The demand changes depending on if its daytime or night, so the prices also change. The seasonality also affects the demand and influence the resulting prices.

Because of that, I would need a model that I can feed with historical data in order to generate new prices for simulations that have "sense" depending on the day of the year or the hour (day/night) I want to simulate.

As I said, I'm looking for a solution with MLlib since I'm working with Spark. I'm blocked in finding the appropiate algorithm when reading the docs. I only would need a name or a few names in order to continue investigating (I, myself, can enrich later the answer in order to help other people that reach this post)


texts:
I downloaded:

!git clone https://www.github.com/matterport/Mask_RCNN.git
os.chdir('Mask_RCNN')

And I've got an error:
which version I should have of Keras?

WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py:1154: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead

Futhermore:

totalMemory: 5.94GiB freeMemory: 5.44GiB
2019-04-03 22:37:38.374934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2019-04-03 22:37:40.343417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-03 22:37:40.344366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1
2019-04-03 22:37:40.344373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N N
2019-04-03 22:37:40.344377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   N N
2019-04-03 22:37:40.345556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11435 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0, compute capability: 5.2)
2019-04-03 22:37:40.450785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 5220 MB memory) -> physical GPU (device: 1, name: GeForce GTX TITAN Black, pci bus id: 0000:01:00.0, compute capability: 3.5)
2019-04-03 22:37:42.518519: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:42.601229: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:51.648032: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:51.678817: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:51.706928: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
[I 22:37:55.611 NotebookApp] Starting buffering for fa2cd5ca-20f3-4472-b6ca-6821e2f56118:02508f46d629494ab46babe6d7611656


texts: Calculating error in each layer of neural network

texts:
I am referring Andrew Ng's course to implement neural network https://www.youtube.com/watch?v=x_Eamf8MHwU&t.
In this course bias is taken single matrix with weights.
I got a error and I am not able to figure in days what error I am doing.
Neural network I'm trying to simulate is 2 node in input layer,3 in hidden and 2 in output layer.
Here is the code
def feed_forward(a):
    for w in theta:
        a=np.concatenate((one,a))
        a=sigmoid(np.dot(w,a))
        activation.append(a)
    delta[network_length-2]=error(a,y)

#delta is the error in each layer

def back_propagate():
    for l in range(network_length-3,-1,-1):
        delta[l]=np.dot(theta[l+1].T,delta[l+1])*sigmoid_prime(activation[l])

I get error in last line saying
  delta[l]=np.dot(theta[l+1].T,delta[l+1])*sigmoid_prime(activation[l])
ValueError: operands could not be broadcast together with shapes (4,1) (3,1)

I don't know why activation and np.dot(theta[l+1].T,delta[l+1]) dimension not matching


texts: When should the last action be included in the state in reinforcement learning?

texts:
I am having some confusion as to whether the action should be included as part of the state input to an agent in a reinforcement learning setting (state-action pair). As from my observation, this is not completely clear as different agents/environments combinations might have different performances if action was included/excluded from input states (I might be wrong).

For my specific problem:

the agent can't influence/control the states through its actions (similar to the case of a simple multi-armed bandit)
the action space is discrete
I am using a DQN based approach

I would also appreciate a general overview/rules of thumb of when to include/exclude actions as state inputs.

ps. when i say "different agents/environments combinations" in the beginning I mean using different agents to solve the same env or same agent to solve different env.


texts: Using R to organize/rearrange CSV - group by multiple columns?

texts:
I have a CSV that I need to clean up / organize in a usable way using R. I need to group by the property ID and then want to take all the unique years for the defor year column and make each year into a sperate column with the amount of deforestation for that year. My data frame / CSV looks like this:
Prop_ID deforYear   deforHA
1           2010         15
1           2011         0
1           2012         10
2           2010         35
2           2011         45
2           2012         0

and I want the output to look like this:
Prop_ID defor_2010  defor_2011 defor_2012
1               15           0         10
2               35          45          0

I'm assuming I use the group_by function in dplyr but I can't seem to figure it out.


texts: How to construct a Binary Tree from a decision tree with branching factor greater than 2 (b>2)?


texts: Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split

texts:
I'm trying to use GridSearchCV from scikit-learn and look at the difference between train/test metrics.

When I do a normal test/train split with RandomForestRegressor, the metrics are comparable.  Something like:
Train R2: 0.97
Test R2: 0.85

However, when I try to use the same data with GridSearchCV, the testing and training metrics seem to be completely different, the Test accuracy is
a large negative number instead of being something between 0 and 1.

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV

...

rf_reg = RandomForestRegressor(max_depth=10,
                              random_state=RANDOM,
                              n_estimators=100,
                              n_jobs=N_JOBS)

param_grid = {
              "min_samples_leaf": [1, 2, 4, 10]
             }
grid_cv = GridSearchCV(rf_reg, param_grid, cv=5, return_train_score=True)
grid_cv.fit(X, y)

I'm surprised that when I examine the scores for testing and training, they appear to be two different metrics.  The default score for RandomForestRegressor is R2, but the results for the test sets look like they're another metric entirely.

results = pd.DataFrame(grid_cv.cv_results_)
print('Train scores:\n', results['mean_train_score'])
print('Test scores:\n', results['mean_test_score'])

Train scores:
0    0.974572
1    0.963771
2    0.936328
3    0.877382
Name: mean_train_score, dtype: float64

Test scores:
0   -5.948434
1   -5.798446
2   -6.034835
3   -6.655515
Name: mean_test_score, dtype: float64

The train scores make sense to me, they should be between 0- 1 because I'm expecting R2 error metrics, the default for a RandomForestRegressor.  But why are the test scores a different metric? They should also be between 0 and 1, how is it possible to get negative numbers?  This does not make sense to me.

The same thing also happens with cross_val_score, I'm expecting an R2 metric, but it returns negative numbers. Even explicitly setting the scoring method to 'r2' returns negative numbers.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(rf_reg, X, y, cv=5, scoring='r2')
print(scores)
[ -5.15970579  -4.67536964 -13.17643335  -2.11630272  -4.51688508]


texts:
This is a similar question like the Statistics Conferences question at CrossValidated

What are the most significant annual Data Science conferences?

Rules:

Include a link to the conference
Please include links for the talks (be it youtube, the conference site or some other video streaming site)


texts: Search the Number of occurrences of the particular words in data using Pandas.

texts: I wanna Search how many times "bad"  and "good" words are repeated in the data frame and visualize  this with histogram.


texts: predicting probability distribution for time series

texts:
I have time series of several variables. Just in one specific case one variable is linear combination of the rest.

I want to predict probability distribution (that is not only best estimate but estimates with probability of that happening) of future value of variables. I want to see when probability of small interval of possible outcome is high.

A priory I don't know rules of the game how variables evolve and inter-depend.

What is the tool to best do such prediction and how easy is it? Will scikit-learn do? Maybe neural networks?

ADDED based on answer:

As I've understood, time series theory data science assumes random walk, however I assume variables are at least partly moved by free will of players of the game under some a priory unknown to me constraints and goals.

Should statistical solution advised work in such case? Can data predict reversal of possible current trend?


texts: Database/warehouse for storing 128D feature vectors

texts:
I use openface to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted features and those stored in the database).

Here is an example of what I'm trying to do:

I use OpenFace to extract the face representation, this is a 128D feature vector.
I then calculate the euclidean distance between this vector and those stored in the database, returning the vector that has the lowest distance (less than 0.9) as a match.
If no vector stored in the database matches this criteria, I store the newly extracted features as a new entry in my dataset.

What is a good database engine to achieve this?


texts:
I have a few basic questions about tracking losses during training.

If I am using mini-batch training, should I validate after each
batch update or after I have seen the entire dataset?
What should be the condition to stop the training to prevent overfitting? Do you save the model at that point?
In case I use mini-batch training losses fluctuate a lot, depending on the random choice of training data, and sometimes validation loss is less than training loss. Is this normal? I think my confusion about this point will be answered in 1 by itself.


texts: Linear Regression - finding thetha using Normal equation

texts: This is to find thetha which will give minimum cost function. Why is the x0 column required? why cant we assign size as x0? why do we need the feature count to be n+1?


texts: How to design optimizer for combined model training in Pytorch

texts:
I am trying to train a embedder. So, I have an architecture for the model to embed texts. And I have another model architecture that will take the inputs from the output of the first model and predict the label.
Now at the end of the training I want the first model to be saved as a pre-trained embedding model.
So, the pseudo code is as follows -
Model1:
input: text
… some BiLSTM , Dense layers…
output: n-dimensional embedding\
Model2:
input: Model1(text1), Model1(text2)
find the similarity of texts
run a linear layer + sigmoid
output: similarity score
Now I want to use a dataset with the following structure - (text1, text2, similarity). I want to train the models and save Model1 as a pretrained model to use later.
How can I train these two with PyTorch? How shall I write the optimizer here?


texts:
im building an mlp with scikit learn. Is there a way I can access weights and biases of the output layer per iteration? There is an option
    mlp.coefs_ But it outputs the train weights when the model is compiled


texts:
I am currently reading this paper on Isolation Forest.

At page 3, there is a definition of Isolation Tree and there are a couple of sentences that I don't understand:

  Given a sample of data X = {x1, ..., xn} of n instances from a
  d-variate distribution, to build an isolation tree (iTree), we
  recursively divide X by randomly selecting an attribute q and a split
  value p, until either: (i) the tree reaches a height limit, (ii) |X| =
  1 or (iii) all data in X have the same values.

Here, I would like to understand what do the points (ii) & (iii) mean. In the point (iii), when the author says 'all data in X have same values', does it mean the same anomaly score?

And there is also a block of text which I am failing to understand completely. Can someone help me understand this text too..

  Assuming all instances are distinct, each instance is isolated to an
  external node when an iTree is fully grown, in which case the number
  of external nodes is n and the number of internal nodes is n − 1; the
  total number of nodes of an iTrees is 2n − 1

Thanks


texts: How to apply Hypothesis test on time series?

texts:
I am working on a project that is related to Wikipedia's revision history. I have around 10 time-series and each one represents how the relative frequency of positive words changes over time.
I need to test the hypothesis, that the batch of females samples have larger relative frequencies than the batch of males samples. But, this I want to hold over time.
I have tried ANOVA, but I am not sure if it is the proper way and how to interpret the results, because I am testing only one value which is the average over time.
What is the proper way to apply hypothesis tests on time series data?


texts: Video segmentation vs image segmentation

texts:
I am new to data science & working on a segmentation model, Basically I need to deploy this segmentation model in android devices using TensorFlow-Lite for real time camera frame segmentation.
I used unet model to do that but could not get the accuracy I wanted. After exploring so much I found something about video segmentation but I am bit confuse How video segmentation is different from normal image segmentation? Can somebody explain the differences between these two?


texts: Random Forest command in R for selected variables

texts:
All I can find online is how a random forest is run on all variables in a dataset using the period:
RF <- randomForest(sale ~ ., data = TrainSet, importance = TRUE)
What if I only want to apply the RF to selected variables in the dataset? Would I need to drop the variables first?
I tried the following, but get an error:
RF <- randomForest(sale ~ v1,v2,v3, data = TrainSet, importance = TRUE)


texts: What are some good loss functions used to minimize extreme errors in regression and time series forecasting?

texts:
E.g. In detriment of a smaller mean error, I want to have fewer big mistakes

I'm working on a time series forecasting task and in some specific cases I don't need perfect accuracy, but the network cannot by any mean miss by a lot.

Any suggestions of loss functions or other methods to solve this issue?


texts: How can I vectorize this code in R? Maybe with the apply() function?

texts:
I am really struggling to replicate the output of the dist() function in R code without using 1 or 2 for loops. (If you're wondering why I'm doing this, it's so that I can play around with the distance calculation, and also to improve my R skills - so please only solutions that involve R!)

Overview: matrix is passed to dist(), which calculates the Euclidean distance row-wise and outputs a full distance matrix of the distance between each row (e.g. distance between rows 1 and 50 will be in distancematrix[1, 50] and distancematrix[50, 1]). The fast code looks like this:

distancematrix <- as.matrix(dist(myMatrix, method="euclidean", diag = T, upper = T))

I have successfully produced the same output in R using the following code:

for (i in 1:nrow(myMatrix)) {
  for (j in 1:nrow(myMatrix)) {
    distancematrix[i, j] <- sum(abs(myMatrix[i,] - myMatrix[j,]))
  }
}

However, using two nested for loops is much slower than using dist(). I have read a lot about using apply() to optimise slower for loops, but I haven't been able to get my head around it so far. I believe that at least one of the for loops is definitely avoidable by just outputting a vector, and dealing with it at the end. However, I cannot for the life of me work out how to remove both for loops.

Does anyone have any thoughts?


texts: Converting a Pandas Series string of multiple attributes into individual attributes?

texts:
I'm working on an Airbnb dataset in order to predict the nightly price--where each example is one Airbnb listing (ie a listing for someone's house, room, apartment, etc). Each feature is a characteristic of the listing (ie # bedrooms, # bathrooms, average review score etc).

One of the features is 'amenities', which are the amenities offered for the listing (ie wifi, tv, kitchen, etc). Each example contains a string of its own amenities like so:

{TV,"Cable TV",Wifi,"Air conditioning",Kitchen,Elevator,Heating,Washer,Dryer,"Smoke detector","Fire extinguisher",Essentials,Shampoo,Hangers,"Hair dryer",Iron,"Laptop friendly workspace","Self check-in",Keypad,"Private living room","Hot water","Bed linens",Microwave,"Coffee maker",Refrigerator,Dishwasher,"Dishes and silverware","Cooking basics",Oven,Stove,"Long term stays allowed"}

Dtype is a string ('O' in pandas).

I plan to remove the {} on the ends and the " quote signs using pd.Series.str.replace() for each character--unless those are useful to keep.

But past that I'm not sure of the best way to go about handling the feature itself.

My questions is:

Should I create individual binary columns for each amenity (ie 1 in column "TV" if TV is an amenity, 0 in column "wifi" if wifi isn't offered)? I'm hesitant because that would add a few dozen columns. If I should do this, how can I convert the current format to binary columns with Pandas?


texts: Normalized metric for comparing regression models performance

texts:
I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It was also hard for me to explain whether performance loss by 225 (in terms of MAE), when switching from one model to another, is significant or not.

To me it was clear that both are little because I knew the context: we we're talking about predicting house prices ranging from \$34,900 (min) to \$755,000 (max), so that

MAE=17,000 is just 2.5% of the difference between max and min
Change in MAE by \$225 is just 0.03% of the difference between max and min.

Are there some normalized metrics for comparing performance of regression models without the need to know the context?

Which of those metrics are available in scikit-learn? For instance, it provides mean_absolute_error and mean_squared_error but they are not normalized.

Edit:

After Joe B suggestions, I've plotted a graph to see deviations between predicted and expected price. In fact, that's gives more insight than a single-number metric:


texts: Are there libraries similar to cleverhans for different types of classifiers?

texts: I am seeking adversarial examples for classifiers (e.g. random forest, logistic regression, multilayer perceptron) that I've already trained (through scikit-learn). But I can't seem to find any existing libraries for this purpose. This could be a fun project for me to start, but before beginning to explore my naïveté, is there any literature/code on this topic that I should be considering?


texts:
Consider a matrix, $\mathbf{V}$, where each row corresponds to one of $m$ electoral district and each column corresponds to one of $n$ candidates or political parties. Each element, $V_{ij}$, is the proportion of votes cast in district $i$ for party $j$.

Now imagine taking leave of your senses and deciding to carry out a low-rank matrix factorisation of $\mathbf{V}$. It turns out that regular non-negative matrix factorisation minimising generalised KL-Divergence will preserve the row and column sums in the approximation1, so where the rank of your factorisation, $k$, is less than $n$, it's straightforward to find $k$ latent variables for each electoral district that describe contributions to the share of each party's vote in that district.

My questions:

1) Are there any obvious reasons I shouldn't be doing this?

2) Is there a sensible interpretation of these $k$ variables?

3) Is there a more canonical method for factorising matrices like this? Or a more canonical method for finding latent variables for data like this?


texts: How to determine activation functions for neural network

texts: I am trying to plan a neural network for regression predictions. The final activation layer should be a linear function, but for hidden layers, do the activation functions need to also be all linear functions, or are sigmoid or ReLU functions also be in the realm of possibility? What difference would having different activation functions in hidden layers have?


texts: Which error metric is good for measuring accuracy

texts: I am estimating water depth with satellite data (predicted value) and would like to validate my result using bathymetry  lidar data collected on the field and believed to be more accurate (observed value). I have different observations at each water depth. For example, number of observations at water depth range of 0-10 m are 300, where as values at deeper depth range (10 - 20 m) are less (~50 points). I have been using RMSE (as I would like to penalize larger error) to measure my accuracy but wondering if there is a better error metric out that is not sensitive to number of observations. In other words, for water depth 10 - 20 m with 50 points, I have RMSE of ~6m, and I was thinking the value could be lower if I have more observations. Where as for shallow water depth (0-10 m), my RMSE are much lower, perhaps because I have lots of observation.


texts: Using LSTM for multi label classification

texts:
I am trying to use LSTMs to train and predict authors using reviews data and metadata

author  phone  country day     review
james   iphone chile   tuesday the book was really amazing

How do i pass all these features into the network?


texts: Performance worsen when normalizing on CNN

texts:
I am facing quite a strange behaviour. As far as I have understood, especially when dealing with CNN, feature normalisation/ standardisation, should help the model at converging faster. Now, I am working with ECG signals, which are characterized by high variability intra and inter subjects. I am dealing with them using CNNs. My model performs quite well when I do not normalise the data. Otherwise, when I do apply either standardisation or normalisation, performance downgrade (higher loss, and 10% less accuracy and F1). This is true if I apply the preprocessing globally as well as if normalize/standardise each signal per se.
Any clue about what could cause this behaviour? My guesses are:

presence of outliers that causes too large "compression" of the data with low values
as before but with numerical problems caused by the usage of float32, that could cause (?) potential loss of data and thus of information.


texts: How do I convert a series of timestamps in seconds to milliseconds in order to distribute them smoothly

texts:
I have a script that collects memory usage data from a device running a Linux stack. I collect samples roughly every 200ms and write the measurement to a csv file with a timestamp.

The problem is that the date binary on the target platform only reports the time in seconds, so I have multiple samples for each distinct timestamp. Some seconds contain more data points than others because the sample rate is affected by what the device is doing.

The goal is to spread my data points smoothly over a millisecond scale.

I could take a naive approach and just take each second in the data as a separate series and average across it based on the number of data points there. However, this would not really smooth the seriet. It might approximate it well enough.

What I'd like to know is how to do this properly. In particular, is there an algorithm or open source library (javascript perfered)?

It could be an optimization problem with the following constraints:

The total deltas of successive timestamps should be minimized
Only the millisecond parts of the timestamps should change (they shouldn't move to another second


texts: Does regularisation make the loss noisy?

texts:
I implemented dropout and got a loss plot like this

Before implementing regularisation the loss the was not noisy at all

I understand why implementing dropout would increase the noise as different neurons would be active in successive epochs, but does L2 regularisation add noise? Why or why not


texts: In "Attention Is All You Need", why are the FFNs in (2) the same as two convolutions with kernel size 1?

texts:
In addition, why do we need a FFN in each layer when we already have attention?

Here's a screenshot of the relevant section from Vaswani et al. (2017):


texts:
I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so when I list them, the most similar are listed together in groups. That's how I would avoid having numerous (almost) same documents listed one, after another.

I was thinking some kind of clustering would come handy. But the problem is, I don't know how many clusters I need. Also the number would have to be dynamic. And still, most of the documents wouldn't belong to a cluster, because they don't have any similar documents. So I would cluster just similar documents.

Can anyone point me to the direction that would help me solve this problem, or provide some examples of similar problems.


texts: How can we assess the importance of the features even if we ended up applying PCA?

texts:
There are multiple techniques to analyze the feature importance (permutations, SHAP values, etc).

It is essential that, in order to improve the interpretability of the model, we can somehow map the transformed features to the original ones. For example, when we assess the feature importance of encoded features, we can somehow see what are the encoded variables related to primordial ones that are predominant at the top score/weight entries.

But, how can we assess the feature importance when we apply feature reduction techniques like PCA that can dramatically reduce the number of features and the interpretability of the model?

Thanks.


texts: How to use Cosine Distance matrix for Clustering algorithms like mean-shift, DBSCAN, and optics?

texts:
I am trying to compare different clustering algorithms for my text data. I first calculated the tf-idf matrix and used it for the cosine distance matrix (cosine similarity). Then I used this distance matrix for K-means and Hierarchical clustering (ward and dendrogram). I want to use the distance matrix for mean-shift, DBSCAN, and optics.

Below is the part of the code showing the distance matrix.

from sklearn.feature_extraction.text import TfidfVectorizer

#define vectorizer parameters
tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=200000,
                                 min_df=0.2, stop_words='english',
                                 use_idf=True, tokenizer=tokenize_and_stem, ngram_range=(1,3))

%time tfidf_matrix = tfidf_vectorizer.fit_transform(Strategies) #fit the vectorizer to synopses

terms = tfidf_vectorizer.get_feature_names()

from sklearn.metrics.pairwise import cosine_similarity
dist = 1 - cosine_similarity(tfidf_matrix)
print(dist)

I am new to both python and clustering. I found the code for K-means and hierarchical clustering and tried to understand it but I cannot apply it for other clusterings algorithms.
It would be very helpful if I can get some simple explanation of each clustering algorithm and how this distance matrix can be used to implement (if possible) in different clustering.

Thanks in advance!


texts: ML algorithms for defining NORMAL user behavior

texts:
I am trying to create a model that would capture the usual behavior of a user: i.e., create a model for user profiling.

I do have 2 million rows of data indicating normal behavior of a user and want to define a model that can detect any variation from the normal behavior of the user.

I am contemplating of using one class svm / isolation forest trained on normal behavior. I would like to get an opinion whether that looks as a good direction.

Or should I venture into deep learning or unsupervised clustering based model and gather data for abnormal behavior of a user as well to possibly use supervised models.


texts:
While running the below pig script I am getting an error in line4:
If it is GROUP then I am getting error.
If I change from 'GROUP' TO 'group' in line4, then the script is running.

What is the difference between group and GROUP?

LINES = LOAD '/user/cloudera/datapeople.csv' USING PigStorage(',') AS ( firstname:chararray, lastname:chararray, address:chararray, city:chararray, state:chararray, zip:chararray );

WORDS = FOREACH LINES GENERATE FLATTEN(TOKENIZE(zip)) AS ZIPS;

WORDSGROUPED = GROUP WORDS BY ZIPS;

WORDBYCOUNT = FOREACH WORDSGROUPED GENERATE GROUP AS ZIPS, COUNT(WORDS);

WORDSSORT = ORDER WORDBYCOUNT BY $1 DESC;

DUMP WORDSSORT;


texts: Abnormal behavior while predicting with training data

texts:
I am trying to predict the appearance count of a particular item using Neural Network.My data has severe class imbalance i.e 90% of data contains zero as output, remaining values ranges from 1 to 5. I am training a neural network with item features and week number of year.Whenever I run the neural network and predict for the training data the sum of all predicted counts are equal to the sum of observed counts but the individual counts are not even getting close.I tried with both RMSE and MAE as error metrics.I am not sure about this is correct,
 any help would be great


texts: Why does bidirectional LSTM have half the parameter count compared to LSTM in keras?

texts:
I want to implement a unidirectional and a bidirectional LSTM in tensorflow keras wrapper with the same amount of units. As an example I implement the unidirectional LSTM with 256 units, and the bidirectional LSTM with 128 units (which as I understand gives me 128 for each direction, for a total of 256 units). The implementation details:
import tensorflow as tf
in_ = tf.keras.Input(shape=(28,28))

x = tf.keras.layers.LSTM(256)(in_)

model_unidirectional = tf.keras.Model(in_,x)
print(model_unidirectional.summary())

y = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128))(in_)

model_bidirectional = tf.keras.Model(in_,y)
print(model_bidirectional.summary())

However, looking at the models' summary, the unidirectional LSTM has double the parameter count compared to the bidirectional LSTM, even if they have the same output shape in both cases:
model_unidirectional summary:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_11 (InputLayer)        [(None, 28, 28)]          0
_________________________________________________________________
lstm_11 (LSTM)               (None, 256)               291840
=================================================================
Total params: 291,840
Trainable params: 291,840
Non-trainable params: 0
_________________________________________________________________

model_bidirectional summary:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_11 (InputLayer)        [(None, 28, 28)]          0
_________________________________________________________________
bidirectional_7 (Bidirection (None, 256)               160768
=================================================================
Total params: 160,768
Trainable params: 160,768
Non-trainable params: 0
_________________________________________________________________

Why does the bidirectional approach have significantly less parameters if their output shape is the same?


texts: Possible Challenges for a Data Science Escape Room

texts:
Dear Data Science Community,

as my project for my bachelorthesis I am working on a concept for a physical Data Science Escape Room. The goal of the escape room is to create awareness for data science / the importance of data in general and motivate "escapees" to learn more about data science.

The escape room is going to be a physical room, that needs to be escaped by completing different challenges related to data science. The target group are people who are already working with data in form of reports or similar, but have no or little to none background in data science.

Possible challenges for this escape room could use simple machine learning algorithms that need to be trained, or easy statistical concepts. Players could also need to curate data, which then needs to be integrated to be usable afterwards.

I am looking for ideas or further input for possible challenges for this escape room, and was hoping to have some data science experts give creative input!

Thank you!


texts: How do I extract the kernel matrix for a classifier created using `sklearn.svm.SVC`?

texts:
I am currently using the kernels that come with sk-learn support vector machine library.
How do I extract the kernel matrix for a classifier created using sklearn.svm.SVC?


texts:
Hi I have this dataset (It has many more columns)

   media   brand    radio        tv            cinema
   <chr>   <dbl>    <dbl>       <dbl>            <dbl>
  radio      0      0             0                0
  tv         0      0             0                0
  cinema     0      0             0                0
  tv         0      0             0                0
  radio      0      0             0                0
  tv         0      0             0                0

I want to obtain the following(Assign a 1 to each column based on the value of media column):

        media   brand    radio        tv            cinema
       <chr>   <dbl>    <dbl>       <dbl>            <dbl>
      radio      0      1             0                0
      tv         0      0             1                0
      cinema     0      0             0                1
      tv         0      0             1                0
      radio      0      1             0                0
      tv         0      0             1                0

Do you have any  idea on how to do it?As I have many more columns, I'm trying to find a solution using dynamic variable assignment.
Thanks


texts: EDA for analysis of nominal variable with high cardinality

texts: I have a nominal variable (car model) with very high cardinality (~8500 labels) and I would like to analyse its relation with a binary target variable. While I can create logical groups and compare the distribution of target variable for each of the groups, can anyone suggest if there are any superior techniques/visualization tools for this type of analysis?


texts: Transfer learning: Poor performance with last layer replaced

texts:
I am using a transfer learning approach. For this I followed the tensorflow for poets tutorial. I use a pre-trained InceptionV3 architecture trained on the Imagenet dataset. The last layer and the softmax classification is replaced and retrained, using a new set of 7 classes.

Data

Per class I have around 4.000 - 5.000 images. I tried multiple training parameters with an AdamOptimizer. The labels are noisy, about 15-20% of the labels are incorrect. The images show products of a certain category (e.g. cars) and the labels classify different type of a feature (e.g. 7 different types of tires, wheels).

Parameters

learning rate: 0.001
iterations: 7.000
batch size: 100

Performance

The test accuracy is 50%, train accuracy 68%. Visualising the learning rate, the network already reached 50% after 2000 iterations. What surprises me is the overall low performance as well as the lack of further improvement during the training time. Its also noteworthy that the network seems to make very hard to understand errors (not only mixing up similar classes but clearly distinguishable ones as well).

Now I wonder: Is this potentially because retraining only a single layer is too limited to pick up the subtle differences in certain parts of the images? How would you go about it to improve?


texts: Random Forest Stacking Experiment for Imbalanced Data-set Problem

texts:
In order to solve a Imbalanced Dataset Problem, I experimented with Random Forest in the given manner (Somewhat inspired by Deep-Learning)

Trained a Random Forest which will take in the input data and the predict probability of the label of the trained model  will be used as a input to train another Random Forest.

Pseudo Code for this :

train_X, test_X, train_y, test_y = train_test_split(X,y, test_size = 0.2)
rf_model = RandomForestClassifier()
rf_model.fit(train_X, train_y)
pred = rf_model.predict(test_X)
print('******************RANDOM FOREST CM*******************************')
print(confusion_matrix(test_y, pred))
print('******************************************************************')
predict_prob = rf_model.predict_proba(X)

X['first_level_0'] = predict_prob[:, :1].reshape(1,-1)[0]
X['first_level_1'] = predict_prob[:, 1:].reshape(1,-1)[0]

train_X, test_X, train_y, test_y = train_test_split(X,y, test_size = 0.2)
rf_model = RandomForestClassifier()
rf_model.fit(train_X, train_y)
pred = rf_model.predict(test_X)

print('******************RANDOM FOREST 2 CM*******************************')
print(confusion_matrix(test_y, pred))
print('******************************************************************')

And I was able to see considerable improvement in the recall. Is this approach mathematically sound. I used the second layer of the Random Forest such that it would be able to correct the error by the first layer. Just looking to combine the principle of boosting to Random Forest Bagging Technique.Looking for thoughts.


texts: Deep RL: Proximal policy optimization gradient calculation

texts:
Case:

Continuous action domain with 4 outputs (control problem)
Policy and Value function approximation with fully connected neural networks

I understand that the loss function for PPO for the overall policy network is the clipped-loss function plus some stuff [Schulman, 2017].

  $L = E[L_{clip}(\theta) + c_1*L_{VF}(\theta) + c_2*S[\pi_{\theta}(s_t)]]$  <1>

  $g_{common} = E[\nabla_{\theta} log(\pi_{theta}(s|a)A]$

My questions are thus:

What is the gradient estimation of loss function <1> with respect to each predicted output? A common gradient estimation is shown above, but it is for a different objective function. The language in the paper is that <1> is a surrogate objective function.
How is loss/gradient calculated for non-output-level neural network weights given the clipped-loss function? Can the inner neurons use something like neg_log_policy with respect to target neuron output or mean-square-error or something else? I'm not sure because loss function <1> and similar use Advantage function and reward.


texts: Intuition behind One Class SVM (Scholkopf)

texts:
I am trying to understand the intuition behind the idea of finding a hyperplane that separates the training data from the origin in the feature space.
Why separation from origin with a hyperplane helps with outlier detection?
Is the origin the 0-vector in the feature space?
Any hint/assistance is appreciated.


texts: optimizing a linear optimization function with linear constarints and binary variables

texts:
I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization.
I have an ojective function that needs to be maximized
def objective(bat1,bat2,bat3,bat4,bat5,bat6,bat7,wk1,wk2,ar1,ar2,ar3,ar4,ar5,bowl1,bowl2,bowl3,bowl4,bowl5,bowl6):
    total_score_batsman =  bat1*60 + bat2*40 + bat3*36 + bat4*35 + bat5*25 + bat6*22 +bat7*9
    total_score_wks = wk1*24 + wk2* 14
    total_score_ar = ar1*45 + ar2*24 + ar3*15 + ar4*1
    total_score_bowler = bowl1*64 + bowl2*47 + bowl3*16 + bowl4*7 + bowl5*5 + bowl6*4
    return total_score_batsman + total_score_wks + total_score_ar + total_score_bowler #needs to be maximized

constraints
#budget constraint

def budget(bat1,bat2,bat3,bat4,bat5,bat6,bat7,wk1,wk2,ar1,ar2,ar3,ar4,ar5,bowl1,bowl2,bowl3,bowl4,bowl5,bowl6):
    batsman_budget = bat1*10.5 + bat2*8.5 + bat3*10.5 + bat4*8.5 + bat5*9.5 + bat6*9 +bat7*9
    wk_budget = wk1*8.5 + wk2*8
    ar_budget = ar1*8.5 + ar2*9 + ar3*8.5 + ar4*8
    bowler_budget = bowl1*9 + bowl2*8.5 + bowl3*8.5 + bowl4*8.5 + bowl5*9 + bowl6*9
    total_budget = batsman_budget + wk_budget + ar_budget + bowler_budget
    return total_budget

total_budget <= 100 #constraint

#player_role constraints

bat1 + bat2 + bat3 + bat4 + bat5 + bat6 + bat7 >= 3
bat1 + bat2 + bat3 + bat4 + bat5 + bat6 + bat7 <= 5

wk1 + wk2 = 1

ar1 + ar2 + ar3 + ar4 + ar5 >= 1
ar1 + ar2 + ar3 + ar4 + ar5 <= 3

bowl1 + bowl2 + bowl3 + bowl4 + bowl5 + bowl6 >= 3
bowl1 + bowl2 + bowl3 + bowl4 + bowl5 + bowl6 <= 5

# no of players in a team constraint
bat1 + bat2 + bat3 + bat4 + bat5 + bat6 + bat7 + wk1 + wk2 + ar1 + ar2 + ar3 + ar4 + ar5 + bowl1 + bowl2 + bowl3 + bowl4 + bowl5 + bowl6 = 11

where bat1,bat2,bat3.....bowl5,bowl6 are 0 or 1

Its completely a linear problem, and there is no requirement of non linear optimization techniques.Can some one help me with how to solve such problems or are there any libraries in python which will help me solving this?
Thanks


texts: Error: Negative dimension size caused by subtracting 3 from 1

texts:
The dataset here is the famous MNIST.
Here's the error: Negative dimension size caused by subtracting 3 from 1 for 'conv2d_8/convolution' (op: 'Conv2D') with input shapes: [?,1,1,64], [3,3,64,128]
Here's the code:

model = models.Sequential()
model.add(layers.Conv2D(16, (3, 3), activation='relu', input_shape=(28,28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(32, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))

I would appreciate to know how can I fix the error, and generally what these type of errors mean.
Thanks everyone.


texts: Tensorflow data API: Building continuous streams of data from a Dataset of Datasets

texts:
I'm trying to build a language model with LSTMs (like ELMO).

I've got a lot of documents and want to split them into words as input, but keep their order. So it should get all words of the first document in order, then get the next document, and so on. So far so good, got a working pipeline for that, until batching comes in.

As Tensorflow RNNs consider every entry in a batch a unique sequence, I need to start batching at document level and split into words afterwards. At this point I'm working with batches of tf.data.Dataset, which isn't all that beautiful, but manageable.

Since these Datasets are highly variable in length, this gets complicated, especially since I don't want to pad, but just want the next document to start.

Basically, what I want is to create batch_num streams like this:

Batch index 1: [ words in a document ] [ words in next document ] [ words ]
Batch index 2: [ words in a way longer document ] [words in short doc ]
Batch index 3: [ words in some document ] [ words in another document ]
...

It's important that there's no padding happening between the documents. Losing some sentences/words at the end is completely fine.

It feels like this shouldn't be such a far-fetched problem, but I can't think of a solution using the tf.data API. Obviously I can do this with normal python code, but I'd love to utilize Dataset features like prefetching.

// EDIT:

Digging deeper, the one function I never bothered to look at, interleave, seems promising.

num_batch = 4
dataset = Dataset.range(1, 6)  # ==> [ 1, 2, 3, 4, 5 ]
dataset = dataset \
    .interleave(lambda x: Dataset.from_tensors(x).repeat(3), cycle_length=num_batch) \
    .batch(num_batch) # ==> 0 :  [1 2 3 4]
                      #     1 :  [1 2 3 4]
                      #     2 :  [1 2 3 4]
                      #     3 :  [5 5 5]

However, as soon as the Datasets are of variable length, they start switching their places as previous ones run out.

given: [ [1], [2,2], [3,3,3], [4,4,4,4], [5,5,5,5,5] ]
=> 0 :  [1 2 3 4]
   1 :  [2 3 4 5]
   2 :  [3 4 5 4]
   3 :  [5 5 5]


texts: Is the hypothesis space spanned by kernel evaluations on datapoints equivalent to the hypothesis space of linear functionals in the feature space?

texts:
when studying kernel methods a few years ago I got a bit confused with the concepts of feature space, hypothesis space and reproducing kernel Hilbert space. Recently, I thought a little about questions that I asked myself back then (with newly acquired math background) and noticed that some things are still unclear to me. I appreciate help and pointers to good - mathematical - literature.

Let's consider the following learning problem: We are given a training sample $((x_1, y_1), \dots, (x_n, y_n)) \in (\mathcal{X} \times \mathbb{R})^n$ and want to learn a functional relationship between $\mathcal{X}$ and $\mathbb{R}$ from this data. Two popular hypothesis spaces are:
$$\mathcal{H}_{\phi} = \{\langle w, \phi(\cdot)\rangle: w \in \mathcal{F}\},$$ in which $\phi: \mathcal{X} \to \mathcal{F}$ denotes a feature space mapping from the input space to a Hilbert space, and $$\mathcal{H}_{k} = \{\sum_{i=1}^m \alpha_i k(x_i, \cdot): m \in \mathbb{N, x_i \in \mathcal{X}, \alpha_i \in \mathbb{R},\ for\ i \in \{1, \dots, n\}}\},$$
in which $k$ denotes the positive definite kernel $k(x, x') := \langle\phi(x),\phi(x')\rangle$. The question is whether those two hypothesis spaces are equivalent and it bugs me. The Moore-Aronszajn Theorem states that there is a unique correspondence between reproducing kernel Hilbert space and positive definite kernel. When I studied kernel theory I proofed, that $\mathcal{H}_{\phi}$ is a reproducing kernel Hilbert space. Still, I did not manage to proof that the two mentioned hypothesis spaces are the same without requiring that $\phi$ is surjective (which is rarely required in ML-literature).

Thank you in advance!


texts: Faster R-CNN: Labels regarding the positive anchors when there are many classes

texts:
The authors of the original paper of Faster R-CNN when they refer to the positive anchors, they are labeled as 1. I guess they refer in binary classification. What happens in the case in a task we have more than two classes?

I mean the purpose of the RPN is to generate just region proposals (there is an object or not, regardless the label) or it generates region proposals and the respective label as well? I guess it is the second case, since the positive anchors that participate in the training process need to have a ground truth label as well, i.e. (cat, dog, e.t.c).

However, I might misunderstand things and interpret things in a wrong way.


texts:
In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff.

What would be the disadvantage of using these packages to read a small file that one would typically read with read.csv?

If their is no disadvantage of using them to read small files, does this mean functions such as read.csv might not have much use in future given the emerging big-data-targeted readers?


texts: Is there a way to classify an alphanumeric string?

texts:
I have data containing various items. Each item has a unique alphanumeric code associated with it (see the example below).
Is there a way to predict the item type based on the alphanumeric code?
Data:
Item          code          type
1       4S2BDANC5L3247151   book
2       1N4AL3AP1JC236284   book
3       3R4BTTGC3L3237430   book
4       KNMAT2MT1KP546287   book
5       97806773062273208   pen
6       07356196706378892   Pen
7       97807345361169253   pen
8       01008130715194136   chair
9       01076305063010CCE44 chair
etc


texts: Where to store dataset when creating neural network models and using github?

texts:
I have been recommended to use GitHub for all my coding projects as it is a great way to demonstrate to employers what you have been working on as well as just sharing anything you have created that might be useful to others. So I am storing my code for Neural Network Models I am working in a GitHub repository but I am not sure when I download a data set to run a test with where I should download that too? Inside or outside the git repository?

Sorry if this is a dumb question but new to this. I can imagine arguments for either answer. Mainly size being too big or the results being easily reproducible.


texts: How to build Generative Model when we have more than one variable

texts:
I have a data-frame which has looks similar to this:
A   B   C
1   2   2
2   4   3
4   8   5
9   16  7
16  32  11
22  43  14
28  55  17
34  67  20
40  79  23

A,B and C can be assumed to be the features in machine learning literature.
I have read maximum likelihood estimation for 1 variable assuming Gaussian distribution.
The equation is something like, where xi's are each data-point:

Where x1,x2....xn are n data points each having dimension 3. If we assume p(x) to be gaussian, then we can use the Gaussian Normal distribution equation as:

This is well understandable if we have only 1 feature.
How can I generalise the above normal distribution equation when we have more than 1 features, here we have 3 features? Can someone help me to write the maximum likelihood for the above data-frame?
Do we learn mu and sigma for each features A,B and C that is total of 6 learnable parameters?
If we have 3 different distribution say Normal, exponential and so on for columns A,B and C then how does the MLE equation looks like over entire-data-frame?
If we do argmax of equation 1, we don't require the ground truth for it. right? We are just maximising the equation?


texts: How does "sameModel" mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

texts:
I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline.  From the example, the model is fit using a pipeline (val model), then the pipeline is saved to a directory.  The next line is

  val sameModel =
  PipelineModel.load("/tmp/spark-logistic-regression-model")

I don't see how/where sameModel is being used.  I see model being used again on test data.  I would've expected the example to use sameModel to show how to invoke the loaded pipeline back into the process. Does sameModel automatically update model?

Any insight would be appreciated as I am interested in saving a pipeline following a fit then load it at a later point, but am having issues with:

Finding good examples saving & loading Spark ML pipelines, then invoking said loaded pipeline
Not fully understanding the examples I see online

Any assistance in clarifying what is going on and how to move forward would be greatly appreciated.  Thanks!


texts: Pandas - Get feature values which appear in two distinct dataframes

texts:
I have a Pandas DataFrame structured like this:

    user_id movie_id    rating
0   1       1193        5
1   2       1193        5
2   12      1193        4
3   15      1193        4
4   17      1193        5
5   18      1193        4
6   19      1193        5
7   24      1193        5
8   28      1193        3

Each row corresponds to a rating event performed by the user user_id for the movie movie_id. For instance, the first row says that user 1 rated the movie 1193 with a rating of 5.

This data comes from the MovieLens project.

My goal is to find all the users who satisfy these two conditions:

rated movie 588 with a rating of 5
rated movie 3578 with a rating of 3

I came up with two filtered DataFrame objects for each of the above conditions:

ratings_588_5 = data[(data.movie_id == 588) & (data.rating == 5]
ratings_3578_3 = data[(data.movie_id == 3578) & (data.rating == 3)]

Which result in, respectively:

>>> ratings_588_5
user_id movie_id    rating
438     588         5
758     588         5
913     588         5
1024    588         5
1214    588         5

>>> ratings_3578_3
user_id movie_id    rating
45      3578        3
321     3578        3
467     3578        3
758     3578        3
1024    3578        3
1381    3578        3

In Pandas, how can I compute a list of all user_id which appear in both DataFrames?

In this example, the result I want to obtain is:

[758, 1024]


texts: I'm getting same MAE value in decision tree model for maxdepth value from 1 to 10

texts:
R code :

library(tidyverse) # utility functions
library(rpart) # for regression trees
library(randomForest) # for random forests
library(modelr)

split_data = resample_partition(melb_data,c(test=.3,train=.7))

get_mae <- function(maxdepth, target, predictors, training_data, testing_data){

predictors <- paste(predictors, collapse="+")
formula <- as.formula(paste(target,"~",predictors,sep = ""))

model <- rpart(formula, data = training_data,
             control = rpart.control(maxdepth = maxdepth))

mae <- mae(model, testing_data)
return(mae)
}

target <- "Price"
predictors <-  c("Rooms","Bathroom","Landsize","BuildingArea",
             "YearBuilt","Lattitude","Longtitude")

for(i in 1:10){
mae <- get_mae(maxdepth = 3, target = target, predictors = predictors, training_data = split_data$train, testing_data = split_data$test)
print(glue::glue("Maxdepth: ",i,"\t MAE: ",mae))
}

OUTPUT :

Maxdepth: 1  MAE: 356628.697268696
Maxdepth: 2  MAE: 356628.697268696
Maxdepth: 3  MAE: 356628.697268696
Maxdepth: 4  MAE: 356628.697268696
Maxdepth: 5  MAE: 356628.697268696
Maxdepth: 6  MAE: 356628.697268696
Maxdepth: 7  MAE: 356628.697268696
Maxdepth: 8  MAE: 356628.697268696
Maxdepth: 9  MAE: 356628.697268696
Maxdepth: 10 MAE: 356628.697268696


texts:
I have two TensorFlow tensors each being outputed from object detector network, each containing this shape (batch, height, width, detected_boxes * num_classes). I want to compare each box from each tensor with the other using a siamese sub network.
I want the network to be end-to-end, so what is the correct approach, should I use TensorFlow while loop, and do some sort of nested loop?


texts: How to do Multivariate Adaptive Regression Splines feature selection in python? Specifically, I need the python equivalent of the earth function in R

texts:
This is the code in R:

marsModel <- earth(eval(parse(text=paste(ResponseVariable,"~."))), data = data) #build model
ev <- evimp (marsModel)

Response variable is self-explanatory, the data contains 178 features and 1 response variable. Each feature has 56 observations.

'ev' ends up with 10 features selected out of 178 and their gcv (Generalized cross validation), rss (Residual sum of squares) and nsubsets (no. of subset models the variable occurs) scores.


texts: When do I have to use aucPR instead of auROC? (and vice versa)

texts:
I'm wondering if sometimes, to validate a model, it's not better to use aucPR instead of aucROC? Do these cases only depend on the "domain  & business understanding" ?

Especially, I'm thinking about the "unbalanced class problem" where, it seems more logical to use the aucPR because recall and precision are well-used metrics for this problem.


texts: How can I do simple machine learning without hard-coding behavior?

texts:
I've always been interested in machine learning, but I can't figure out one thing about starting out with a simple "Hello World" example - how can I avoid hard-coding behavior?

For example, if I wanted to "teach" a bot how to avoid randomly placed obstacles, I couldn't just use relative motion, because the obstacles move around, but I don't want to hard code, say, distance, because that ruins the whole point of machine learning.

Obviously, randomly generating code would be impractical, so how could I do this?


texts: Ordering quotes in a list based on user input and text analysis

texts:
A little bit of context:

I have a website that has many quotes. These quotes are organized automatically by Solr into lists of quotes, so e.g. there is a list called 'Smart Quotes' that includes some quotes from Einstein, Confucius, Platon, Nietzsche, etc. Solr returns a list of quotes based on a query that uses tags to identify a quote as "smart" or some other category.

A list returned by Solr can have from 0 to 1000 results. These results are split into pages to be presented to the user (my-website.com/smart-quotes/1/, my-website.com/smart-quotes/2/, my-website.com/smart-quotes/3/ ...)

Until here it's fine, the quotes are very relevant to the list. The problem is how quotes are ordered.

If I use something like Hotjar to track where users click or move their mouse, I can see that many times the first quote that appears on the list is not the most interacted with. Many times it's the 3rd, 5th or even the last one of a page. By more relevant, I mean the one that got the most clicks on and mouse moves.

Also if I go to the next pages of that list, I can manually identify quotes that are probably more relevant than the ones that appear on the first page.

Assuming that:

the quotes in a list are relevant to the list itself
I have a way of extracting the user interaction data in any format I want
The current ordering is based solely on the Solr text analysis without having in mind any user interaction at all.
There might be other very relevant quotes that have no interaction whatsoever because they are on page 200 and no user gets there to read it.

What algorithm/machine learning method can be applied to this problem in order to get a list of quotes ordered "in a better way"? By that I mean it has in mind the user interactions with the quotes and can identify quotes that are relevant but are not seen.


texts: What if Training and testing dataset comes from the same source?

texts:
I am working on a classification problem in which I have to distinguish between healthy and damaged plates. when I use the combination of k-means clustering and SVM algorithm together with 10-fold cross validation, I can achieve the accuracy up to 95%. All the training and validation datasets come from the experiment.
For the testing, can I get the datasets after repeating same experiments with same specimen or I have to use different sets of specimens?


texts:
I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal.
Does anyone have experience with this kind of information gathering and can recommend a tool or API for it? (eg. in Apache Airflow, Luigi, RapidMiner or KNIME)
May I just missed the right term that describes this kind of meta-data?


texts: NLP CounterVectorizer (sklearn), not able to get it to fit my code

texts:
I was starting an NLP project and simply get a "CountVectorizer()" output anytime I try to run CountVectorizer.fit on the list.  I've had the same issue across multiple IDE's, and different code.  I've looked online, and even copy and pasted other codes with their lists and I receive the same CountVectorizer() output.
My code is as follows:
cv = CountVectorizer()

messages = ['This is a good product', 'It was a bit pricey for what it does', 'I found good value here']

cv.fit(messages)

**output ----->  CountVectorizer()**

I'm really stumped on this issue.  Any advice would be greatly appreciated.  Thanks.
Update:  This seems to be a local issue as I am able to get it to fit on Colab.  If anyone can suggest what might be going on I'd be estatic.