texts: What are the strategies for feature engineering in a hierarchical/relational structured data?

texts:
I have a feature set with each feature having its own feature-set, and they are laid out in a SQL database with foreign key relationships. The target is defined on the top level feature-set and is a binary Win/Loss variable.

  Target;
  feature -> sub-features->sub-features;
  feature -> sub-features;

All the classification algorithms expect a flat feature file as an input. What are the methods for flattening out such a structured dataset, while preserving the cross feature dependence on the sub-feature set for other features?

How would you decide which sub-features to select? Currently I'm aggregating total Wins/Attempts and regressing (PCR) the individual sub features to see which ones have an influence over the final target, what would be a more generalized way of selecting these? And then combining these into a flattened feature?

I would also appreciate if someone can identify the correct machine learning jargon for this exercise.


texts: svm KKT condition violation that alpha choice in SMO

texts:
when I was read the paper about SMO　for SVM.
John C. Platt paper

choosing the alpha which violated KKT condition is very confusing me. I do not very understand the code as below.

if ((r2 < -tol && alph2 < C) || (r2 > tol && alph2 > 0))

why does it choose r2 < -tol when alph2 < C, not r2 < tol?


texts: Optimizing the distribution on an unknown of input / Data Audit Pattern

texts:
I want to develop a new autoencoder. Once the network itself is developed, the customer wants to train and deploy the it all by himself. Due to data protection he can't give me anymore information, as advising me about the usage in transactional ERP data and the number of inputs/outputs, which will be 10.

Usually I would start a project by analysing the data and checking the distribution by running histogram. The initial check on the data usually gives me insight about Skewness, max-values, min-values, averages and medians.

Particularly the skewness worries me, as in my experiences neural-networks such as auto-encoders perform extremely bad on such inputs. Usually I would offset a strongly skewed data distribution by using logarithms, root functions or power transformation.

I wonder if I can just us any of these "Anti-Skewness" methods just blindly, in order to offset "bad-data" but not worrying about destroying the performance of my network.


texts: Converting non-numeric data values into equivalent rank scores

texts:
Consider a data-frame similar to  the one shown (the actual data-frame is much larger)

ID EDUCATION   OCCUPATION      BINARY_VAR
1  Undergrad   Student              1
2  Grad        Business Owner       1
3  Undergrad   Unemployed           0
4  PhD         Other                1

The final objective is to apply PRIDIT scoring on individual profiles (ID) based on discrete "rank" scores of the individual values in the cell. These ranks can be thought of as indicator variables which will be used to collectively rate any $ID_i$

So for example, the ranks could signify the possibility of some $ID_i$ committing fraud:

1 : Low
2 : Medium
3 : High

The variable, BINARY_VAR is something like a "training variable" or rather, a "predictor variable", such that

$Var = 0:$ Fraud

$Var = 1:$ Non-fraud

By this reasoning, an unemployed Undergraduate would be a Rank 3 profile.

In order to apply PRIDIT, I must first convert the non-numeric variables into scores or levels.

The way it is currently being done is by applying correspondence analysis on each column against BINARY_VAR and then calculating the distance of the column contribution scores from row contribution score for non-fraud.

Row and column scores look something like this (respectively):

            CONTR
0           1.654
1           98.346
------------------------------
                  CONTR
Undergraduate     2.803602e-04
Graduate          3.147824e+00
PhD               9.176451e+00
Other             1.179664e+01

The obtained distance (supposedly) gives the required score for the level, which is written back to the data-frame as a rank (higher value resulting in a higher rank).

My main concerns about this technique are:

The data-frame is really large, and resources are limited - it is a computationally expensive method.
It involves a lot of steps, and the result of the scoring can not really be verified (can it?).

My questions are:

Does this technique seem viable?
What are betters way to assign "ranks" to non-numeric variables?


texts: Image Segmentation Class weight using tensorflow keras

texts:
I remember definitely being able to pass a list to class_weight with keras (binary image segmentation specifically). For example:
class_weight = [1, 10] (1:10 class weighting)
But now it's saying it has to take a dictionary instead of a list.
weights = {0: 1, 1: 10}

Well, when I do that I get the error:
`class_weight` not supported for 3+ dimensional targets.

Has something changed? I didn't think I'd have so many issues here.


texts: Is it possible for a model with a large amount of data to perform very well and reach an extremely low cost within a single epoch?

texts:
I am working on a project to detect human awareness levels using this dataset.
I preprocessed the video data as the following:

Convert video to frames(taking a frame every 5 seconds.
Rotate the frames to be vertical.
Apply OpenCV DNN to extract the faces from the images.
Split the data into 90% train, 5% validation and 5% test.

All in the dataset has a size of about 570,000 images.
I am using the model on a mobile device so I used transfer learning with MobileNetV2.
The model classification is extremely good but it feels odd seeing it do so well and reach a very low loss so fast.
Is this even possible on a dataset this big? I am feeling that I did something wrong cause when I try to use the model on the mobile device with Tensorflow.js it does not perform well at all. After doing some research I realized that I should be using a model that combines a CNN and a LSTM as this is video data. But I am bit strapped for time to redo the whole preprocessing of the data to convert the images into a sequence of frames and then do the training once more.
What I was planning to do was make an average of the predictions on the mobile device to improve the accuracy there but I am wondering if I messed up the process anywhere.


texts: Why Scikit and statsmodel provide different Coefficient of determination?

texts:
First of all, I know there is a similar question, however, I didn't find it so much helpful.

My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can be quite different if I use statsmodels and Scikit-learn.

First of all my snippet:

import altair as alt
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm

np.random.seed(0)
data = pd.DataFrame({
'Date': pd.date_range('1990-01-01', freq='D', periods=50),
'NDVI': np.random.uniform(low=-1, high=1, size=(50)),
'RVI': np.random.uniform(low=0, high=1.4, size=(50))
 })

Output:

          Date        NDVI        RVI
 0    1990-01-01    0.097627    0.798275
 1    1990-01-02    0.430379    0.614042
 2    1990-01-03    0.205527    1.383723
 3    1990-01-04    0.089766    0.142863
 4    1990-01-05    -0.152690   0.292427
 5    1990-01-06    0.291788    0.225833
 6    1990-01-07    -0.124826   0.914352

My independent and dependent variable:

X = data[['NDVI']].values
X2 = data[['NDVI']].columns
Y = data['RVI'].values

Scikit:

regressor = LinearRegression()
model = regressor.fit(X, Y)
coeff_df = pd.DataFrame(model.coef_, X2, columns=['Coefficient'])
print(coeff_df)
Output:
    Coefficient
NDVI    0.743

print("R2:", model.score(X,Y))

R2: 0.23438947208295813

Statsmodels:

model = sm.OLS(X, Y).fit() ## sm.OLS(output, input)
predictions = model.predict(Y)
# Print out the statistics
model.summary()

Dep. Variable:  y   R-squared (uncentered): 0.956
Model:  OLS Adj. R-squared (uncentered):    0.956
Method: Least Squares   F-statistic:    6334.
Date:   Mon, 18 May 2020    Prob (F-statistic): 1.56e-199
Time:   11:47:01    Log-Likelihood: 43.879
No. Observations:   292 AIC:    -85.76
Df Residuals:   291 BIC:    -82.08
Df Model:   1
Covariance Type:    nonrobust
    coef    std err t   P>|t|   [0.025  0.975]
x1  1.2466  0.016   79.586  0.000   1.216   1.277
Omnibus:    14.551  Durbin-Watson:  1.160
Prob(Omnibus):  0.001   Jarque-Bera (JB):   16.558
Skew:   0.459   Prob(JB):   0.000254
Kurtosis:   3.720   Cond. No.   1.00

And scatterplot of data:

How should I proceed with this analysis?


texts: Binary predication from binary variables

texts:
Logistic Regression generates a binary outcome for a non-binary variable. I need a binary outcome from a binary variable. This is the requirement.

  How to predict binary A using previous values of A?

or

  How to predict binary A,B,C,D values from previous A,B,C,D values?

I know I can gather data and calculate basic probability. I that the most suitable approach? Thank you in advance.


texts: If i use ImageDataGenerator class train_datagen = ImageDataGenerator() this way without rescaling, then my model converge faster and gives better accuracy than this way train_datagen = ImageDataGenerator(rescale=1./255). Is it normal or something wrong? But without rescaling, when i show images, then it shows WHITE images without visible content.


texts: What is the highest accuracy for classifying cats and dogs from CIFAR-10?

texts:
Resnet, DenseNet, and other deep learning algorithms achieve average accuracies of 95% or higher on CIFAR-10 images. However, when it comes to similar images such as cats and dogs they don't do as well. I am curious to know which network has the highest cat vs dog accuracy and what it is.

I am aware of Cat vs Dog competition on Kaggle and such, but none that I know of are on CIFAR-10.


texts: Minimum number of features for Naïve Bayes model

texts:
I keep on reading that Naive Bayes needs fewer features than many other ML algorithms. But what's the minimum number of features you actually need to get good results (90% accuracy) with a Naive Bayes model? I know there is no objective answer to this -- it depends on your exact features and what in particular you are trying to learn -- but I'm looking for a numerical ballpark answer to this.
I'm asking because I have a dataset with around 280 features and want to understand if this is way too few features to use with Naive Bayes. (I tried running Naive Bayes on my dataset and although I got 86% accuracy, I cannot trust this number as my data is imbalanced and I believe this may be responsible for the high accuracy. I am currently trying to fix this problem.)
In case it's relevant: the exact problem I'm working on is generating time tags for Wikipedia articles. Many times the infobox of a Wikipedia article contains a date. However, many times this date appears in the text of the article but is missing from the infobox. I want to use Naive Bayes to identify which date from all the dates we find in the article's text we should place in the infobox. Every time I find a sentence with a date in it I turn it into a feature vector -- listing what number paragraph I found this in, how many times this particular date appears in the article, etc. I've limited myself to a small subset of Wikipedia articles -- just apple articles -- and as a result, I only have 280 or so features. Any idea if this is enough data?
Thanks!


texts: How to predict Estimated Time for Arrival given only trajectory data and time?

texts:
I have data of latitude, longitude and timestamp. I am trying to build a graph based on pincodes (in India) (equivalent to zipcode). Based on this graph and trajectory data that I have, I want to predict ETA given source, destination and time for dispatch.

I am very new to this field, So I dont know how to handle and make structures using this data.

I have already gone through this question
Learning time of arrival (ETA) from historical location data of vehicle but it is not very clear to me. Any kind of help/suggestions is appreciated.

Thanks in advance.

Edit : Pincodes are equivalent to zipcodes.

Graph : Nodes of the graph will be pincode boundaries and if these pincodes share boundary then there will be edge between them.

Trajectory : It is the data collected from GPS when vehicle is travelling on the road. Data is in format (Latitude, Longitude, TimeStamp, vehicleId)


texts: What exactly is .csv in machine learning?

texts: I already have dataset of dogs and cats , so do i need to make .csv file or can i directly use the dataset for classification


texts:
How to evaluate the "betterness" of competitive good models?

Lets say I could get good models (> 90% prediction rate) with e.g.:

LinearSVC
F-test based sklearn.feature_selection.f_regression
mutual info -based sklearn.feature_selection.mutual_info_regression

But since these treat e.g. the independence/dependence of features differently, particularly e.g. LinearSVC assumes "relatively independent" features, where as mutual info particularly measure dependence between variables, then

How can I compare these models to each other?

Tests? Knowledge of the data a priori? Something else?


texts:
After performing association rule mining, (using mlxtend library), I have gotten
a pandas dataframe containing all the association rules. This contains too many rules,
therefore we must reduce the number.
For an association rule $R: X \implies Y$
For any $Z \subset X$
If any rule $R': X - Z \implies Y$ exists, such that $$confidence(R) \pm \alpha = confidence(R')$$
Then remove $R$
I tried to implement that logic directly into python as such
def filter_useless(rules):
    shadow = rules
    def remove_one(tup):
        x , y, conf= tup['antecedents'] , tup['consequents'], tup['confidence']
        for s in powerset(x):
            res = rules[  (rules['antecedents'] == s) \
                        & (rules['consequents'] == y) \
                        & (abs(rules['confidence'] - conf) <= 0.1)]
            if not res.empty:
                return False
        return True
    return shadow[shadow.apply(remove_one, axis=1)]

However, this didn't filter any rules. What am I doing wrong?


texts:
I am attempting to implement the T-DBSCAN algorithm described in T-DBSCAN: A Spatiotemporal Density Clustering for GPS Trajectory Segmentation.  I have been able to implement most of the logic between the definitions (Page 3) and the pseudo-code (page 4), but I have not been able to implement the logic to determine if a cluster is a stop as described in Definition 9 of the paper (Page 3).  Specifically, I am having trouble understanding Definition 7 "Temporally Continuous", which is referenced in Definition 9:

  Definition 7.  Temporally continuous (TC). Let the min
  and max time stamps of a cluster $C$ ($\subseteq D$) $ \text{mint} $ and
  $ \text{maxt} $, respectively. $ C  $ is known as being “temporally
  continuous” if, for $ \forall{p_t \in D} $ and $ mint < t < maxt $,
   $ p_t \in C $.

where $ D $ is the trajectory, and $ p_t $ is a point at time $ t$.

As I read Definition 7, it sounds like it means that a cluster is temporally continuous if the points in the cluster are in the trajectory, and if all the timestamps of the point in the cluster is between the minimum time and maximum timestamp in the cluster, which doesn't make sense, because all points in a cluster will always be between the minimum and maximum timestamps in the same cluster, making all clusters temporally continuous.

Would someone be willing to point me in the right direction of how to interpret this section?  Definition 9 logic is also not included in the pseudo-code, so I only have the definition itself to go off of.

I've read through an existing implementation on Github which does not have this logic implemented either, which leads me to believe either this section is trivial and does not need to be implemented, or that I am not the only one who has had issues implementing this last section of the algorithm.

Thanks for any help that you can provide.  I'm willing to provide anymore information that I have that will be helpful.


texts: How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

texts:
I am using XGBoost for a machine learning model that learns from tabular data.

XGBoost uses boosting method on decision trees. When I look at the decision-making logic of decision trees, I notice the logic is based on 1 feature at one time.  In real life, certain multiple features are related to each other.

Currently, when I feed data to the model, I simply feed all the features to it without telling the model how certain features are related to each other.

Let me describe a hypothetical example to be clearer. Suppose I have 2 features - gender and length of hair. In this hypothetical problem, I know from my domain knowledge that if gender is female, length of hair matters in determining the outcome. If gender is male, length of hair is irrelevant. How do I tell the machine learning model this valuable piece of information so that the model can learn better?

I am using XGBoost on python 3.7


texts: What proxies could be used to assess economic value of Stackoverflow for its users?

texts:
I'm intrigued by the open data provided by StackExchange, and have been running some really interesting queries on the data.stackexchange.com page (using the Stackoverflow dataset).

In particular, I would like to dive deep on this claim taken from the Stackoverflow Developer Survey 2015 summary page:

In other words: what is the "value" of Stackoverflow? Not in terms of its own economic success or funding amount, rather how it affects its users, for example accelerating a pay increase for active users.

My ideal solution would be to correlate the activity of its users to some economic independent variable such as salary. Although the data on activity exists on Stackoverflow servers, this Salary data is not available (the downloadable StackOverflow Developer Survey dataset is anonymized).

What other proxies/datasets could be used to conduct such a study? I would use github commits but it is more oriented towards productivity, and there already is a great paper on this topic.


texts: Why this calculation of weight vector in linear regression is only for small dataset?

texts:
Slides from my university says, that the following way of calculating the weight vector is suitable only for small datasets. Can you please explain, why it may be suitable for small datasets?

Here, X is observed variable and Y is dependent variable{usually 0 or 1}


texts: Training binary classifier on only one data point ( Theoritical question)

texts:
Say, I'm training a binary classifier to classify Dog vs Cat. Now, say I train my model only on one imagee ( cat). Now I mirror this cat image that I used to train my model. Now on the mirror image I want to make predciton.
Question : What will my model predict ?
does it predict cat, or will it randomly predict either cat or dog
I think it will predict cat beacuse the parameters have been trained only for cat, so irrespective of any input data, the parameters will support prediction for cat. Am I correct ? is my reasoning correct?


texts: Pytorch : Loss function for binary classification

texts:
Fairly newbie to Pytorch & neural nets world.Below is a code snippet from a binary classification being done using a simple 3 layer network :

n_input_dim = X_train.shape[1]
n_hidden = 100  # Number of hidden nodes
n_output = 1   # Number of output nodes = for binary classifier
# Build the network
model = nn.Sequential(
    nn.Linear(n_input_dim, n_hidden),
    nn.ELU(),
    nn.Linear(n_hidden, n_output),
    nn.Sigmoid())

x_tensor =  torch.from_numpy(X_train.values).float()
tensor([[ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ...,   0.1538,   5.0000,   0.1538],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000,   6.0000,   0.2381],
       ...,
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000]])
y_tensor =  torch.from_numpy(Y_train).float()
tensor([0., 0., 1.,  ..., 0., 0., 0.])
#Loss Computation
loss_func = nn.BCELoss()
#Optimizer
learning_rate = 0.0001
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

train_loss = []
iters = 500
for i in range(iters):
    y_pred = model(x_tensor)
    loss = loss_func(y_pred, y_tensor)
    print " Loss in iteration :"
    print (i, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    train_loss.append(loss.item())

In the above case , what i'm not sure about is loss is being computed on y_pred which is a set of probabilities ,computed from the model on the training data with y_tensor (which is binary 0/1).
Is this way of loss computation fine in Classification problem in pytorch? Shouldn't loss be computed between two probabilities set ideally ? If this is fine , then does loss function , BCELoss over here , scales the input in some manner ?

Any insights towards this will be highly appreciated


texts: Does class_weight solve unbalanced input for Decision Tree?

texts: I've read in sklearn's documentation that we have to take special care in balancing the input for a decision tree, but it doesn't tell you what function to use. However, I've found the parameter class_weight. If I use class_weight: balanced as a parameter, will that mean that I can omit balancing the input by hand?


texts:
I need to train a CNN (YOLO to be precise) on RGB + depth information. I have the RGB input as a separate PNG file and the depth map as a separate PNG file. These PNG files are 8 bit depth, 3 channel files. I need to create a single PNG file of 16 bit depth, 3 channels which is essentially a combination of both the original files. What is the right way to combine / encode RGB + depth information?

Data 1: R, G, B channels of 8 bits each
Data 2: D1, D2, D3 channels of 8 bits each
Required: C1, C2, C3 channels of 16 bits each

Current idea

I'm currently using a bitwise shifting.
C1 = R << 8 | D1
C2 = G << 8 | D2
C3 = B << 8 | D3
(shift by 8 bits left and then OR)

Questions

Notice that the channels left shifted (R, G and B) obtain higher significant bits and dominate over D1, D2, and D3 while forming C1, C2, C3. The CNN loses the information present in the depth map. Is there a better encoding?
A small note about the depth map. The depth information requires 16
bits and is contained in D2 and D3. D1 channel is a pseudo channel
with only gray scale information of the image. The final encoding
should focus on D2 and D3 more.

Another idea

To make sure no channel dominates over the other, I was thinking of interleaving the two 8 bit sequences to form a 16 bit sequence. If channel R has b1, b2, ..., b8 bit sequence and channel D1 has bb1, bb2, ..., bb8 bit sequence then the combined 16 bit C1 channel will have b1, bb1, b2, bb2, ..., b8, bb8 bit sequence. Any thoughts on this?


texts: The impact of using different scaling strategy with Clustering

texts:
I'm currently learning about clustering. To practice clustering, I am using this dataset.
After running K-means clustering for multiple values of k and plotting the results, I can see that scaling is affecting the results (within-cluster SSE) and I want to use this post to confirm my intuition as to why this is the case.
I don't believe that this is a meaningful reduction in the Within-Cluster SSE because the numerical distances are sensitive to scale, and I don't think that this has any effect on how accurate the model is. Is that intuition correct?
I just wasn't expecting the reduction to be this drastic between standardizing and normalizing.
Code and the results:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('customers.csv')
X = df.iloc[:, [3, 4]].to_numpy()

from sklearn.preprocessing import StandardScaler, MinMaxScaler
ssc, mmsc = StandardScaler(), MinMaxScaler()
X_ssc = ssc.fit_transform(X)
X_mmsc = mmsc.fit_transform(X)

from sklearn.cluster import KMeans
# Unscaled
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

# Standard Scaler
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X_ssc)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)

# MinMax scaler
k_vals = list(range(2, 21))
WCSSE = []
for k in k_vals:
    kmeans = KMeans(n_clusters=k)
    model = kmeans.fit(X_mmsc)
    WCSSE.append(model.inertia_)
plt.plot(WCSSE, marker='o', markersize=10)


texts: Do modern R and/or Python libraries make SQL obsolete?

texts:
I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to methodically process incoming data so that it can be standardized and put to work in reports, visualizations, and analytics projects. Prior to starting here, I had very little experience with SQL, aside from writing the most basic of queries. The vast majority of my analysis prep work was all done in R. My boss insists that I improve my SQL skills, even though there seem to be very few assignments that can't be done more efficiently and with far fewer lines of code using R packages like dplyr, data.table, and tidyr (to name a few). My question is--does this make sense?

A couple weeks ago, I found myself faced with the task of getting a list of column names for each row in a table that met certain criteria and concatenating them into a vector of strings. There was a tight deadline and at the time, I was experiencing some blockage and couldn't quite wrap my head around the problem. I asked my boss, who in turn asked my colleague to write a script TSQL to solve the problem. While he was working on it, I figured out a way to do it in R writing a fairly simple function and applying it over the data frame. My colleague came back with his script about two hours later. It was at least 75 lines comprising two nested for loops. I asked him to tell notify when it was finished running and he said it would take several hours. Meanwhile my R script was able to loop over the ~45,000 records in about 30 seconds.

Am I right to assume that R is a much better choice for cleaning and munging data?  Maybe the SQL developer in my office is just inept?  I'm curious if anyone who has worked with both R and SQL (or Python and SQL for that matter) have any thoughts on this.


texts: Is there a way to get the sample weights that were used to fit a scikit-learn estimator?

texts:
Many sklearn estimators support two weighting schemes:

Per-class weights: given when creating the estimator object (e.g. clf = DecisionTreeClassifier(class_weight={'Cat':0.4, 'Dog':0.6})
Per-sample weights: given when fitting a created object (e.g. clf.fit(X, y, sample_weight=[0.1, 0.1, 0.8])

For an already fitted estimator, I can get the per-class weights with clf.get_params()['class_weight'].

But what's the right way (if any) to get, given a fitted estimator, the per-sample weights that were used?


texts: Statistical machine translation word alignment for FR-ENG and ENG-FR: what is p(e) and p(f)?

texts:
I'm currently trying to implement this paper, but am struggling to understand some of the math here. I'm pretty sure I understand how to implement the E-step, but for the M-step, I'm confused on how to compute the M-step. It says just before section 3.1 that $p_1(x, z; \theta_1) = p(e)p(a, f|e; \theta_1)$, and then the same for $p_2$ but with $e$ and $f$ swapped. The second part of this makes sense to me, but what is $p(e)$ or $p(f)$? From my understanding, $e, f$ are sentences in the bi-text. So how would we compute the probability of a sentence?

It says earlier that $p(e)$ and $p(f)$ are arbitrary distributions that don't affect the optimization problem, but then how do we compute $p_1(x, z; \theta_1)$?

Thanks!


texts: Is there HMM that looks two elements back?

texts: The original HMM model looks only one element back (considers only previous state). Sometimes it viable to look two elements back. Is there an extension to HMM that allows that?


texts: What are the methods to ensure that the population split for A/B test is random?

texts: Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.


texts: Issue with using Sparse Data Frame in Mlxtend Apriori function

texts:
I am running python 2.7 in anaconda and have installed mlxtend. Based on the latest version of mlxtend, the aprioir class supports sparse dataframe as its input. I have over 500k products that I want to run a market basket analysis on.

I have created a onehot encoded sparse dataframe using a small dataset to test but I am running into df.to_coo() issue on the sparse data frame inside the mlextend apriori function.

Please find the code, the input data file and the errors I get here -
https://github.com/nshahHome/pycode

Click on the view code to see the files.

code = code2.py  ,  input data file= mbatest.txt   ,   errors =    code2-error.html (pdf version)  ,   condalist.txt

I expect the code to not throw errors and try to create frequent_itemsets. The set could be empy if there are no sets > min_support.


texts: Where do the "semantics" of a Bayesian network come from?

texts:
On Bayesian Networks, Ghahramani (2001) says:

A node is independent of its non-descendants given its parents.

This point is fundamental enough that Ghahramani calls it the “semantics” of a Bayesian network.  It is certainly useful, and it is simple enough to prove using d-separation.  But his characterization suggests that the property should be even more primitive than something provable by d-separation.
Overall, I feel that I am missing something.   Is there a more primitive way to verify the statement than to use d-separation?  Why does Gharamani equate that fact specifically to the semantics of a Bayesian network, rather than equating the semantics to the overall conditional independencies in the network (given by d-separation)? And if the statement is a consequence of d-separation, why focus on this fact specifically rather than the (arguably equally useful) fact about Markov blankets?
Reference: Ghahramani, Z. (2001). An introduction to hidden Markov models and Bayesian networks. In Hidden Markov models: applications in computer vision (pp. 9-41).


texts: Which models implicitly consider interaction between features?

texts:
I would like to understand more how different models (NN and RF specifically, but any other as well) consider interaction between features in tabular data?

For example, can the model figure out while training that "while feature 1 may not be directly correlated to the response, when feature 1 is low, feature 2 works really well at predicting the response". So the trained model would use the interaction between the two features and weight them accordingly, ie. weight feature 2 higher when feature 1 is low in the example.


texts: How to measure confidence in prediction?

texts:
I built a prediction model and predicted on new data. I now want to specify a value of my confidence in this predicted value, e.g. ranging from 0 to 1.
Three methods come to my mind

Built 100 models on bootstrapped data, predict 100 times on each new observation, then calculate confidence intervals. Smaller intervals mean higher confidence. High computational effort.
use oob predictions of a random forest from every tree
Bayesian methods can give confidence intervals through the posterior

Are there other/better ones?


texts: Cross-sell models and additional holders

texts:
I would like to pose a question about how to treat additional holders in the propensity-to-buy models of banking products.

Up to now I was only taking into considerations the clients as first holders.
For example, if a client ‘1’ appears as the first holder of a saving account ‘A’ with a balance at the end of the month of 100€ and as an additional holder of a saving account ‘B’ with a balance at the end of the month of 50€, the saving balance at the end of the month for the client is considered to be just 100€.
Moreover, if a client only appears as an additional holder (and he/she is not a first account holder of ANY product), he/she is dismissed by the model.

However I have been told to include additional holders in the models (additional holders have the same rights of the first holders).

One possibility is to recalculate all the variables summing up the position as first and additional holder (in the previous example, the balance at the end of the month of client ‘1’ would be 150€). Together with this, I would create some variable that represents the maximum degree of intervention of the client in the account (ex. 'first holder', 'second holder').

Another possibility would be to “double” all the variables, considering the client as first and additional holder (in the example, we would create two variables:  the balance at the end of the month as FH =100€, :  the balance at the end of the month as AH =50€).

Did any of you encounter a similar problem?It would be very helpful to understand how you solved it.

Thanks


texts: Classification Threshold Tuning with GridSearchCV

texts:
In Scikit-learn, GridSearchCV can be used to validate a model against a grid of parameters. A short example for grid-search cv against some of DecisionTreeClassifier parameters is given as follows:
model = DecisionTreeClassifier()
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
GSCV = GridSearchCV(model,params,scoring="f1_micro")
GSCV.fit(X_train,y_train)
GSCV.best_params_

Now, I am only concerned with binary classification. It is the case for many algorithms that they compute a probability score, and set the decision threshold at 0.5. My question is the following: If I want to consider the decision threshold as another parameter of the grid search (along with the existing parameters), is there a standard way to do this with GridSearchCV? For instance, something like the last parameter "decision_threshold" in the following tunegrid would be ideal:
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"], "decision_threshold": [0.1,0.2,...,0.9]}]

Needless to say, I am not interested in a particular solution working only for DecisionTreeClassifier. Instead,  a general solution  for any classifier that uses a probability decision threshold that can be tuned. Preferably, I would like to keep my current GridSearchCV  if possible.


texts: Orange 3 - How can a String feature behave as a coefficient?

texts:
I'm studying machine learning with data. I have a table including features and a target variable which is the price as in the following.

When I want to figure out the coefficients to obtain linear regression formula, I get the following table in which I don't understand how the coefficient is found for strings e.g. KARTAL, 3+1?

Secondly, There are coefficients as you can see, how can I write its formula since there are negative values and strings? I don't know how to apply them as a formula?


texts: Which scaling size is better? [0,1] or [-1,1] for LSTM?

texts: I see some scale their data between 0 and 1 and some others do that between -1 and 1. But which one is better? Or better to ask: How to make a true/good decision for that?


texts:
1) How does the batch normalization layer work with multi_gpu_model?

Is it calculated separately on each GPU, or is somehow synchronized between GPUs?

2) Which batch normalization parameters are saved when saving a model? (Since when using multiple-gpus in Keras, the original model must be saved, as suggested here)?

In the docs of multi_gpu_model is says:

  Specifically, this function implements single-machine multi-GPU data
  parallelism.

What does it mean for Batch Normalization?


texts: How to encode a class with 24,000 categories?

texts:
I'm currently working on a logistic regression model for genomics. One of the input fields I want to include as a covariate is genes. There are around 24,000 known genes. There are many features with this level of variability in computational biology and hundreds of thousands of samples are needed.

If I LabelEncoder() those 24K genes
and then OneHotEncoder() them ...

Is 24,000 columns going to make my keras training times unreasonable for a 2.2 GHz quad-core i7 CPU?

If so, is there a different approach to encoding that I can take with this?

Should I somehow try to dedicate a layer of my model to this feature?

Does this mean I need 24K input nodes?


texts: How to develop this type of neural network?

texts:
I was given this task to develop, this type of neural network but I never done speech recognition and that type of recognition in Matlab, so could anyone please help and give some help about this task.

Thanks in advance!

Picture of task


texts: open jupyter notebook file in linux from folder system

texts:
I have .ipynb files stored locally on my computer. The way I currently open these is by opening the terminal, typing "jupyter notebook" then navigating through jupyter notebook's file system to my .ipynb file.

It would be nice if I can just double click the .ipynb file in ubuntu's native file system and have it automatically open jupyter notebook with that file in my browser. Is this possible?


texts: How to cross validate WDBC.csv breast cancer classification dataset in Stacked Autoencoders?

texts:
I need kind guidance regarding the context of how to cross validate WDBC.csv (Wisconsin Breast cancer diagnostic) dataset for breast cancer binary classification in Stacked Autoencoders as I put the features as X matrix to autoencoder for training and feature learning and then learnt features are used for classification purpose. Don't know how to cross validate it?
Help is much appreciated in anticipation.


texts: Classification of data generated by radar using FFT

texts: I have time domain data which is having binary label in form of 0 and 1. I applied FFT to all the features and label as well. The problem is my output label is now not binary anymore. How to solve this classification problem?


texts: How to find possible subjects for given verb in everyday object domain

texts:
I am asking for tools (possibly in NLTK) or papers that talk about the following:

e.g. Input: Vase(Subject1) put(verb)

Ans I am looking for: flower, water

Is there a tool that can output subjects (objects) that can be associated to this verb? (I was going through VerbNet but didn't find anything)


texts: Simulating datasets of species abundances with varying degrees of turnover in R

texts:
I am working with a data set containing the abundance of 100+ species at 50+ locations over 20+ time-points. I want to simulate equivalent data sets with varying degrees of community turnover i.e. abundant species staying abundant/rare staying rare --> abundant becoming rare and vice versa while maintaining the species abundance distribution in the original data set. I have seen mentions of resampling simulation techniques using bootstrapping where the desired difference between "template" and simulated data sets can be specified in terms of the proportion of all species that were present in both assemblages.

I am wondering if that is the best way forward to get the desired simulated data sets and if anyone could point me towards a tutorial on how to do this in R/give me some tips themselves.

Thanks!


texts: How to count categorization instances in a NN?

texts:
Say one-hot encoding is the perfect way to represent a series of objects such as clothing items. Ie: A hat is [1 0 0], a tie is [0 1 0].

I want to predict what a customer buys if they spend a certain amount of money, which will serve as an input to the NN. Eg: For one example, the customer spends $22. The thing is, the customer may buy multiples of the same clothing item.

What would be the best way to make the neural network output something like [2 1 0] to show its prediction as 2 hats and 1 tie, given the input of $22?

As far as I understand binary crossentropy is good for multilabel problems, but nowhere could I find an example of how I could apply this when the same label occurs more than once.

The only solution I could think of was turning this into a regression problem, but doing so would seem bad since this is a categorical problem, and the network should output fixed natural numbers depending upon the number of times an object is predicted.


texts: CNN - multidimensional matrix as input or parallel input for parallel CNN

texts:
I want to run CNN on 20 channels of images.
One way is to run on a 20-channel multidimensional matrix (like RGB ). Another way is to run 20 CNN on one channel at a time ( R apart from G separately from B as separate inputs ) and finally connect by concatenate

What is more appropriate to do and is there a difference in the results?


texts:
I've a dat file in the following format which I'm trying to load using pandas.

28::Persuasion (1995)::Romance
29::City of Lost Children, The (1995)::Adventure|Sci-Fi
30::Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)::Drama
31::Dangerous Minds (1995)::Drama
32::Twelve Monkeys (1995)::Drama|Sci-Fi
33::Wings of Courage (1995)::Adventure|Romance
34::Babe (1995)::Children's|Comedy|Drama

Code

movies = pd.read_csv('movies.dat')

I'm getting this error

ParserError: Error tokenizing data. C error: Expected 1 fields in line 11, saw 2

Line 11:

11::American President, The (1995)::Comedy|Drama|Romance

I understand that it's because of the comma. One option I found is to skip these lines, which I don't want to. Is it possible to set newline as the delimiter here? Any help will be truly appreciated.


texts: How to train a model on a data where there are multiple data inside a data point?

texts:
I'm trying to do prediction on capacity column, however each data point consist of more data.

Each data point represent a cycle data. Each cycle has a capacity. Each cycle runs for some time duration, and in that duration some data is collected over which capacity is dependant

I tried exploding the dataset and copying the capacity values to each row, but that shouldn't be the case because each row will get different capacity predicted. Is there a way to train such kind of dataset?


texts:
I am looking for a python package that calculates how well one sentence of a natural text follows the next. One could simply count how many identical words are in the next sentence but the better method would be to compare word similarities using something like word vectors (=semantically similar words instead of exact matches or synonyms).

Coherent:

Tom loves reading books.
He prefers reading books at library.
So he always goes to library.

Non-coherent:

Tom loves reading books.
He missed his lunch today.
So he always goes to library.

I guess there must be several well written packages for such an automatic assessment (or similar methods) but I just can't find it. Any ideas?


texts: How do I properly write scipy.stats.binom.cdf() details

texts:
I need to calculate the probability of my random variable being $\le 0$. It's a binomial distribution, $10000$ trials, probability of success is $\frac{10}{19}$ (roughly $0.53$).

How do I properly use the scipy.stats.binom.cdf() to do that?

I've tried the following:

stats.binom(10000, a).cdf(0)

But it gives me an answer $0$.

I feel like I might be missing something about the formula itself.


texts: Is there any data tidying tool for python/pandas similar to R tidyr tool?

texts: I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and spread(), which can be found in R tidyr tool.


texts: How to aggregate Weather Data from county to state level?

texts:
I am doing a prediction of forest fire risk using the weather data and the fire incident data. I have data related to different county and the weather data on the day of the incident. As the incident data shows all incident occurred, I am generating negative data of fire event not occuring in the days when the incident is not reported.
My question is that on what basis should I give the weather data for the negative data points. Because they don't belong to any county(I have data of all the counties from california)
Other question is how to predict a fire event for the entire state if I have county data. Can I predict that a fire incident may occur in the state. Please suggest how to aggregate data and approach this problem.


texts: Weighted degree in Multidimensional networks

texts:
Does there exist a definition for weighted degrees of multidimensional networks?

I understand that the basic definition would be:

  Let $v\in V$ be a node of a network $G=(N,V,L)$. The function Weighted Degree: $$V \times P(L) \longrightarrow \mathbb{N}$$ is defined as

  $${weighted\_degree}(v,D)=\sum_{(u,v,d)\in G, d\in D}\ weight(u,v)$$

  where $D\subseteq L$ and $weight(u,v)$ is the weight of the edge $(u,v)$.

I cannot find this definition anywhere on the Internet. But would that be right? Is a multidimensional version of any interest?


texts: We are building ASR system for native spanish language. We are looking for the spanish speech datasets to train our model. If anyone can provide the links/sources of spanish speech datasets(paid/free) it would be very useful.


texts: Stochastic gradient descent in logistic regression

texts:
I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through.

I'm using logistic regression with R's glmnet package and alpha = 0 for ridge regression.

I'm using ridge regression actually since lasso deleted all my variables and gave very low area under curve (0.52) but with ridge there isn't much of a difference (0.61).

My dependent variable/output is probability of click, based on if there is a click or not in historical data.

The independent variables are state, city, device, user age, user gender, IP carrier, keyword, mobile manufacturer, ad template, browser version, browser family, OS version and OS family.

Of these, for prediction I'm using state, device, user age, user gender, IP carrier, browser version, browser family, OS version and OS family; I am not using keyword or template since we want to reject a user request before deep diving in our system and selecting a keyword or template. I am not using city because they are too many or mobile manufacturer because they are too few.

Is that okay or should I be using the rejected variables?

To start, I create a sparse matrix from my variables which are mapped against the column of clicks that have yes or no values.

After training the model, I save the coefficients and intercept. These are used for new incoming requests using the formula for logistic regression:

Where a is intercept, k is the ith coefficient and x is the ith variable value.

Is my approach correct so far?

Simple GLM in R (that is where there is no regularized regression, right?) gave me 0.56 AUC. With regularization I get 0.61 but there is no distinct threshold where we could say that above 0.xx its mostly ones and below it most zeros are covered; actually, the max probability that a click didn't happen is almost always greater than the max probability that a click happened.

So basically what should I do?

I have read how stochastic gradient descent is an effective technique in logit so how do I implement stochastic gradient descent in R? If it's not straightforward, is there a way to implement this system in Python? Is SGD implemented after generating a regularized logistic regression model or is it a different process altogether?

Also there is an algorithm called follow the regularized leader (FTRL) that is used in click-through rate prediction. Is there a sample code and use of FTRL that I could go through?


texts: Can I update my model using partial_fit after training my model using fit?

texts:
I have trained my model using MLPClassifier using fit method and saving in pickle object for the first time.

clf = MLPClassifier(hidden_layer_sizes=(50,50,5), max_iter=100, alpha=0.0001,
                     solver='sgd', verbose=10,  random_state=21,tol=0.000000001)
clf.fit(X_old, y_old)
joblib.dump(clf, 'saved_model_clf.pkl')

After few hours I am getting new data from online and I am updating my model using partial_fit .

clf_load = joblib.load('saved_model_clf.pkl')
y_new = to_categorical(y_new,num_classes=3) #I knew classes from training set
clf_load.partial_fit(X_new, y_new, classes=list(range(y_new.shape[1])))
joblib.dump(clf_load, 'saved_model_clf.pkl')

After partial_fit I'm predicting my results

clf_pred = joblib.load('saved_model_clf.pkl')
predictions = clf_pred.predict(df[0:1])

I found that my model is not learning on new incoming data while performing Prediction after the partial_fit

I am not sure where is the mistake

Case 1:
I am doing mistake in saving and retrieving pickle object .

Case 2:
My assumption of updating my model is wrong .

Can someone help me out to solve my problem please ?

Example dataset for clf.fit

X_old = {"TABLE":"XYZ","MODULE":"ABY", "MAX_TIME":647,"RESPONSE":"SUCCESS","SEVERITY":"Critical", "SERVICE":"ZC Service", "EVENT_NAME":"SendEmail"}

Y_old = {"Classification":"Ignore_Anyway"}

Example dataset for clf.partial_fit

X_new = {"TABLE":"XYZ","MODULE":"ABY", "MAX_TIME":647,"RESPONSE":"SUCCESS","SEVERITY":"Critical", "SERVICE":"ZC Service", "EVENT_NAME":"SendEmail"}

Y_new = {"Classification":"Not_Ignore"}


texts: Feauture selection for clustering regarding zero-correlated feature

texts: I want to cluster a 5 feature data-set. Firstly to explore the data I did a correlation matrix to see if some features where highly correlated so I could reduce them. Then I saw a feature that have close to zero correlation against all the other features. This got me wondering if I should exclude this parameter since it acts as a kind of "noise" relatively to all the other features. What's your opinion?


texts: In the context of image binary classification, is it necessary to divide dataset into true positive, false positive, true negative, false negative?

texts:
I am working through this course. It seems that the professor is not dividing the dataset into true positive, false positive, true negative, and false negative.

In the context of image binary classification, is it necessary to divide the dataset into true positive, false positive, true negative, and false negative?


texts:
I had generate in Ra logistic model using glm using binomial as family, but each session that I started in R the variable that I used to store the glm output gives me another output.
Why this happen? Why R didn’t save/store the output from glm in the variable that I created for.
I’ve been googling for an answer, but I didn’t find anything, so maybe you could help me to understand.
Thanks in advance!


texts: Classification with a lot of the classes

texts:
I’m trying to make model which will classify text into about 500 different classes. I think that I have to customize architecture of the Pooling Classifier which looks now like this:

(1): PoolingLinearClassifier(
(layers): Sequential(
   (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
   (1): Dropout(p=0.2, inplace=False)
   (2): Linear(in_features=1200, out_features=50, bias=True)
   (3): ReLU(inplace=True)
   (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
   (5): Dropout(p=0.1, inplace=False)
   (6): Linear(in_features=50, out_features=498, bias=True)
)

I think that I have to change in (2): Linear layer to have more out_features because in the last (6) Linear layer I predict more out_features than I’ve got in_features. What do you think?

Best regards


texts: Why does map reduce have a shuffle step?

texts: I'm looking at a diagram of map reduce where there is a map step, a shuffle step and then the reduce step. Why shuffle?


texts: Visualizing a big number of polynomials as atlas of graphs?

texts:
Suppose you have a big number of polynomials such as xyz+xy+z where xyz, xy and z are monomials. You wanna visualize the polynomials together something such as atlas below and show how they are related graphically.

The example is from here.

The edges could correspond to sum while the vertices could be monomials. It is also possible to do it so that each variable in monomial has their own vertex while two different kinds of edges are drawn: edges inside monomials and distinct edges to connect monomials of the same polynomial. Other visualizations are possible as well.

How can you visualize a big number of polynomials graphically?

P.s. Aspiring discussion in Mathematica SE.


texts: Can theano work on mapreduce or on spark?

texts:
I am not sure whether the Theano library can be used to write parallelized code in map reduce or in spark. Any expert opinion is welcome. A discussion was on at:
Theano-dev


texts: How to account for reduced student capacity when calculating program retention?

texts:
I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer.
I am working on calculating our student retention between Afterschool 2019-20 and SummerCamp 2020 to see which students we retained, however, Between AS2019-20 and SC2020, we reduced our maximum student capacity by 10. I figure that this definitely impacts the retention rate calculation, but I have no idea how to account for this reduction in capacity. Does this impact anything, or am I overthinking?
I have to assume it would be some sort of calculation like returned students / eligible students = retention and then maybe dividing that percentage by the quotient of SC2020 capacity / AS2019-20 capacity = proportion to account for the proportion. Is this correct?


texts: How does keras (TF) know how to differentiate my custom loss?

texts:
Suppose I have this custom loss:

def custom_loss(y_true, y_pred):
            out = K.clip(y_pred, 1e-8, 1-1e-8)
            log_lik = y_true*K.log(out)
            return K.sum(-log_lik*advantages)

How does keras (with TF as backend) know how to differentiate in terms of the input specifically and ignore the 'advantages' simply as coefficient?
Does it do it numerically? If so is that the same as with it's own loss functions or only with custom?


texts: LSTM for time series forecasting with H20.ai

texts:
I want to implement a time-series prediction model using LSTMs like the one mentioned here: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

The model works pretty well using a similar Keras code above, though I want to implement the same using H20.ai

This documentation here on H2O Deep Water says "The H2O Deep Water project supports CNNs and RNNs though third-party integrations of other deep learning libraries such as TensorFlow, Caffe and MXNet.". Though there are no demos for the same.

In this conversation here in Jan '16 on 'Recurrent Neural Networks in H2O for time series prediction?', they say it hasn't been implemented yet.

Any advice or links to similar implementations on H2O would be deeply appreciated

Note: I am looking for a data parallel architecture implementation rather than model parallelism


texts: How to use scikit-learn normalize data to [-1, 1]?

texts: I am using scikit-learn MinMaxScaler() to normalize to $[0, 1]$, but I want to normalize to $[-1, 1].$ What function do I have to use to normalize this way?


texts: A good reference for the back propagation algorithm?

texts:
I'm trying to learn more about the fundamentals of neural networks. I feel like I understand the basics of back propagation, but I want to solidify the details in my mind.

I was working through Ian Goodfellow's famous Deep Learning text. However, I found their exposition of back propagation unsatisfactory. They model the computational graph the wrong way, making the weights into the vertices and the operators into the edges. But this makes no sense, as it is the operations (not the variables) which must have an in-degree and out-degree to them.

I was wondering if there is a better reference, either in a textbook, a paper, or a blog that would rigorously outline the details of back propagation in its full generality.


texts: Communication between Keras and Backend like Tensorflow, Theano

texts:
Keras is a high level neural network API providing python library which uses tensor flow or theano or cntk as backend. What are the primary roles of backend libraries? Is it implementation? or Is it computational heavylifting using GPU, threading etc?

I couldn't find any good resources online to understand how keras interacts with tensor flow or theano(backend) session. Any such resource or direction for understanding interaction is helpful !


texts: Deduplicating e-commerce products based on title/description/images

texts:
I am trying to clean a set of e-commerce data.
Data consists of products which has multiple images, a title and a description.
Some of these products has duplicates and I need to de-duplicate the set.
I have worked on NN's and Genetic Algorithms back in college but that's it.

What I need is a starting point to dive into. What methods/techniques should I use, what should I learn first?


texts: K-Fold and Random Subsampling (RSS) Dataset generation?

texts: Let say if I have a large dataset of 300k instances with 200 features, I want to reduce its size. Can I apply K-Fold technique to the 200 features then the trimmed dataset are applied with RSS to trim the instances? Its like K-Fold to reduce Features then RSS to reduce instances then I get a small-scaled dataset with the less number of features and randomized smaller instances. Can this be done?


texts: NLP+Non Text Features . How to give more weight to non NLP features?

texts:
I have some text and some metadata associated with it. Before classification I have made feature matrix in such way:

Non Text features + TFIDF vectorizer matrix

So if had 5 non text features (like position of text, does text contain digits etc) and TFIDF matrix contains 100 columns then total columns fed to classifier is 105. I am using DecisionTree classifier.

Now, the problem is that, the classifier is using only features derived from text to make decisions. In some cases of mis-classifications I observed that the non text feature (position of text) was actually the decision factor.

I think it is because the number of features generated by TFIDF vectorizer are too many (1800+). Any idea on how to make other features also contributor to classification.

In case you want to see code :

train_x =  train[["Diff","col9","col10", "col6"]]  # Non text features
tfidf_vect = TfidfVectorizer(analyzer='word', token_pattern=r'\w{1,}', use_idf=False, ngram_range=(1,3))
train_x1 = pd.concat([train_x, xtrain_tfidf_ngram_df], axis=1)  #Merge non text and text features
model.fit(train_x1, y=train_y)
predicted=model.predict(test_x1)

Sample training matrix after merge:

     index  Diff  col9  col10  col6    0    1    2    3    4    5    6    7  ...  1875  1876  1877  1878  1879  1880  1881  1882  1883  1884  1885  1886
1887
0        0 -2158     1      1     0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0
 0.0
1        1    15     1      1     1  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0
 0.0
2        2    27     1      1     0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0
 0.0


texts:
I am new to ML and still learning it.

My problem is to identify duplicate products. I have a dataset containing product details such as name, colour, size, description, features etc (there are roughly 70 columns).

I need to remove duplicate products.

I just completed some of the supervised ML model(classification and regression) and unsupervised clustering(K means and HC). I am also on the way of learning w2v and d2v.

But due to time constrain, I need to deliver a solution to the above problem statement. I am unsure as to how to proceed.

Any help and guidance would be appreciated


texts: Drastic shift in feature importance upon adding other features

texts:
I have a model (GBDT) where adding a feature X is not important (according to SHAP), but when I add other features, and add X again, now feature X is the second most important!
What could explain that? How do I investigate what is going on?


texts: Not able to connect to GPU on Google Colab


texts:
I have studied the paper Fully Convolutional Networks for Semantic Segmentation (Shelhamer, Long and Darrell) and understand the process. They provide extensive code in Caffe. I have been analyzing the code for some time, but I however don't understand how to use this code for my own data. I have everything installed correctly (Caffe, Python, etc). I would, for example, like to:

Test the code on an arbitrary image and obtain a segmented image as an output.
Train the network on own image pairs {image, segmented image}.

Is there a tutorial I could follow on how to use this code. Or could someone give me a some guidelines. I would appreciate it very much.


texts: Why does a filter need to be applied to the output of the input gate before cell state is added to?

texts:
In a neural network there are 4 gates: input, output, forget and a gate whose output performs element wise multiplication with the output of the input gate, which is added to the cell state (I don't know the name of this gate, but it's the one in the below picture with the output C_tilde).

Why is the addition of the C_tilde gate required in the model? In order to allow the input gate to subtract from the cell state, we could change the activation function that results in i_t from sigmoid to tanh and remove the C_tilde gate.

My reasoning is that the input gate already has a weight matrix W_i that can is being multiplied to the input gate's input, hence it already does filtering. However, when C_tilde is multiplied with i_t that seems to be another unnecessary filter.

My proposed input gate would then be i_t = tanh(W_i * [h_t-1, x_t] + b_i) and i_t would directly be added to C_t (C_t = f_t * C_t + i_t rather than C_t = f_t * C_t + i_t * C_tilde_t).


texts: Which one is easier to debug between theano and caffe?

texts: I am investigating whether to use theano or caffe for convnets. I would like to know which one provides a better debug environment. In caffe, it seems you don't write any code just a config in protobuf - how do you debug the convnet then?


texts:
I would like to generate synthetic data which are ordinal, i.e. ordered, in Python.  But how would I do this?  What are the differences in generating ordinal data vs categorical data?

I'm reading the paper "Automatic Discovery of the Statistical Types of Variables in a Dataset," by Valera and Ghahramani.  In it, they write: "We account for categorical data by sampling a multinomial variable with $R$ categories, where the probability of the categories is sampled from a Dirichlet distribution....To account for ordinal observations, we first sample the first variable in our dataset from a uniform distribution in the interval $(0,R)$, which we randomly divide into $R$ categories that correspond to the ordinal variable in our dataset."

Can someone help me understand the latter part about generating ordinal data?  Thank you!


texts: training neural net with multiple sets of time-series data

texts:
I have the following data

($x^1_i$, $y^1_i$) for $i=1,2,...N_1$

($x^2_i$, $y^2_i$) for $i=1,2,...N_2$

...

($x^m_i$, $y^m_i$) for $i=1,2,...N_m$

Is it possible to train a neural net to produce some $y_k$ where $k<=min(N)$ given a input ${x_1, x_2, ..., x_{k-1}}$?

If so any suggestion of documentation/ library I can look at (preferably python)?


texts:
I wanted to know if there is any methodology in Deep/Machine learning, where given a set of input/output values, it can derive rules for the same.

Lets say I generate training input and output by $y=x^2$

i/p  |  o/p
 0       0
 2       4
 .       .
1000   1000000

It sort of generate rule like, $y=x*x$


texts:
Updated to 3.19 using Anaconda navigator, from CLI I receive these messages

orange-canvas
realloc(): invalid old size
Aborted (core dumped)

python3 -m Orange.canvas
Segmentation fault (core dumped)

Using Anaconda navigator got this log

/home/davide445/.anaconda/navigator/scripts/orange3.sh: line 3:  8261 Aborted                 (core dumped) orange-canvas > /home/davide445/.anaconda/navigator/scripts/orange3-out-1.txt 2> /home/davide445/.anaconda/navigator/scripts/orange3-err-1.txt

Same problem with 3.20.1 or 3.18, is working using 3.17. Working on Kubuntu 18.04 LTS.
Any suggestion for troubleshooting, many thanks.


texts:
I am analysing a bunch of data files which represent responsiveness of cells to addition of a drug. If a drug is not added, cell responds normally, if it is added, it shows abnormal patterns:
, .
We decided to analyse this using an amplitude histogram, in order to distinguish between a change in amplitude and in change of a probability of elliciting the binary response. What we get with file 1 is :

So we fit a pdf on it:

But the issue is: how should we quantify this change: an obvious answer is to do an integral, but as the addition of the drug may increase the 'noise' spectrum (first peak auc) it is not entirely obvious that this is what should be done. Furthermore, other data files do not necessarily contain the same ammount of ion channels (responsive units), so due to biological variability their pdfs may have more or less peaks. How does one summarise the characteristics of multiple peak gaussians should the number of peaks vary? I was thinking about integral over freq ranges, but that will certainly differ between a file with two gaussian peaks, and one with 5: how could one make a summary statistics of an array of gaussian distributions with unknown ammount of peaks?

Another analysis file does literally not contain the triple openings: , . What is a reasonable way of showing the difference between multiple peak gaussians in a condition a and condition b, if the number of peaks is unknwon?

I was thinking of integral, but that looses the potential increase in noise caused by treatment. I was thinking of integral/frequency delta, but I have never seen it published in literature, and I would be really happy if I could support my reasoning for choosing this analysis with some previously published work. Am I missing something really obvious, or is there no simple answer to this? Thanks, J.


texts: What are the 'hottest' future areas of Machine Learning and Data Science?

texts:
What would you guys identify as the most important areas of machine learning and data science in the future? What kind of business do you see companies focused on ML doing in the future?

To give some background to the question, I'm asking this as someone hugely interested in ML at a conceptual level but with no real experience on it save for some online courses. My intention is to learn the necessary math, coding, etc. and as I'm very entrepreneurial person, eventually found my own ML company. I've found that the field is vast and thus I feel one of the biggest challenges is to select a specific area (I guess method would be a better word here) to focus on and build expertise.


texts:
I have a dataset containing headlines and sentiment related to those headlines. The headlines have been filtered out from another bigger dataset using the following criteria: keep the ones that have a very negative or a very positive sentiment. At the end, I have a dataset with a very positve and a very negative sentiment headlines.
My goal is to create a deep learning classifier using tf and keras in order to classify new observations into three class: positive sentiment, negative sentiment and neutral sentiment. In other words, my goal is to use a binary labelled dataset to create a classifier that outputs a 3-label classification and I want to do it by predicting the probability of a headline to be positive or negative.
If the predicted probabilities of a headline are:
p(positive) = 80%
p(negative) = 20%

than the headline is positive. But if:
p(positive) = 50%
p(negative) = 50%

than the headline is neutral.
What do you think?


texts:
I've the following dataset:

**Strenght  Movie1  Movie2**
  23            2        3
  80            1        2
  10            4        3

And I want to create a graph with the relationships between movies having the first column as the strenght of the relationship. How can I do this using R?

Many thanks!


texts: Making Keras RNN to proceed input sequence step by step

texts:
I'm currently trying to create a neural network for playing Tetris. I'm using evolutionary algorythms for it's learning, so the behavior that I need to get from the neural network is the following:

Create a RNN model
Give it a single input (current field state) and take an answer that takes into account previous moves
Use the neural network response to make a move in the game
Get new game field state, pass it to the neural network, repeat from point (2)

The problem is that on the same inputs I get the same outputs, which indicates that the RNN does not take into account its previous actions:
model = keras.Sequential(
        [
            layers.SimpleRNN(10, return_sequences=True) ,
            layers.SimpleRNN(10, return_sequences=True) ,
            layers.Dense(3, input_dim =10, activation='softmax')
        ]
    )

    x1 = np.random.rand(1, 1, inp_size)

    y1 = model(x1)
    y2 = model(x1)
    y3 = model(x1)

    print(y1)
    print(y2)
    print(y3)

Output:
tf.Tensor([[[0.3274583  0.42117843 0.25136325]]], shape=(1, 1, 3), dtype=float32)
tf.Tensor([[[0.3274583  0.42117843 0.25136325]]], shape=(1, 1, 3), dtype=float32)
tf.Tensor([[[0.3274583  0.42117843 0.25136325]]], shape=(1, 1, 3), dtype=float32)

So, when i do the same thing, but feed input as a sequence packed to tensor:
x = np.ones((1, 3, inp_size))
y = model(x)
print(y)

I get different results on the same inputs:
tf.Tensor(
[[[0.23801968 0.4496906  0.31228966]
  [0.30908147 0.3184522  0.37246636]
  [0.2849004  0.5014079  0.21369171]]], shape=(1, 3, 3), dtype=float32)

How can I create a RNN model which will not need a full match history as an input sequence to take proper decisions?
The question is about opportunity to force network to work step by step, as if processing the sequence, but at the same time receiving only a SINGLE element at the input and immediately giving an answer (since the n-th element of the input sequence depends on the n-1 network output, and we cannot get it until previous output will not be processed by the game)


texts: SHAP value analysis gives different feature importance on train and test set

texts:
Should SHAP value analysis be done on the train or test set?

What does it mean if the feature importance based on mean |SHAP value|  is different between the train and test set of my lightgbm model?

I intend to use SHAP analysis to identify how each feature contributes to each individual prediction and possibly identify individual predictions that are anomalous. For instance, if the individual prediction's top (+/-) contributing features are vastly different from that of the model's feature importance, then this prediction is less trustworthy. Does this approach make sense?


texts: Why can decision trees have a high amount of variance

texts:
I've heard that decision trees can have a high amount of variance, and that for a data set $D$ split into test/train the decision tree could be quite different depending on how the data was split. Apparently, this provides motivation for algorithms such as Random Forest.

Is this correct? Why does a decision tree suffer from high variability?

edit

just to note that I don't really follow the current answer
 and haven't been able to solve that in the comments.


texts: LFR as synthesized data for dynamic community detection methods

texts:
In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper Overlapping Communities in Dynamic Networks: Their Detection and Mobile Applications, LFR is used. But I don't know how this dataset is generated.

Here is the code of this paper and ground truth included is only for one snapshot.

Question: How is the dataset generated?


texts: Different definitions of Macro F1 score, which one is used in Scikit-learn?

texts:
In this article Macro F1 and Macro F1 two different definitions of the F1 used in the literature are demonstrated.
The first F1 score is computed such as:

  F1 scores are computed for each class and then averaged via
  arithmetic mean

The second such as:

  The harmonic mean is computed over the arithmetic means of precision
  and recall

I was wondering which definition is actually implemented in Scikit-learn.
From the docs I cannot derive which definition is used:

  Calculate metrics for each label, and find their unweighted mean. This
  does not take label imbalance into account.


texts: Are there any tools/ python packages for Fuzzy Grouping?

texts:
I'm trying to get to a tool for Fuzzy Grouping as I do not have a reference column for matching the string. Is there any package on Python or R?
I looked at a package called textpack but the results aren't good.

found here:
https://pypi.org/project/textpack/

I'd really appreciate if someone could suggest a tool or a package so I can go ahead and research.


texts: Classification with feature not available at time of model creation

texts:
I have problem statement to predict the probability of solving a task depending on multiple features for e.g. when the task was created, the time needed to work on a task, etc Please find a dummy snippet attached

task_id  date_time_open    time_needed   day_created  time_created   status

aa      12/09/2019             20 hrs     Tuesday        3 pm      done
cc      17/10/2019             4 hrs      Friday        10 pm      not_done

I know I can run a classification model to identify the class. However, things complicate when  I add a time dimension to it since the data set now gets an added feature which highly impacts the status

The task was scanned at suppose 7 pm and a new feature added for 7 pm

task_id  date_time_tsk_open    time_needed     day_created  time_created    status_7pm     status

    aa      12/09/2019               20 hrs     tuesday       3pm              done           done
    cc      17/10/2019               4 hrs      friday        10 pm            done           not_done
    dd      19/10/2019               6 hrs      friday        2 pm             done          done
    ff      19/10/2019               9 hrs      Monday        4 pm             not_done      not_done

The task id was again scanned at a fixed interval of 1 hr and added new features to data

task_id  date_time_tsk_open    time_needed     day_created  time_created    status_8pm     status

    aa      12/09/2019               20 hrs     tuesday       3pm              done         done
    cc      17/10/2019               4 hrs      friday        10 pm            not_done     not_done
    dd      19/10/2019               6 hrs      friday        2 pm             done            done
    ff      19/10/2019               9 hrs      Monday        4 pm             not_done        not_done

The final prediction of status == resolved / un_resolved in my understanding should be based on features including status_7pm and status_8pm.

How should the data structure for training such a classification model look like to generate a prediction at time 9 pm for sample task ff respectively

  task_id  date_time_tsk_open    time_needed     day_created  time_created    status_7pm status_8pm     status

    ff      19/10/2019               9 hrs      Monday        4 pm            not_done           not_done      not_done

I assume the classification model should be trained on all status_1, status_2 ....status_8pm to classify  status. Or would the model be trained every time in memory once it gets a new column updated status every hour


texts: Why is it wrong to train and test a model on the same dataset?

texts: What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?


texts: How are natural language generation algorithms given a target

texts:
I've started learning about NLP and NLG and I'm fascinated! I've been blown away by the things I've seen from NLP; but I have a few questions about NLG. All my questions boil down to this:

Given a network or Markov chain how does one specify what you want the system to talk about?

To explain this a little; if I ask my 5 year old nephew to tell me something he'll talk about his toys, or what's on TV etc. whatever he decides - but I can also ask him to talk to me about something specific eg. tell me about dinosaurs, then he'll give me a few sentences about his favourite dino. How does one specify they want their NLG system to talk about dinosaurs specifically once you've fed the system the whole of Wikipedia?


texts: Joint spatial clustering: How to force clusters to minimally contain datapoints from all datasets

texts:
I want to jointly cluster datapoints coming from different datasets (50 datasets with around 2000 points each). I would like to then extract information associated to the datapoints belonging to the different clusters to compare aspects of the datasets.

Now my problem is that if I just place the datapoints originating from the different datasets into the same feature space, too few clusters comprise datapoints coming from all datasets simultaneously, so that after filtering I'm left with too few clusters (I tried with kmeans and similar methods).

My question is: Which is the best way to jointly cluster my points while imposing the condition that a given cluster should contain points from all datasets? The ideal solution would also allow some outliers to not fulfill this condition. The first thing I could think about is to define distances between points and clusters which are updated depending on whether a point belonging to the same dataset is already present in the cluster? Seems too far fetched though.

I would appreciate any ideas, thanks a lot in advance!

Edit: Three of my six features are spatial coordinates and ideally I would also want the clusters to be connected within a given dataset.


texts: Dissmissing features based on correlation with target variable


texts:
I am a bit confused by the difference between the terms "Machine Learning" and "Deep Learning". I have Googled it and read many articles, but it is still not very clear to me.

A known definition of Machine Learning by Tom Mitchell is:

  A computer program is said to learn from experience E with respect to
  some class of tasks T and performance measure P, if its performance at
  tasks in T, as measured by P, improves with experience E.

If I take an image classification problem of classifying dogs and cats as my taks T, from this definition I understand that if I would give a ML algorithm a bunch of images of dogs and cats (experience E), the ML algorithm could learn how to distinguish a new image as being either a dog or cat (provided the performance measure P is well defined).

Then comes Deep Learning. I understand that Deep Learning is part of Machine Learning, and that the above definition holds. The performance at task T improves with experience E. All fine till now.

This blog  states that there is a difference between Machine Learning and Deep Learning. The difference according to Adil is that in (Traditional) Machine Learning the features have to be hand-crafted, whereas in Deep Learning the features are learned. The following figures clarify his statement.

I am confused by the fact that in (Traditional) Machine Learning the features have to be hand-crafted. From the above definition by Tom Mitchell, I would think that these features would be learned from experience E and performance P. What could otherwise be learned in Machine Learning?

In Deep Learning I understand that from experience you learn the features and how they relate to each other to improve the performance. Could I conclude that in Machine Learning features have to be hand-crafted and what is learned is the combination of features? Or am I missing something else?


texts:
Let $\{p_i : i \in \mathbb{Z}\}$ be an i.i.d sequence with $p_i \in (0, 1)$.
Fix this random environment, then consider the random walk
$$P[ X_{n+1} = i + 1 | X_n = i] = 1 − P[ X_{n+1} = i − 1 | X_n = i] = p_i.$$

(a) Let $p_i$ be $\frac13$ or $\frac23$, with probability $\frac12$ each. Find a deterministic sequence $a_n$ such that $\frac{X_n}{a_n}$
seems to be converging in distribution to a non-degenerate variable.

(b) Find a distribution with $\mathbb{E}[p_i] = \frac12$ such that $X_n$ is transient.

(c) In both cases, draw pictures of the space-time trajectorie

I need to simulate the above,I have started working on (a) but I don't get how to find such a sequence.
here is my code for the random walk:

import random
import numpy as np
import matplotlib.pyplot as plt

prob = [0.333333333333333333,0.666666666666666666]

start = 0
positions = [start]

rr = np.random.random(1000)
downp = rr < prob[0]
upp = rr > prob[1]

for idownp, iupp in zip(downp, upp):
    down = idownp and positions[-1] > -100
    up = iupp and positions[-1] < 100
    positions.append(positions[-1] - down + up)

plt.plot(positions)
plt.show()

how can I proceed??


texts: TensorFlow stopped working after installing opencv

texts:
I am working on projects that use TensorFlow and Keras and I did the hello-world MNIST project successfully and later I tried to run some other project and TF suddenly stopped working. Now it doesn't even work on the old project. I am working in conda envs and my editor is VSC. Here is the error that I am getting:
Traceback (most recent call last):
  File "c:/Users/miliv/MLRadionica/TensProba.py", line 3, in <module>
    from tensorflow import nn
  File "C:\Users\miliv\Miniconda3\envs\tf\lib\site-packages\tensorflow\__init__.py", line 101, in <module>
    from tensorflow_core import *
  File "C:\Users\miliv\Miniconda3\envs\tf\lib\site-packages\tensorflow_core\__init__.py", line 40, in <module>
    from tensorflow.python.tools import module_util as _module_util
ModuleNotFoundError: No module named 'tensorflow.python.tools'; 'tensorflow.python' is not a package

I tried reinstalling TensorFlow a couple of times but it didn't help. The only thing that comes to my mind is that it has to do something with installing OpenCV maybe because of some packages, but I haven't changed anything else.
I have tried creating a new conda env with TF and without OpenCV but it still throws me the same error.


texts: Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

texts:
I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm that can take data with more than 2 dimensions and cluster the data similar to how k-means clustering works with 2 dimensions. Ultimately, I would like to take financial data (e.g. P/E, operating margin, earnings growth) and divide different stocks into groups (e.g. growth stocks vs. value stocks).

I have tried googling to see if k-means clustering would work with a dataset that has more than just X and Y values but couldn't find anything super helpful. Also, I thought about using k-nearest neighbor but I don't have a large amount of "training" data so I think an unsupervised algorithm would work the best. Any and all help is greatly appreciated!

Ross Leavitt


texts: Model that predicts probability of correctness of another model

texts:
Problem:
Given a neural network for image classification with $1000$ classes, the objective is to create another model which will output the probability of the neural network giving the correct prediction for a specific input image.
Thoughts:
My ideas so far have been:

Creating a convolutional network and training it with raw images together with a label which indicates if the first NN predicted it correct or not.
Creating a fully connected network and training it with outputs of hidden layers/features of the first NN instead of the raw image, together with the labels as before.
Creating a fully connected network and training it with the top-k outputs of the softmax layer of the first NN together with the labels.

The first method yielded an accuracy of $0.51$, the second $0.58$ and the last one $0.79$.
Can you suggest another method (or a modification of one from the above) which can achieve greater accuracy?


texts: How to implement multi class classifier for a set of sentences?

texts:
I'm currently involved in an aspect level sentiment analysis project and using Stanford CoreNLP toolkit to implement the system. My knowledge on these concepts is very limited and I'm seeking your help to clarify some things related to both machine learning and classification.

I have a set of sentences, which are travel reviews that needs to be categorized under different labels (AMBIANCE, SCENERY, COST, ENTERTAINMENT etc). This is done by checking if certain aspect terms related to the aforementioned categories are found in the sentences.

Now, I want to train a classifier (I'm using a Stanford classifier), to classify these sentences to respective categories and I have a train data set containing around 3000+ sentences.

My problem is, a sentence may contain aspect terms that belong to not just one, but multiple aspect categories. In that case, I want the classifier to classify them all at once.

For example:

Review Sentence:

  On the upside it was very **calm** there and good for **swimming**

Categories

  AMBIANCE, ENTERTAINMENT

I have tried to pre-process the training dataset in this manner, and trained a classifier. But when I tried to make it classify a sentence containing aspect terms belonging to two different categories, it only identified one category.

Training dataset was of the following format

sentence1    [tab]   category1
sentence2    [tab]   category1, category2
sentence3    [tab]   category2, category3

Can someone please tell me if my approach is wrong? How can I achieve the desired output? I would greatly appreciate any help on this matter, as I'm currently stuck at this phase of my project.


texts:
I want to applicate the randomForest to my data for predicting target variable, but I have got a warnings message saying:

Warning message:
In randomForest.default(m, y, ...) :
  The response has five or fewer unique values.  Are you sure you want to do regression?

I didn't know what was going on?

this is my code:

library(randomForest)
M=data.frame(Type_peau,PEAU_CORPS,SENSIBILITE,IMPERFECTIONS,BRILLANCE ,GRAIN_PEAU,RIDES_VISAGE,ALLERGIES,MAINS,
             INTERET_ALIM_NATURELLE,INTERET_ORIGINE_GEO,INTERET_VACANCES,INTERET_COMPOSITION,DataQuest1,Priorite2,
             Priorite1,DataQuest4,Age,Nbre_gift,w,Achat_client)

factor_vars <- c("Type_peau","PEAU_CORPS","SENSIBILITE","IMPERFECTIONS","BRILLANCE","GRAIN_PEAU",
                 "RIDES_VISAGE","ALLERGIES","MAINS","INTERET_ALIM_NATURELLE","INTERET_ORIGINE_GEO",
                 "INTERET_VACANCES","INTERET_COMPOSITION","DataQuest1","Priorite2","Priorite1","DataQuest4")

head(M)
str(M)

sample.ind <- sample(2, nrow(M), replace = T,prob = c(0.6,0.4))
cross.sell.dev <- M[sample.ind==1,]
cross.sell.val <- M[sample.ind==2,]

table(cross.sell.dev$Achat_client)/nrow(cross.sell.dev)

table(cross.sell.val$Achat_client)/nrow(cross.sell.val)

varNames <- names(M)
# Exclude ID or Response variable
varNames <- varNames[!varNames %in% c("Achat_client")]

# add + sign between exploratory variables
varNames1 <- paste(varNames, collapse = "+")

# Add response variable and convert to a formula object
rf.form <- as.formula(paste("Achat_client", varNames1, sep = " ~ "))

test$Variable1 <- factor(test$Variable1,levels=levels(train$Varialbe1))

# Buiding
cross.sell.rf <- randomForest(rf.form,cross.sell.dev,ntree=500,importance=T)
plot(cross.sell.rf)

Thanks for your help!


texts:
I want to make a 3D scatter plot of multiple data selections on a single plot (i.e same axes).
I know that in 2D this is possible by using par() function like so:

plot(6:25,rnorm(20),type="b",xlim=c(1,30),ylim=c(-2.5,2.5),col=2)
par(new=T)
plot(rnorm(30),type="b",axes=F,col=3)
par(new=F)

(source: http://cran.r-project.org/doc/contrib/Lemon-kickstart/kr_addat.html)
Can I do something like that on a 3D plot, preferably an interactive 3D plot, like the ones created using plot3D from 'rgl' package?


texts: Searching for a 3D Dataset, segmented by 2 or more Experts

texts:
We are looking for data sets with 3D images, preferably from the medical field. It is important that they have been segmented by more than one person/expert. An example of this is the BraTS Challenge dataset (https://www.med.upenn.edu/sbia/brats2018/data.html) or the LIDC IDRI Dataset (https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI), but we need more datasets for our research.

Many thanks in advance!


texts: How to deal with millions or rows of data for analysis/visualization purpose

texts:
I have data in 2 tables in Sql server.
First table has around 10 million rows and 8 columns.
Second table has 6 million rows and 60 columns.

I want to import those tables into a Python notebook using pandas ( I am importing in "chunksize")  and then merge them and then run analysis on resulting table.

I am unable to import the data due to possible hardware constraints.
System hardware configuration is as follows:

RAM: 4GB
Storage: 160 GB,
CPU : Dual core CPU.

Even if the import goes through (which seems difficult), my resulting table after the merge will have 5 million rows and 40 columns.

Is it feasible to perform analysis/visualization in python notebook using pandas, seaborn on columns on the resulting table?

Would love to understand:

What should be my next steps to solve the problem?

Is installation of Anaconda on Windows 2012 server feasible? (This server has more memory)


texts: Work experience in Data Science: Couple hours with A-level student

texts:
Our company has accepted an A-level student (18-year old, just before uni) for some work experience. The student will be with me for a couple of hours. I do data analysis and I'm quite busy at the moment.
Not sure what I can show him in such a short time, and I would rather do it hands-on.

Any suggestion? Some online resources?


texts: Evaluating performance of Generative Adverserial Network?

texts: What is the best way to evaluate performance of Generative Adverserial Network (GAN)? Perhaps measuring the distance between two distributions or maybe something else?


texts: Is there any problem with the following Python+TF+Keras code for a custom loss function and network?

texts:
I am trying to code a custom loss function for variational autoencoder. I am not using mse for reconstruction loss since I am not learning p(x|z) ~ N(mu,I). Instead of that, I want to learn the sigma as well. So p(x|z)~N(mu,sigma**2I). If I consider this, then my reconstruction loss function part changes to include additional items: input_dim*ln(2*pi) and (x-mu)**2/(sigma)**2  along with the KL divergence term. (x-mu)**2/(sigma)**2 is equivalent to 0.5*input_dim*mse(input,output)/sigma**2.
My question is, with this modification, is my code valid.
def loss(inputs,outputs):
    # Note log_var = ln(sigma**2)

    from tensorflow.keras import backend as K

    log_var = K.Variable(0.0)

    reconn_loss = 0.5*input_dim*(mse(inputs,outputs)/K.exp(log_var)+log_var+ np.log(2*np.pi))

    #kl loss
    kl_loss = -0.5*K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)

    vae_loss = K.sum(reconn_loss + kl_loss)

    return vae_loss

Now the actual encoder-decoder function. Note, since I want to output mu, and sigma from decoder as well, it's architecture is similar to the encoder's.
def VAEmodel(params):
    input_dim = params['input_dim']
    latent_dim = 2
    intermediate_dim = 32
    intermediate_dim_2  = 16 # extra added Delete later

    # encoder
    inputs = Input(shape=(input_dim,),name='encoder_input')
    x = Dense(intermediate_dim, activation='relu')(inputs)
    x = Dense(intermediate_dim_2, activation='relu')(x)
    z_mean = Dense(latent_dim, name='z_mean')(x)
    z_log_var = Dense(latent_dim, name='z_log_var')(x)

    # reparametrization trick to sample latent code (z) from Q(z|x)
    z = Lambda(sampling, output_shape=(latent_dim,), name='z')([z_mean, z_log_var])

    # instantiate encoder model
    encoder = Model(inputs, [z_mean, z_log_var, z], name='encoder')
    #encoder.summary()

    # decoder
    latent_inputs = Input(shape=(latent_dim,), name='z_sampling')
    x = Dense(intermediate_dim_2, activation='relu')(latent_inputs) # Extra layer added,Delete later
    x = Dense(intermediate_dim, activation='relu')(x)
    outputs_mean = Dense(input_dim, activation='linear',name='output_mean')(x)
    outputs_var = Dense(input_dim, activation='linear',name='output_var')(x)
    # instantiate decoder model
    decoder = Model(latent_inputs, [outputs_mean,outputs_var], name='decoder')
    #decoder.summary()

    # instantiate vae model
    outputs = decoder(encoder(inputs)[2])
    vae = Model(inputs, outputs, name='vae_mlp')

    #get the loss function
    vaeloss = loss(inputs,outputs)
    #compile
    vae.compile(optimizer= 'adam',loss=vaeloss)

    return vae,encoder,decoder,outputs

Note, I can get the mu, and sigma from outputs[0], and outputs[1] respectively.
Is this set up going to work the way I want i,e learn sigma of p(x|z)?
Thank you very much. Your inputs will be greatly appreciated.


texts: What branch of data science to use for revealing preceding conditions of desired outcome

texts:
On financial data series, I would like to determine if some conditions precede a desired outcome.
For example, let's take as an example the daily data of a stock. On some days, the stock experience big trends.
I would like to know if conditions on preceding days, such as volume, day ranges, price moving average... plus many others occur with any statistical significance to have a predictive value.
So I have an outcome (large trending days), a list of preceding conditions (can be a big list), and would like to know if some relation exists.
Ideally I want to specify as much preceding conditions as possible,  and have the system discover statistically significant correlations with the occurence of the outcome.
What branch/sub-branch of data science should I explore? A branch of statistics? Supervised Machine Learning? Unsupervised?


texts: How to run neural network in gpu instead of cpu?

texts: I installed cuDNN and cuda toolkit 11 in my machine. Now I installed tensorflow-gpu in pip3 of version 2.2.0. But while running neural network, cpu is being used 90% with four cores and gpu is being used just 7%-15%. But I want my cpu will not be used.


texts: How to get mean of column using groupby() and another condition

texts:
For the following

df = pd.DataFrame()
df['1'] = 1,2,1,2,1,2
df['2'] = 3,6,5,4,7,8
df['3'] = 1,1,1,2,2,2

I want to do

mean(df.groupby().loc[df['1']==df['3'],'2'].mean()

which doesn't work. And simply doing this :

a=df.groupby(['1','3'])['2'].mean()

gives

OUTPUT:
1  3
1  1    4
   2    7
2  1    6
   2    6

But I only want cases where column 1 and 3 have the same elements:

1  3
1  1    4
2  2    6


texts: How can I know how to interpret the output coefficients (`coefs_`) from the model sklearn.svm.LinearSVC()?

texts:
I'm following Introduction to Machine Learning with Python: A Guide for Data Scientists by Andreas C. Müller and Sarah Guido, and in Chapter 2 a demonstration of applying LinearSVC() is given.  The result of classifying three blobs is shown in this screenshot:

The three blobs are obviously correctly classified, as depicted by the colored output.

My question is how are we supposed to know how to interpret the model fit output in order to draw the three lines?  The output parameters are given by

print(LinearSVC().fit(X,y).coef_)
[[-0.17492286  0.23139933]
 [ 0.47621448 -0.06937432]
 [-0.18914355 -0.20399596]]

print(LinearSVC().fit(X,y).intercept_)
[-1.07745571  0.13140557 -0.08604799]

And the authors walk us through how to draw the lines:

from sklearn.svm import LinearSVC
linear_svm = LinearSVC().fit(X,y)
...
line = np.linspace(-15, 15)
for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):
    plt.plot(line, -(line * coef[0] + intercept) / coef[1])  #HOW DO WE KNOW
plt.ylim(-10, 15)
plt.xlim(-10, 8)
plt.show()

The line of code with the comment is the one that converts our coefficients into a slope/intercept pair for the line:

y = -(coef_0 / coef_1) x - intercept/coef_1

where the term in front of x is the slope and -intercept/coef_1 is the intercept.  In the documentation on LinearSVC, the coef_ and intercept_ are just called "attributes" but don't point to any indicator that coef_0 is the slope and coef_1 is the negative of some overall scaling.

How can I look up the interpretation of the output coefficients of this model and others similar to it in Scikit-learn without relying on examples in books and StackOverflow?


texts: Machine Learning : Choosing number of nodes in hidden layer when number of input nodes and output nodes are the same

texts:
I'm trying to implement a multi-class classifier using ANN. I have 4 input and 4 output variable (4 classes). I've read various guidelines on choosing the number of nodes in the hidden layer and I was wondering what the best strategy is for my architecture.
I've read that the number of hidden nodes should be between the number of input nodes and output nodes, that would give me a 4-4-4 architecture. I've also seen some people use the mean of the number of output and input nodes which results in the same architecture which I'm not sure about.