texts: How can I obtain datasets about human poses?

texts:
Lots of machine learning tutorials always mention about the identification of hand written numbers or cats or dogs, and there is always a prebuilt dataset for training. But no one said that how to build that dataset.

Now we want to build our own model about human poses identification. We know how to train, how to inference, and how to integrate it to our project sources. But the most difficult task we seem is how to obtain LOTS human pose images and mark them with correct tags to train.

Would someone kindly show us a way to achieve that? Thanks in advance.


texts: Can an entire data frame be used as a prediction variable?

texts:
I am attempting to use XGBoost in R to train a model that predicts a fixed number of target variables using all data from previous dates, as well as the two categorical variables (Cat1 and Cat2) for the current date as predictors. The original data is in this format:
╔═════════╦═════════╦══════════╦══════╦══════╦══════╦══════╗
║ Target1 ║ Target2 ║ Date     ║ Cat1 ║ Cat2 ║ Var1 ║ Var2 ║
╠═════════╬═════════╬══════════╬══════╬══════╬══════╬══════╣
║ 1       ║ 2       ║ 01/01/20 ║ A    ║ B    ║ 3    ║ 4    ║
║ 5       ║ 6       ║ 02/01/20 ║ C    ║ D    ║ 7    ║ 8    ║
║ 8       ║ 7       ║ 03/01/20 ║ A    ║ D    ║ 6    ║ 5    ║
║ 4       ║ 3       ║ 04/01/20 ║ C    ║ B    ║ 2    ║ 1    ║
║         ║         ║          ║      ║      ║      ║      ║
╚═════════╩═════════╩══════════╩══════╩══════╩══════╩══════╝

And I conceptualise the training data looking like this for each row of the data frame, where the Train_DataFrame column contains all data from previous dates:
╔═════════╦═════════╦══════╦══════╦═════════════════╦═════════╦══════╦══════╦══════╦══════╗
║ Target1 ║ Target2 ║ Cat1 ║ Cat2 ║ Train_DataFrame ║         ║      ║      ║      ║      ║
╠═════════╬═════════╬══════╬══════╬═════════════════╬═════════╬══════╬══════╬══════╬══════╣
║       4 ║       3 ║ C    ║ B    ║ Target1         ║ Target2 ║ Cat1 ║ Cat2 ║ Var1 ║ Var2 ║
║         ║         ║      ║      ║ 1               ║ 2       ║ A    ║ B    ║ 3    ║ 4    ║
║         ║         ║      ║      ║ 5               ║ 6       ║ C    ║ D    ║ 7    ║ 8    ║
║         ║         ║      ║      ║ 8               ║ 7       ║ A    ║ D    ║ 6    ║ 5    ║
║         ║         ║      ║      ║                 ║         ║      ║      ║      ║      ║
╚═════════╩═════════╩══════╩══════╩═════════════════╩═════════╩══════╩══════╩══════╩══════╝

First question; is it possible to pass an entire data frame as a variable to a model?
If so, then how can this be done in a memory efficient manner, so as to avoid data duplication? I know I could have the Train_DataFrame column contain lists of each "past" data, however, this would lead to data duplication and inefficient memory usage. Is there a way to have this column contain a sub-setting function to pass to the original data frame, for example?
Or is there a better approach to the problem that I am potentially missing?
Thanks in advance.


texts:
I'm on a NLP project and found a resource that has the three word representations mentioned in the question name, and I am struggling to find one place where they all are explained and compared. As I understand, word2vec is implemented by training a FFNN to recognize, what words will be near the one that's inputed, and GloVe is some "other" way of making word2vec, but which of these algorithms is the best at which task?
My specific task in the project is to find words that are the closest to the input word contextually.


texts: Are 'error value' and accuracy equivalent or can be used interchangeably in machine learning?

texts:
I started studying machine learning.
I have a simple question about the definition of the error value and the accuracy. I know the meanings of two. Error value means simply the difference between the actual label and the predicted value.
Error value = Expected - Predicted

Accuracy is
Accuracy = (TP+TN)/(TP+FP+FN+TN)

I know they are not exactly the same but found they are used together frequently. So, I am wondering whether if I can use them interchangeably.
For example,
train a model to decrease error value = train a model to increase accuracy ???


texts:
I'm trying to plot two different size matrices using one graph (in R), but can't manage to do so.
I tried using matplot and the regular plot, but it didn't work.

Does anyone know how to plot it?


texts: What if there is huge difference between train data and test data?

texts:
I trained a model which does well on unseen data,but after deployed on production the data I got is very different,like the highest values in  train & test data is ~23,but the data I got from production have values which is ~80,and there is just 1 feature and it is regression problem.
Can anyone please recommend what can I do to tackle this problems.


texts: How to estimate real distance between two detected objects in an image?

texts:
You may think this is a duplicate, but my situation is different than previously asked questions. The only information I have is the width and height of the bounding boxes of detected people. The dataset I'm working on has images captured in different environments (street, garden, mall, ...). In other words, there is no fixed object in all images I can use as scale. The angle at which each image is captured varies drastically from almost parallel to the ground to prependicular to the ground. This means that bounding boxes don't always mimic the real size of a person.
My objective is to measure the real distance between people. Any hints on how I can do this ? I am starting to think it is not feasible
Thank you!


texts: Is GridSearchCV in combination with ImageDataGenerator possible and recommendable?

texts:
I want to optimize some hyperparameters for a CNN architecture by using GridSearchCV (Scikit-Learn) in combination with Data Augmentation (ImageDataGenerator from Keras).
However, GridSearchCV only offers the fit function and not the fit_generator function.
Is it even recommended to use data augmentation with GridSearchCV?
The parameters for the ImageDataGenerator are already fixed and should not be changed. Would it be better to first determine the hyperparameters via grid search without data augmentation and only to use data augmentation for the final model?

What do you think about this topic? What are your experiences?


texts:
I need to use a metaheuristic algorithm to solve an optimization problem on a Python codebase.

Metaheuristics usually need to be written in C++ or Java as they involve a lot of iterations, while Python is weak from this point of view.

Questions:

do any Python metaheuristic packages which wrap faster languages as
C++/Java exist?
do any Python metaheuristic packages based on maybe
cython on numba exist?
other solutions?


texts: deep learning - how to train silhouette image?

texts:
I want to train silhouette images to figure out the original image

I'm planning to use Tensorflow.

Once I've thought about using Automatic colorization, but I have no idea if it works well since what it only has is just an outline of the image.

Is there any way to achieve this?

thanks


texts: Can anyone suggest a blog where Variational Autoencoder has been used for time series forecasting?


texts:
I want to do a very simple cross validation using LogisticRegression.
Here is my code:
logreg = LogisticRegression(labelCol = "churn", featuresCol = "features")

pipeline = Pipeline(stages = [logreg])
paramGrid = ParamGridBuilder().addGrid(logreg.regParam, [.1, .01]).build()

crossval = CrossValidator(
    estimator = pipeline,
    estimatorParamMaps = paramGrid,
    evaluator = BinaryClassificationEvaluator(),
    numFolds = 2)

bestLogReg = crossval.fit(df_train)

When I run this, I get the following error on bestLogReg = crossval.fit(df_train):
IllegalArgumentException: label does not exist. Available: features, churn, CrossValidator_764038c00edc_rand, rawPrediction, probability, prediction
Here is my df_train dataset's schema:
root
 |-- features: vector (nullable = true)
 |-- churn: integer (nullable = true)

I have fit this to a LogisticRegression before and it predicts fine.
Can you help me figure out what I did wrong?


texts: How do I obtain the weight and variance of a k-means cluster?

texts:
I am trying to reproduce the results of this paper, but using python and the HMMlearn library instead of matlab. The paper describes a procedure for using HMM (Hidden Markov Model) in order to predict stock prices.

The paper details the use of a 4-state, 5-mixture Gaussian distribution as the model. The transition probabilities and the initial state probabilities are uniform, however the emission probabilities are determined based on the results of a k-means algorithm using the data-set of existing stock prices.

This latter part is where I have gotten stuck, the paper advises to use the means, variance and weight of each cluster returned from the k-means algorithm as the mean, variance and weight of each component of the mixture. As I understand it the mean of the cluster is simply the center of each centroid, however I'm not sure how you would obtain the variance or the weight.

TL;DR
Given a 3 dimensional dataset X (in the form [[a, b, c], [d, e, f]...]) and using the k-means algorithm where k = 5 (k = number of mixture components), how would I determine the weight and variance of each cluster?


texts: I am trying to obtain raw data for (violent) crime rates of a US/Canadian city (any city would do), but I need the data to be granular and raw. All I could find is either interpretations, summary data or useless editorials. I'm trying to do analysis, and I need day-by-day (detailed-level, granular) data that shows the number of crimes recorded per day. Does anyone have any good sources/suggestions for finding this? Thank you immensely!


texts: adding a supervising process during knn process

texts:
I am trying to improve my KNN regression process (I would like to use sklearn / python, but it doesn't matter).I would like to improve my results and to gain insight. Here is an example:
I have data measured from an electric motor: an input voltage (U) and current (I) and an output torque (T) and speed (S).
First intend is a simple approach where I'm giving those data in the state to a KNN algorithm and I use the results. But every result is not logical (even if they are statistically close).
if you add a knowledge layer during the process, a "human in the loop approach", it must be better! For example here, you know that the input power is Pin = UI and the output power is Pout = TS. The efficiency of the system eff = Pout / Pin and the efficiency cannot be higher than 1. Whereas KNN results can generate results with eff > 1.
My question is, how to use this knowledge (additional condition/human in the loop approach) during my KNN learning process phase to improve my results? Do I continue to learn on the initial data or transformed data? Do I modify the learning process? Is it possible to add "supervisor condition" that are influencing the learning process?
Thank you for your help!


texts: Heterodox use of Deep Learning to find hidden patterns

texts:
I would appreciate your comments/help about a strategy I am applying in one of my analysis. In short, my case is:

1) My data have biological origin, collected in a period of 120s, from a
 subject receiving, each time, one of possible three stimuli (response label 1
 to 3), in a random manner, one stimulus per second (trial). Sampling
 frequency is 256 Hz and 61 different sensors (input variables). So, my
 dataset has 120x256 rows and 62 columns (1 response label + 61 input
 variables);

2) My goal is to identify if there is an underlying pattern for each stimulus.
 For that, I would like to use deep learning neural networks to test my
 hypothesis, but not in a conventional way (to predict the stimulus from a
 single observation/row).

3) My approach is to divide the whole dataset, after shuffling per row
 (avoiding any time bias), in training and validation sets (50/50) and then to
 run the deep learning algorithm. The division does not segregate trial events
 (120), so each training/validation sets should contain data (rows) from the
 same trial (but never the same row). If there is a dominant pattern per
 stimulus, the validation confusion matrix error should be low. If there is a
 dominant pattern per trial, the validation confusion matrix error should be
 high. So, the validation confusion matrix error is my indicator of the
 presence of a hidden pattern per stimulus;

I would appreciate any input you could provide me regarding the validity of my logic. I would like to emphasize that I am not trying to predict the stimulus based on row inputs.

Thanks.


texts: External cluster evaluation for a varying number of cluster

texts:
There are many external clustering indices like (Adjusted) mutual information, (Adjusted) Rand index, and many more. However, they are not very good at comparing clusterings where the number of clusters is not the same.

For example, if I take one cluster and split it into two new ones, a comparison would yield a low score.

To evaluate the clustering in my case, this operation should not affect the score a lot. Also according to [1], none of the compared cluster evaluation indices utilize optimal pairing, only greedy methods, to find the cluster matches (solve the linear assignment problem).

So my question is: Are there any methods to calculate a matching score between two clusterings where the number of clusters differ and clusters are not only paired but also optimally matched in a 1:n (or even n:m) way? The computation complexity doesn't matter to me very much.

Using an example with NMI:

>>> normalized_mutual_info_score([0, 0, 1, 1, 2, 2, 2, 2], [1, 1, 2, 2, 0, 0, 3, 3], average_method='arithmetic')
0.8571428571428573
>>> normalized_mutual_info_score([0, 0, 1, 1, 2, 2, 2, 2, 2, 2], [1, 1, 2, 2, 0, 0, 0, 3, 3, 3], average_method='arithmetic')
0.8204614780379502
>>> normalized_mutual_info_score([0, 0, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2], [1, 1, 2, 2, 0, 0, 0, 0, 3, 3, 3, 3], average_method='arithmetic')
0.7896900821428475

The scores get progressively worse for an increasingly larger split cluster.

[1]  Mohammad Rezaei, Pasi Fränti. Set Matching Measures for External Cluster Validity.


texts:
I'm newbie in R and time series analysis. I'm trying to build an Arima model.
My dataset has this structure:
DATA         Ora    VALORE
<chr>       <dbl>    <dbl>

2018-09-01    1      3646742
2018-09-01    2      3273110
2018-09-01    3      3069245
2018-09-01    4      2969621

I've converted this dataset in a hourly time series and split it in a training set  and a test set with this code:
y <- ts(data$VALORE,start=c(2018,09, 00:00), frequency=24*365)

y_train <- window(y, c(2018, 09), c(2020, 06))

y_test <- window(y,c(2020, 07), c(2020, 08))

Problems:

test set contains two observations (instead 1488 = all July and all August 2020);

training set contains 17518 observations (instead 16032).

Can I solve these problems?


texts:
After downloading the imagenet urls (link), I see that it is a single 1.1 GB text file which starts like this:

n00004475_6590    http://farm4.static.flickr.com/3175/2737866473_7958dc8760.jpg
n00004475_15899    http://farm4.static.flickr.com/3276/2875184020_9944005d0d.jpg
n00004475_32312    http://farm3.static.flickr.com/2531/4094333885_e8462a8338.jpg
n00004475_35466    http://farm4.static.flickr.com/3289/2809605169_8efe2b8f27.jpg
n00004475_39382    http://2.bp.blogspot.com/_SrRTF97Kbfo/SUqT9y-qTVI/AAAAAAAABmg/saRXhruwS6M/s400/bARADEI.jpg
n00004475_41022    http://fortunaweb.com.ar/wp-content/uploads/2009/10/Caroline-Atkinson-FMI.jpg
n00004475_42770    http://farm4.static.flickr.com/3488/4051378654_238ca94313.jpg
n00004475_54295    http://farm4.static.flickr.com/3368/3198142470_6eb0be5f32.jpg
n00005787_13    http://www.powercai.net/Photo/UploadPhotos/200503/20050307172201492.jpg
n00005787_32    http://www.web07.cn/uploads/Photo/c101122/12Z3Y54RZ-22027.jpg

I'm pretty sure n00004475_6590 is the class of the image. How can I look up what this means (in natural language)?


texts: Using RNN (LSTM) for Gesture Recognition System

texts:
I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it detects the sequence and maps it to it's corresponding class (sleep, help, eat, run, etc.)

The thing is I've already built a similar system but for static images (no motion included), it was useful for translating alphabets only in which building a CNN was a straight forward task, as the hand doesn't move so much and the data set structure was also manageable as I was using keras and maybe still intending to do so (every folder contained a set of images for a particular sign and the name of the folder is the class name of this sign ex: A, B, C,..)

My question here, how can I organize my data set to be able to input it into a RNN in keras and what certain functions should I use to effectively train my model and any necessary parameters, some people suggested using TimeDistributed class but I don't have a clear idea on how to use it for my favor, and take in account the input shape of every layer in the network.

also considering that my data set would consist of images, I'll probably need a convolutional layer, how would it be feasible to combine the conv layer into the LSTM one (I mean in terms of code).

For example I imagine my data set to be something like this

Folder named 'Run' contains 3 folders 1, 2 and 3, each folder corresponds to it's frame in the sequence

So Run_1 will contain some set of images for the first frame, Run_2 for the second frame and Run_3 for the third, my model's objective is to be trained with this sequence to output the word Run.


texts: How to get a dataframe values in one single column for the following dataset?

texts:
m38     m78        alpha

4.4717  4.8745
4.4569  4.6491
4.5101  4.7262
4.4407  4.8234
4.1184  4.3862
3.8448  4.2816
3.7246  4.6183
3.2857  4.6744

For the above sample data (actually 8000 rows) i need to calculate alpha(column), using the log function.

In: alpha = (np.log([dataset.m38,dataset.m78])/np.math.log(38,78.7))

OUT: array([[1.79754463, 1.90105609],
       [1.7935659 , 1.8442364 ],
       [1.80780671, 1.86397626],
       ...,
       [2.06431358, 2.40416332],
       [2.08820691, 2.41635699],
       [2.09982107, 2.39551918]])

I converted this into dataframe by using the below code:

In: alpha = pd.DataFrame(data = alpha)

OUT: 2 rows × 8046 columns

Now, i used alpha.T to get this in 8046 rows and 2 columns.

Why i am getting 2 columns not one? and how should i convert it into one column?


texts:
I am a beginner in ML, and I am working on a classification problem on big data (its shape is (8921483, 52)) which its features are mostly categorical. One of the features has 175365 different categories. I encoded it via the frequency encoding method and reduced the categories to 2081. However, the value of the encoded categories mostly is very small numbers (s.t. 1.120890e-07).
Can I normalize the encoded values (after frequency encoding)?


texts: Training the network with some batch size - code

texts:
There is my "training" code below, I wrote it based on one youtube tutorial. I don't understand actually one part:  batch_X = train_X[i:i+BATCH_SIZE], batch_y = train_y[i:i+BATCH_SIZE]. How to understand this thing inside the brackets (i:i+BATCH_SIZE)?
def train(net):
            BATCH_SIZE = 32
            EPOCHS = 10
            for epoch in range(EPOCHS):
                # training loop
                net.train()
                for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
                    batch_X = train_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = train_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                net.zero_grad()
                outputs1 = net(batch_X)
                matches1 = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs1, batch_y)]
                acc = matches1.count(True)/len(matches1)
                loss = loss_function(outputs1, batch_y)
                loss.backward()
                optimizer.step()

            print(f"Epoch: {epoch}. Training acc: {round(float(acc),2)} Training loss: {round(float(loss),3)}")


texts: How to show two pictures in one cell in Jupyter Notebook? (matplotlib) (python)

texts:
As a beginner, can someone please help me with this? Is there any way to show both pictures in a single cell's output? My output could only display one picture. Thank you for your time and attention!
Here is my code:
from skimage import data
image_coffee = data.coffee()
image_horse = data.horse()
fig = plt.figure(dpi=100)
plt.imshow(image_coffee)
plt.imshow(image_horse)


texts:
I am just a young learner in Data Science. Could you help me to resolve this exercise of K-NN?

Ex.1

The table below provides a training data set containing six observations, three predictors, and one qualitative response variable:

x1  x2  x3  y
0   3   0  Red
2   0   0  Red
0   1   3  Red
0   1   2  Green
−1  0   1  Green
1   1   1  Red

Suppose we wish to use this dataset to make a prediction for Y when x1 = x2 = x3 = 0 using
K-nearest neighbors.

1) With the euclidean distance, what is the prediction with K = 1 and with K = 3 for the test
point (0, 0, 0)?

2) If the Bayes decision boundary in this problem is highly non-linear, then would we expect the
best value for K to be large or small? Why?


texts:
I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed?

I mean I see no mapreduce operations. Or maybe, when submitted in Spark, the framework actually makes the needed tricks under the hood to distribute the algorithm?

I also found that Spark shows mapreduce the exit and I am using Spark 1.6.

EDIT: This code produces a runtime error, check here.


texts: Is it possible to have a default class in multi class classification?

texts:
In the general text classification problem, training a machine learning model to detect if a text belongs to one of N number of classes always yields a value in N. Even if the text that was passed to the model is very far outside any of the N classes (like say you trained it on categories for movie genres, but someone passed a recipe to the classifier), it will always choose one of N as the output even though the text passed to it doesn't belong to any of the classes.

Is there a method to have an additional class for 'unknown' so that if the model output is one of the N types but matches it with very low probability then assign it to the default or unknown class?

EDIT: we are using LinearSVC from sci kit learn


texts: How to determine whether a semantic concept is present in a string

texts:
I need to find a way to detect if a large string contains a specific substring.
Imagine that I have a full contract page converted to string in my Python program. What I want to do is to say if a specific term (a smaller string than the whole page string) exists in the page converted to string. The matching must be done semantically.
As an example, in the following case, I expect to have a good score for Query 1 and a bad score for Query 2. Notice that it's possible that my query has more than one sentence.
Query 1
"The supplier will accomplish the fulfill the delivery of the products in one week"
Query 2
"I like Phil Colins"
Page text
"The supplier will be paid annually in jully.
The delivery of products will be made in one week.
.........
"
How would you do this task?


texts: Scikit Learn's RandomForestRegressor is not giving results on large data set

texts:
I have a Pandas dataframe X(20346, 4116). All independent columns have binary variable as 0 or 1. Whereas dependent column has continuous variable. When I execute below code using the scikit-learn library, it takes for ever and I am not getting any result.

X_train, X_test, y_train, y_test = train_test_split(X13, y, train_size=0.8)

rf = RandomForestRegressor(n_estimators=100,  max_features='auto', oob_score=True,
                            n_jobs=1, verbose=0)
rf.fit(X_train, y_train)

I am using this for feature selection. In case of less number of rows I can get result. But when number of rows go up then this problem starts happening. Do you have any idea how to make it work?


texts: Clustering of news combining headline and main article

texts:
I want to classify German police news articles and do an automated classification/clustering with regards to the kind of crime committed. Thus far I am not getting great results. Often times the headline is pretty telling but sometimes it is not that revealing, so I need to include the main article to do a proper clustering on all articles.

However the headline is a very good indicator and I don't want to throw it away. I could simply combine the words from headline and main article but I think there should be more weight given to the headline. Has somebody an idea how to do this?


texts: PCA scikit-learn - ValueError: array must not contain infs or NaNs

texts:
I use PCA from from sklearn.decomposition to reduce data dimension.

    pca = PCA(n_components=pca_components, whiten=True, svd_solver='full')
    y = pca.fit_transform(x)

The numpy array shape is (512, 48), dtype is float64. The minimum value in the array is 0.0, the maximum value is 1.7976931348623157e+308. The array does not contain infs or NaNs but I get an error

ValueError: array must not contain infs or NaNs

I have looke for reasons of this error and found nothing useful for me.
I suppose that maybe the maxim value in the array is too large for PCA. But I still do not know how deal with it.
Could you advice me something. Thanks alot in advance.


texts: What exactly is a trained model for machine-learning?

texts:
I am curious about the deployment phase of a machine-learning model. So, after you run your script using python to train your millions of data and it works, what are some methods of deploying your model to the real world? I know libraries such as pickle which is being used to serialize a model exists, but what exactly is a model?

Is it some sort of blueprint which is then used to craft the predicted answer? If so, what does that mean that having a pickled object in the real world is much faster at churning predictions than re-training your model again?


texts: Regarding the number of iterations functionality

texts:
I have a query regarding the number of iterations functionality in Orange, which is available with several widgets.

For instance, in the neural networks widget, the maximum number of iterations is currently 10000. However, I often come across problems where it takes more than 10000 iterations for a problem to converge.

Is there a way to increase the number of iterations to for instance 1000000?


texts: Train a deep learning model in chunks/sequentially to avoid memory error

texts:
How do I train/fit a model in chunks so as to escape the dreaded memory error?

def TFIDF(X_train, X_test, MAX_NB_WORDS=75000):
    vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)
    X_train = vectorizer_x.fit_transform(X_train).toarray()
    X_test = vectorizer_x.transform(X_test).toarray()
    print("tf-idf with", str(np.array(X_train).shape[1]), "features")
    return (X_train, X_test)

# In[3]:

def Build_Model_DNN_Text(shape, nClasses, dropout=0.5):
    """
    buildModel_DNN_Tex(shape, nClasses,dropout)
    Build Deep neural networks Model for text classification
    Shape is input feature space
    nClasses is number of classes
    """
    model = Sequential()
    node = 512  # number of nodes
    nLayers = 4  # number of  hidden layer
    model.add(Dense(node, input_dim=shape, activation='relu'))
    model.add(Dropout(dropout))
    for i in range(0, nLayers):
        model.add(Dense(node, input_dim=node, activation='relu'))
        model.add(Dropout(dropout))
    model.add(Dense(nClasses, activation='softmax'))
    model.compile(loss='sparse_categorical_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])
    return model

# In[18]:

df = pd.read_csv("ExtractedData.csv")

# In[19]:

df = df.dropna()
X_train, X_test, y_train, y_test = train_test_split(df['body'], df['user_id'],
                                                    test_size=0.3, random_state=42, shuffle=True)
X_train_tfidf, X_test_tfidf = TFIDF(X_train, X_test)
y_train_tfidf, y_test_tfidf = TFIDF(y_train,y_test)
output_nodes = len(list(set(df['user_id'])))
model_DNN = Build_Model_DNN_Text(X_train_tfidf.shape[1], output_nodes)
model_DNN.fit(X_train_tfidf, y_train_tfidf,
              validation_data=(X_test_tfidf, y_test_tfidf),
              epochs=10,
              batch_size=256,
              verbose=2)

predicted = model_DNN.predict(X_test_tfidf)
print(metrics.classification_report(y_test, predicted))

When I run the above, I keep getting the memory error: Unable to allocate 37.9 GiB for an array with shape (67912, 75000) and data type float64. I know I can send the data in chunks pd.read_csv( , chunksize= 5000), but what I don't know is how do I implement it here. My data has 98000 rows and 4 columns. Thank you.


texts:
I am using an udemy course for MachineLearning and I am trying to form a dummy for my variable the column is Country
I want to change to   France   Germany   Spain
- France                             1        0       0
- Spain                              0        0       1
- Germany                            0        1       0  etc

I tried this but I got this error
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_values = [0])
X = onehotencoder.fit_transform(X).toarray()

Traceback (most recent call last):

  File "<ipython-input-9-f7654080cf6a>", line 4, in <module>
    onehotencoder = OneHotEncoder(categorical_values = [0])

TypeError: __init__() got an unexpected keyword argument 'categorical_values'


texts: Convolution Neural Network with 1 input channel

texts: I am trying to make a sudoku solver using a convolution neural network with pytorch. I am failing to implement the architecture with one channel. There are many great tutorials which deal with images in CNN but I couldn't find any tutorial for CNN with 1 channel input matrix. Any suggestions or help is appreciated.


texts: how to pass parameters over sklearn pipeline's stages?

texts:
I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API.
So I builded a Sklearn Pipeline for that:

def create_model(optimizer="adam", nbr_features=100):
    model = Sequential()
    model.add(Dense(512, activation='relu', input_shape=(nbr_features,)))
    ...
    model.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=["accuracy"])
    return model

estimator = Pipeline([("tfidf", TfidfVectorizer()),
                      ('norm', StandardScaler(with_mean=False)),
                      ("km", KerasClassifier(build_fn=create_model, verbose=1))])
grid_params = {
     'tfidf__max_df': (0.1, 0.25, 0.5, 0.75, 1.0),
     'tfidf__max_features': (100, 500, 1000, 5000,),
      ... }

gs = GridSearchCV(estimator,
                   param_grid,
                   ...)

I want to pass max_features parameters from tfidf stage to km stage as nbr_features. Any Hack/Workaround  to do that ?


texts: Stopwords for programming languages (for, while, print,...)

texts:
I am trying to do some analysis about user behaviour when typing (keystroke biometrics). Ideally, it will include traits extracted when people are writing code. Although not technically Natural Language, code also has some structured characteristics as language and I wanted to leverage that. I was wondering if there has been some research about performing language analysis focusing on programming languages instead of traditional spoken languages.

Mainly, I am interested in having a comprehensive list of stopwords for as many languages as possible. For example, stopwords will include: for, while, return, break, string, if, else, and so on. Although it would be nice to have them separated by languages, I wouldn't mind a list comprising several languages.

I know this could be done for example by getting some sample code and retrieving the most frequent terms, but I also wanted to know if there has been some research towards this direction.

Any ideas, papers, methods would be welcome.

Thanks!


texts:
[Update] Multiple target variables can exist.

If I find that a variable is not a predictor variable, does that mean it automatically becomes a target variable?


texts: some algorithms to identify of a pattern in text data

texts:
I have few sentences like below in my project(around 25000),

sentence1 = 'Must be able to multi-task in a fast-paced, deadline-driven environment'
sentence2 = 'Strong organisational skills and proactive approach'
sentence3 = 'in this role you will design develop and revise application simulations in alignment with product implementation timelines'
sentence4= 'ensures appropriate cross-referencing between documents within and across qms chapters/topics'

From these sentences, I am trying to determine what functionality is being expected from the person or like what person has to do. I need to identify those keywords. I can use any ML techniques or any other method to determine the keywords,

for example in

sentence1 = [multi-task, fast paced, deadline-driven environment]
senetence3 = [design develop and revise application simulations]

I tried to identify those words with help of POS but that wasn't very useful. Are there any algorithms or ideas or methods that can detect such keywords ?


texts: Chi-Squared test: ok for selecting significant features?

texts:
I would have a question on the contingency table and its results.
I was performing this analysis on names starting with symbols as a possible feature, getting the following values:
Label          0.0  1.0
with_symb      1584 241
without_symb     16 14

getting a p-value which lets met conclude that variables are associated (since it is less than 0.05).
My question is if this result might be a good result based on the chi-squared test, so if I can include in the model.
I am selecting individually features to enter the model based on the chi-squared. Maybe there is another way to select the most appropriate and significant features for the model.
Any suggestions on this would be great.


texts: Why YOLO algorithm predicts B boxes for each grid cell S?

texts: In yolo each grid cell predicts multiple bounding boxes lets say in YOLOv1 it predicts B=2, what is the advantage as it only predicts class probabilities only once for each grid cell. If that so why not only use one bounding box...


texts: Anomaly detection without any knowledge about structure

texts:
I have an interesting question, my code needs to be able to handle structured data where I don't know much about the structure at development time. I know the samples follow a schema that can be nested and the leafs contain some basic primitives like floats, integers and categoricals. I will have access to the schema at training time. I want to train a model that can detect anomalies or more directly whether or not a sample comes from the original distribution of data.

One way would be to just flatten the data and train an isolation forest. From my intuition, this breaks with highly correlated features. Another approach would be to build a neural network architecture that contains some of the structure and primitive types of the passed schema. You could build an autoencoder approach and measure reconstruction error, which I feel like would handle correlated features a lot better because these correlations are captured by the encoder-decoder network. A possible issue would be how to build this reconstruction error metric with different primitives.

Does anyone have any ideas on this or possibly papers that handle related challenges?

EDIT:

I will clarify some things, sorry for being vague. At the time of designing the algorithm I don't know the structure of the dataset involved but at the moment of training the model this structure is known. The fact that the schema could be nested is not so relevant, let's ignore that part and assume we have a flat, tabular dataset where we know the dtypes. It's likely these dtypes are mixed so I'm looking for a way to reason about anomalies on this set.


texts: What is the risk of removing a feature with low correlation...?

texts:
I'm running a linear regression model as baseline for a specific estimation problem. Based on the resulting R-squared, regressor coefficients and their respective p-values, I can conclude that specific independent variables can be removed from the model.

What is the induced risk of removing these variables from the feature set?
Can other models—that are better at modelling nonlinear relationships—suffer from this decision?
How can I be sure that I am not loosing valuable nonlinear information without running "nonlinear" regressors?


texts:
import scipy.stats as st
st.t.ppf(0.95, df=9)

Output: 1.8331129326536335

As per this, I should get 2.262 with dof = 9.


texts:
I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code:

import numpy as np
import pandas as pd

r_cols = ['user_id', 'movie_id', 'rating']

ratings = pd.read_csv('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.data', sep='\t', engine='python', names=r_cols, usecols=range(3))                         # The file is u.data from MovieLens
print(ratings.head())

movieProperties = ratings.groupby('movie_id').agg({'rating': [np.size, np.mean]})
print(movieProperties.head())

movieNumRatings = pd.DataFrame(movieProperties['rating']['size'])
movieNormalizedNumRatings = movieNumRatings.apply(lambda x: (x - np.min(x)) / (np.max(x) - np.min(x)))
print(movieNormalizedNumRatings.head())

movieDict = {}
with open('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.item') as f:                       # The file is u.item from MovieLens
    temp = ''
    for line in f:
        fields = line.rstrip('\n').split('|')
        movieID = int(fields[0])
        name = fields[1]
        genres = fields[5:25]
        genres = map(int, genres)
        movieDict[movieID] = (name, genres,
movieNormalizedNumRatings.loc[movieID].get('size'),           movieProperties.loc[movieID].rating.get('mean'))

print(movieDict[1])

from scipy import spatial
def ComputeDistance(a, b):
    genresA = a[1]
    genresB = b[1]
    genreDistance = spatial.distance.cosine(genresA, genresB)
    popularityA = a[2]
    popularityB = b[2]
    popularityDistance = abs(popularityA - popularityB)
    return genreDistance + popularityDistance        # Everything seems fine up to this point. But the problem seems to be the next line of code.

print(ComputeDistance(movieDict[2], movieDict[4]))

Note: This code can be found here: https://hendra-herviawan.github.io/Movie-Recommendation-based-on-KNN-K-Nearest-Neighbors.html

Note: This code can also be found on pg 245-250, chp7, Hands-On Data Science and Python Machine Learning by Frank Kane. I have an e-copy and I can send it on request.

Note: I don't know how to share the csv files here. I will appreciate it if somebody can tell me how I can do that here.


texts: Training of word weights in Word Embedding and Word2Vec

texts:
I want to know how are the word weights updated for the embedding layer in Keras and for Word2Vec. Like for the normal model.add(Embedding(..)) and  from gensim.models import Word2Vec. Though after using Word2Vec() we put them in the Keras Embedding layer. So I want to know how this is being done mathematically.

I've gone through this post, but I just still want a clear mathematical difference between Word2Vec and normal embedding. I want to know the math, how is gradient descent used for updating the weights of the words, how is backpropagation done etc.

P.S. What I understood is that Word2Vec() converts words into vector space and similar words are represented closer to each other. And in case of Embedding, unique integers are assigned, and represented in vector space. So are the words converted to one-hot vectors, or how are they just being put up as numerical values. I want to get this concept cleared, I haven't been able to find any resources which explain this difference mathematically from scratch.


texts: Policy Gradient methods not converging to useful mean values

texts:
I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my system using Normal distributions over the actions. My neural network generates mu and sigma values for each of the actions and I select new actions by choosing values from the provided distribution.

What I am unclear on is executing the policy when it is not being trained. In other RL approaches when we are not training we would want to pick the greedy solution, so as to generate the action-state path with the highest possible (and hopefully optimal) reward. In discrete action spaces we accomplish with the argmax function. But due to the intractability of performing argmax over a continuous environment I have been implementing my own solution: just using the value of mu for the given actions.

Unfortunately, this solution hasn't been as effective as I anticipated. A closer inspection of my policy shows that in the initial states my policy has  a mean located over actions that do nothing (ie. velocity = 0.0) at the limits of my clipped continuous space. As such when trying to execute my greedy policy the system does atrociously.

However, in these states I possess a very high sigma/std dev. and therefore, when I train my system it quickly, but randomly, propels itself into other states (ones which generate a positive value for mu). And while I can simply select actions according to the probabilistic-greedy approach instead of the pure-greedy approach I feel that this is not a complete solution for my problem. Most notably for the reliance on random actions to reach the goal.

In conclusion, I'd like to know: first, is my approach to a greedy solution appropriate? and second, are there any theoretical approaches I should consider to either limit sigma or encourage mu to move?

Thank you

EDIT: I have managed to have some luck by clipping the sigma value to be between 1e-7 and 0.5. While, the mean has moved the sigma now remains solidly at 0.5 and has not decreased. Are there more standard ways to accomplish the desired implementation?


texts: Difference: Replicator Neural Network vs. Autoencoder

texts: I'm currently studying papers about outlier detection using RNN's (Replicator Neural Networks) and wonder what is the particular difference to Autoencoders? RNN's seem to be treaded for many as the holy grail of outlier/anomaly detection, however the idea seems to be pretty old to, as autoencoders have been there for a long while.


texts: Is there a model that can adapt to additional new training data with different columns?

texts:
My training data comes in batches. Sometimes, new batches (completely new samples) come with new columns that are not in old batches, or they may be missing some of the old columns.

For example, suppose there are two ingestions. In the 1st ingestion, we have ETL on a set of fields. In the 2nd ingestion, we have added a new field and we are not allowed to ingest and update the old records again (they may have been deleted for good).

Ideally, I want to train a classifier using all batches of data. What kind of algorithms would perform well under this scenario.


texts:
I am working on some algorithms for anomaly detection

The dataset is clean our anomalies so I want to add some artificial anomalies.

I have added some anomalies. I get the maximum value of the dataset and add 20-25%, meaning these added anomalies are bigger than the max value by 20 to 25%

Are there any other types of anomalies that would be nice to have had in an anomaly detection algorithm dataset?

My dataset is with integers and float


texts: Is there a way to force a transformer to return a pandas dataframe?

texts:
I am having issues with scikit-learn converting dataframes to numpy arrays.

For instance, the following code

from sklearn.impute import SimpleImputer
import pandas as pd

df = pd.DataFrame(dict(
    x=[1, 2, np.nan],
    y=[2, np.nan, 0]
))

SimpleImputer().fit_transform(df)

Returns

array([[1. , 2. ],
       [2. , 1. ],
       [1.5, 0. ]])

Is there a way to use an imputer that returns a pandas dataframe instead of a numpy array? Is there a scikit-learn implementation for that? I am aware of sklearn-pandas, but the interface is kind of different.


texts: What type of NN should be used for a turn-based game (with direct access to data)?

texts:
I am very intrigued and I decided to do a little project in my free time to get a hold of it.The goal of my project is to make a NN that can play a turn-based game.

I have searched for a bit about this subject and all I could find was NNs that use images of the game to make decisions, but what if I already have the game's data directly? What type of NN would work best in this situation?

I am hoping you can point me a bit to the right direction if possible.

Thank you.


texts: What is the easiest way to scale a data science project based on scikit stack?

texts:
This is an issue for all Data Scientists who have worked with this stack:

python
scikit-learn
scipy-stats
matplotlib
etc.

We are looking for ways to have a project already implemented in the aforementioned stack scale for very large datasets by doing the minimum amount of work

Counter examples would be to rewrite everything in Tensorflow framework or use industry tools that are unrelated with Python.


texts: Clustering with Only Categorical Features

texts:
I am trying to do clustering with a bunch (24) of categorical features. I have done some research and found a lot of people recommending something such as K-Modes. I tried running K-Modes on my data and the best run had a cost of 27069.0, which seems pretty high.

Some of my features have only a few values, such as P, O, C, T, so I thought I could encode them. But others have many different values. Any tips on a clustering algorithm or some other approach? I would like to use Python.

EDIT: What about using Gower distance on the data and then using K-Means on that?


texts:
I've gotten the classes for my model, but I want to get probabilities but am not sure how to set the 'Task' parameter. I get this error: Error in train(lrn, task = cr.fit) :
Assertion on 'task' failed: Must inherit from class 'Task', but has classes 'train','train.formula'.
Can someone please help me figure out how I would set the 'Task' parameter?
trCtrl <- trainControl(method = 'cv', number = 5, classProbs = TRUE, search="grid", summaryFunction=twoClassSummary)

cr.fit <- caret::train (Outcome~., data=balanced.train, method = "knn",
               trControl = trCtrl, metric = "ROC")

pred = predict(cr.fit, newdata=select(test_covid_predictor,-Outcome))

# adjust threshold and predict probabilities

lrn = makeLearner("classif.kknn", predict.type = "prob")
mod = train(lrn, task = cr.fit)

threshold = c(FATAL = 0.4)
predict= setThreshold(pred, threshold = threshold)

Thank you!!


texts: SPARK 1.5.1: Convert multi-labeled data into binary vector

texts:
I am using SPARK 1.5.1, and I have DataFrame that looks like follow:

labelsCol, featureCol
(Label1, Label2, Label 32), FeatureVector
(Label1, Label10, Label16, Label30, Label48), FeatureVector
...
(Label1, label 95), FeatureVector

The first column is the list of labels for that sample, and in total I have 100 label.

I would like to build a binary classifier for each label, so I want to transform the labels list column into a binary vector.

The binary vector will have a length of 100 and the value will be 0 or 1 depends on the existence of the label for sample.

Is there any strait forward solution for this?


texts: Using GridSearchCV for custom kernel SVM in scikit-learn

texts:
I would like to use scikit-learn's GridSearchCV() to do a grid search on custom parameters in a kernel I have specified. Specifically, the kernel is of the form

SeqKernel(x, y, orig_kernel, cut_off, order)

Here, orig_kernel is a kernel typically used in SVM learning (such as linear, polynomial, RBF, or sigmoid). I wish to perform a grid search over values of cut_off and order, with the additional caveat that only the pairs such that order $\le$ cut_off are considered. What would be the best way to implement this?


texts: Bias-variance tradeoff in practice (CNN)

texts:
I first trained a CNN on my dataset and got a loss plot that looks somewhat like this:

Orange is training loss, blue is dev loss. As you can see, the training loss is lower than the dev loss, so I figured: I have (reasonably) low bias and high variance, which means I'm overfitting, so I should add some regularization: dropout, L2 regularization and data augmentation. After that, I get a plot like this:

Now we see that the variance has decreased and the bias has increased. The model is overfitting less, is this correct? However, I would actually select the first model because it has lower validation loss.

My question is: in most literature, for the bias-variance tradeoff, they show the validation loss going up, but in my experiments this is not the case, so are these models actually overfitting? Are you overfitting as soon as training loss goes below validation loss, or only if validation loss goes back up? And is it okay to choose a model with high variance if the validation loss is lower?

I found this answer on a similar question, but what if your problem is so complex that you can't find an architecture that can overfit and then properly regularize the architecture? I can find an architecture that gets a training loss close(r) to zero, but then I can't really add enough dropout to make sure the variance is low. Also if I add augmentation my validation loss also goes up. Finally the answer confuses me, the answerer is talking about variance on the training set? But isn't bias always related to training loss and variance to dev loss?

Or am I just misinterpreting information and should I plot in function of dataset size rather than number of epochs to find out if I am overfitting?


texts: Flair Embeddings - Significance of Backwards vs Forwards?

texts:
I'm working on a project that makes use of Flair for stacked embeddings. I'm looking at the built in embeddings on this page. I noticed that the table shows news-X as being "Trained with 1 billion word corpus". However when actually making use of the embeddings it seems you either use news-forward or news-backward.

I'm assuming this means both of these embeddings are trained on the same dataset and related. However one of the projects I'm looking at actually stacks both of these embeddings:

    embedding_types: List[TokenEmbeddings] = [
        FlairEmbeddings('news-forward'),
        FlairEmbeddings('news-backward'),
        # ... additional embeddings
    ]

I'm having trouble understanding what the forward and backwards means in this situation. Furthermore if they're both based on the same data what would be the benefit of stacking them?

The page linked above also says: "We recommend combining both forward and backward Flair embeddings" but doesn't explain exactly how the two different embeddings are generated from the training data.


texts: derivation for expected value for variance

texts:
Hi Im taking a course about probability distribution in datascience and below  is  derivation of the expected value for the variance

Variance = expected value of the squared difference from mean for any value. But generally, variance is just the difference between the value and its mean.

Why are we squaring and adding the expected value symbol?
$$\sigma^2 = E((Y - \mu)^2) = E(Y^2) - \mu^2$$

For the first step in derivation, why do we multiply the summation of $p(x)$  with  $(x - \mu)^2$?

How is this substitution valid? I cannot understand it. I know that  $E(X)=p(X).X$

$E(X^2) = \sum P(X)*X^2$


texts: What is the difference between K-Means & Self Organized Maps?

texts:
It seems they both perform clustering. They both reduce the dimensionality of the input data and classify further inputs based upon their distance/similarity to the center points. These points then update to accommodate the new data.

I am yet to understand how these two methods are different. I suppose it depends on the problem to be solved. How could each be suited to different problems (advantages/disadvantages)?


texts: Find top features that determine movie rank

texts:
I am trying to analyze a movie dataset in order to find the specific features which determine whether or not the movie is in the top-10 movies of the year (or likewise the worst-10 movies of the year). The dataset contains each movie's name and ~50 categorical/binary features such as genre, type of ads before launch, whether it was filmed in the US, whether it had a celebrity lead etc. as well as the movie rank (e.g. 1st best movie, 10th best movie, etc.).
For my results I am hoping to find something like: if the movie had a celebrity and is of the action genre it is likely to be in the top-10 movies of the year. In addition, I am trying to find the optimal combination of all features that make the best possible movie.
I am very new to this, so here is my proposed approach, but I am not sure if it is correct:

Create dummies for all categorical variables
Remove correlated features
Take top-10 movies and find percentage occurrence of each feature. If those occurrences are significantly (t-test) higher than the occurrences of the feature over the entire dataset, we say that that feature is important in creating a top-10 movie.

That's all I have right now - any help on which approaches to take would be very much appreciated!


texts: Keras, DNN ending with sigmoid - model.predict produces values < 0.5. This indicates...?

texts:
I'm trying a simple Keras project with Dense layers for binary classification. About 300000 rows of data, labels are

training_set['TARGET'].value_counts()
0    282686
1     24825

My model looks like this

def build_model():
    model = models.Sequential()
    model.add(layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001),
                           input_shape=(train_data.shape[1],)))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(32, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

    return model

So it's binary classification that ends with a sigmoid. It's my understanding that I should get values close to 0 or close to 1? I've tried different model architectures, hyperparameters, epochs, batch sizes,  etc. but when I run model.predict on my validation set my values never get above 0.5. Here are some samples.

20 epochs, 16384 batch size
max 0.458850622177124,  min 0.1022530049085617
max 0.47131556272506714,  min 0.057787925004959106

20 epochs, 8192 batch size
max 0.42957592010498047,  min 0.060324762016534805
max 0.3811708390712738,  min 0.022215187549591064

20 epochs, 4096 batch size
max 0.3163970410823822,  min 0.0657803937792778

20 epochs, 2048 batch size
max 0.21799422800540924,  min 0.03832605481147766

Is this an indication that I'm doing something wrong?

Training and validation loss


texts: Same coefficient in multivariate regression with dummy variables

texts:
Hello Data Science community,

I have a model with 1 quantitative variable (y) and 2 categorical variables. In order to work with the categorical variables I have created dummy variables (binary) for the first one this is binary by nature so I still have one variable No = 0, Yes = 1, but for the second categorical value I have 48 different inputs (locations), so I've created 47 new dummy variables in which 1 = if it is that location (ref. prnt.sc/payj7q )

But the problem I'm getting know is that my output for the coefficients, looks the same for all of the x's, like this... (prnt.sc/payio0) I'm doing something wrong here or where should I look to improve the model? Again, thank you so much for your support.

Here's the code I'm using...

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as seabornInstance
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn import metrics
%matplotlib inline

# Main files
dataset = pd.read_csv('namaste_econ_model.csv')
dataset.shape
dataset.describe()
dataset.isnull().any()
#Dividing data into "attributes" and "labels". X variable contains all the attributes and y variable contains labels.
X = dataset[['Read?', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6' , 'x7','x8','x9','x10','x11','x12','x13','x14','x15','x16','x17','x18','x19','x20','x21','x22','x23','x24','x25','x26','x27','x28','x29','x30','x31','x32','x33','x34','x35','x36','x37','x38','x39','x40','x41','x42','x43','x44','x45','x46','x47']].values
y = dataset['Change in Profit (BP)'].values
#Place columns back on df
X_df = pd.DataFrame(X, columns = dataset[['Read?', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6' , 'x7','x8','x9','x10','x11','x12','x13','x14','x15','x16','x17','x18','x19','x20','x21','x22','x23','x24','x25','x26','x27','x28','x29','x30','x31','x32','x33','x34','x35','x36','x37','x38','x39','x40','x41','x42','x43','x44','x45','x46','x47']].columns)
#Plot 'Change in Profit (BP)'
plt.figure(figsize=(15,10))
plt.tight_layout()
seabornInstance.distplot(dataset['Change in Profit (BP)'])
X_train, X_test, y_train, y_test = train_test_split(X_df, y, test_size=0.2, random_state=0)
regressor = LinearRegression()
#Train the algorithm
regressor.fit(X_train, y_train)
coeff_df = pd.DataFrame(regressor.coef_, X_df.columns, columns=['Coefficient'])
coeff_df
print(regressor.coef_)

Thank you in advance for the support!


texts:
From the demo of HinSAGE link prediction I’ve understood that the model can predict the label of a link between two nodes.
I’m looking to know if a link should be existing (binary).
For example, let’s say I have a list of users and movies they watched and I want to predict if a user will watch a movie or not. There is no link between a user that did not watch a movie. So, I’m not looking to learn the label of the link but to predict if a user saw a movie (the existence of the link).
I thought about changing the data so users that saw a movie will be linked as rank 5 and users that didn’t will be linked as rank 1 and then using the HinSAGE as is.
There is a better alternative?


texts: How to label smart meter/plug data (time series)?

texts:
I would like to label smart plug data recorded in different household appliances so that I can train a machine to recognize for example when the oven is turned on. I am new to this topic and I am looking for tools/approaches that help me get started. The data would be time stamps and power consumption values.

**Edit because the question was too broad
The data would be in .csv format and I would use samples of the data where I KNOW what the appliance is doing, so that I can label it correctly. For example a washing machine on a delicate cycle or on a regular cycle.


texts: where to start in natural language processing for a language

texts:
My native language is a regional language and few people speak it. I have some assignements in a machine learning course and i was thinking about doing some natural languge processing on my native language but i don't know where to start since there is almost no research about this language ( no corpus , no research papers , ... ) and i'm new to machine learning.

I want to start doing everything from bottom and i want to do things right , can you please guide me to steps i should follow?

I also want to build my own corpus ,since it's a very tiring work , is there a way to build a single corpus that can be used on several NLP applications ( at least for translation and speach recognition)  ?


texts:
from sklearn.preprocessing import MinMaxScaler, StandardScaler
scaler_X, scaler_Y = MinMaxScaler(), MinMaxScaler()
scaler_X.fit(X)
scaler_Y.fit(Y)
X_scale, Y_scale = scaler_X.transform(X),scaler_Y.transform(Y)
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X_scale,Y_scale, test_size=.2, random_state=42)
X_train, X_validation, Y_train, Y_Validation = train_test_split(X_train,Y_train, test_size=.4, random_state=42)
model = KerasRegressor(build_fn=base_model1, epochs=50, verbose=1)
def base_model1():
    input_dim = X.shape[1]
    output_dim = 1
    model = Sequential()
    model.add(Dense(4181, input_dim= input_dim,kernel_initializer ='normal', activation= 'sigmoid')) ## 19 Fib
    model.add(Dense(2584, input_dim= 4181,kernel_initializer ='normal', activation= 'sigmoid'))      ## 18 Fib
    model.add(Dense(1597, input_dim= 2584 ,kernel_initializer ='normal', activation= 'sigmoid'))     ## 17 Fib
    model.add(Dense(987, input_dim= 1597,kernel_initializer ='normal', activation = 'relu'))       ## 16 Fib
    model.add(Dense(610, input_dim= 987,kernel_initializer ='normal', activation = 'relu'))        ## 15 Fib
    #now output
    model.add(Dense(output_dim, activation='relu'))
    # Compile model with appropirate loss, optimizer - variant of gradient descent and metrics to keep tab on progress
    #sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
    model.compile(loss='mse', optimizer='adam', metrics=['MeanSquaredError',
        'AUC',])
    return model
h5filename = str(input("Name of the H5 file named: "))
trained = model.fit(X_train,Y_train,validation_data=(X_validation,Y_Validation),validation_batch_size=None)
trained.model.save(h5filename)
y_hat = trained.model.predict(X_test)
pd.DataFrame(trained.history).plot(figsize=(5,5))
plt.grid(True)
plt.gca().set_ylim(0,1)
plt.show()

I keep getting the same Validation error in the following code, and it comes out to be straight lines. I set up my Neural Network to do regression, and it keeps giving me the same value, and a horizontal line as the line. I am bit confused could there advice to the right direction to get a better regression?


texts: Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I first use a simple function to determine the frequency of all the four and above character words. I then use this as the feature of each training sample. Now i want my algorithm to be able to cluster the training sets to according to their features, which here is the frequency of each word in the article. (Note that in this example, each article would have its own unique feature since each article has a different feature, for example an article has 10 "water and 23 "pure" and another has 8 "politics" and 14 "leverage"). Can you suggest the best possible clustering algorithm for this example?


texts: Correcting log-bias in the output of an XGB

texts:
I have previously worked with GAMs, where I was trying to do regression on a log-transformed variable. The log-transformation introduced a negative bias in the average of the predicted variable, and I corrected for this by multiplying each of the predictions $\exp(\hat y)$ by the factor

$$\langle \exp( \delta \hat y) \rangle$$

where $\delta \hat y$ was the residuals from the GAM.

Now I am using XGB, and trying to do regression on a log-transformed variable $y$ once again. The predictions $\hat y$ satisfy

$$
\frac{\sum_i \hat y_i}{\sum_i y_i} = 0.999
$$

so overall it looks good. However, when I exp-transform the variables I get

$$
\frac{\sum_i \exp(\hat y_i)}{\sum_i \exp(y_i)} = 0.861
$$

which is considerably worse. I suspect that this is due to the negative bias. Is there a way to correct for the bias in the XGB as in a GAM/GLM?


texts: Does variation in data density over time affect regression models?

texts:
So, I've been building data collectors for the stock market. Basically
this is just to give myself a resource that I can query as much as I
want and not get dinged by over usage. (the learning experience is
also a plus)

I'm curious though if the change in the frequency of data will skew
the results of a regression model (to keep things simple y = price
and x = time_stamp).

The problem comes in when the amount of data is considered. I have
the collectors recording the change in the stocks once a minute. For
data in the past that happened before I was collecting, I load the daily
open and close prices and whatever other information companies are
willing to share for free.

My question is, will the much more detailed data that I'm collecting
each minute  throw off or enhance the regression models that can be
created by just looking at the daily open and close prices?

I would assume that it would enhance it but, I would like to know if that's not the case and that in order to get an accurate measurement I would need to just use the interval that applies universally.

Here's an example chart to better explain what I mean.


texts:
What are the limitations of having too many hidden units in the Neural Network
Does it take more memory or takes longer time to train the model


texts: A model that only works by setting all initial weights to zero

texts:
In this model from MusicNet, they set the initial weights of their neural network to all zeros.

self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda()
torch.nn.init.constant(self.linear.weight, 0)

However, people normally randomize the initial weights as discussed in the following threads.

Initialize perceptron weights with zero
What are the cases where it is fine to initialize all weights to zero
Danger of setting all initial weights to zero in Backpropagation

Basically, it means that we shouldn't set the initial weights to all zeros.

However, when I tried using random weights in the MusicNet model by commenting one line out

self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda()
# torch.nn.init.constant(self.linear.weight, 0

The model doesn't learn anymore, the accuracy is very low (0.02 compared with 0.6 in the original model)

What I discovered

If the initial weights are random, the network seems unable to learn any meaningful weights. But when we set the initial weights to zero, the network can learn something meaningful.

Why is it so? I don't understand why only zero weights works in this model.


texts: how to use predictions on a single value?

texts:
I am comfortable using Machine learning on my train data and test data and validate it.
But the question here is if I want to predict a single variable how do I do it?
Let's suppose I have done feature selection using lasso or applied PCA  or done feature engineering like transformation or scaling but when I want to utilize it for new data, maybe single data I am getting confused. Should I have to follow all the steps mentioned above even for the single row of data or how do I proceed with this. Please suggest.


texts: What does embedding mean in machine learning?

texts:
I just met a terminology called "embedding" in a paper regarding deep learning. The context is "multi-modal embedding"

My guess: embedding of something is extract some feature of sth,to form a vector.

I couldn't get the explicit meaning for this terminology and that stops me from fully understanding the author's idea and model mechanism

I check the dictionary and search on line,but the explanation is based more on the real life meaning rather than meaning as a machine learning terminology.

And that raise a more generalized and frequently met question, when you find some machine learning terminology/word that you can't understand well, where can you get the solution, some specific way to google? join a machine learning group? raise a question in stack exchange?


texts: Paramaeter estimation in noisy conditions with Machine Learning, possible?

texts:
Let's take two constants, $\alpha$ and $\beta$, both are given by two functions $f_1(\vec{\theta})$ and $f_2(\vec\theta$) (the model). These functions are known: we have an analytical closed expression, that for a given set of values $\vec\theta$, $\alpha$ and $\beta$ are retrieved.

PROBLEM:

The components of $\vec\theta$ are noisy measurements, so, the variance of $\alpha$ and $\beta$ is high: for different measurements, we have quite different values for them... though our model defines them as constants. Furthermore, the model is not actually realistic, and obviously it imposes some assumptions.

formulation:

The observed values are $\vec\theta = [\beta, \dot\psi, \delta, A, B, C]$

\begin{equation}
\alpha = \frac{\dot\beta + \dot\psi}{-2A\beta + B\dot\psi A\delta}
\end{equation}
\begin{equation}
\beta = \frac{\alpha K\beta + \alpha C\dot\psi + \alpha D\delta}{\ddot\psi}
\end{equation}

(The derivatives are computed using finite differences)

NOTE: the observed values of $\vec\theta$ are not independent since they come from the same sensing unit (which little do we know about), so I think assuming statistical independence won't work here...

QUESTIONS:

Are there common approaches in Machine Learning to mitigate the effect of the noise? Can Machine Learning help to get the true values? Can Machine Learning model the variance of the estimators? By Machine Learning I mean algorithms that are not as known as Kalman filtering.

MOTIVATION AND CONSIDERATIONS:

I was asked this question by a colleague at my job office, since I work on Machine Learning. I personally think that a better approach is to face it from the estimation theory (ref.). For instance, modeling the noise of the sensors and try to get Maximum Likelihood estimators, and if possible, derive a bound for their variance (CRLB) or a closed expression for the bias. For handling the noise, I suggested u-Kalman, Particle Filtering... But other terms that came to my mind were: "fuzzy", "GMM", "EM"...

EDIT:

I think what my colleague wants is a "data-driven/black-box" approach that can help him skip the math, since he has not the enough background...


texts: How to train neural network that has different kind of layers

texts: If we have MLP then we can easily compute the gradient for each parameters, by computing the gradient recursively begin with the last layer of the network, but suppose I have neural network that consist of different type of layer for instance Input->convolution layer->ReLu->max pooling->fully connected layer->siftmax layer,  how do I compute the gradient for each parameters ?


texts: Machine Learning: Take into account a variable if a condition is met (depending on another variable)

texts:
I am working on a set of student data to train some models. I have the gender variable and I can also retrieve data about how many other girls go to this student's class. I would like to take into account in the modelling that if it is a girl, whether there are other girls in her class or not. This variable shouldn't have any interference in the cases where the student is a boy.

Any idea how I should treat this?


texts: Max-pooling vs. zero padding: Loosing spatial information

texts:
When it comes to convolutional neural networks there are normally many papers recommending different strategies. I have heard people say that it is an absolute must to add padding to the images before a convolution, otherwise to much spatial information is lost. On the other hand they are happy to use pooling, normally max-pooling, to reduce the size of the images. I guess the thought here is that max pooling reduces the spatial information but also reduces the sensitivity to relative positions, so it is a trade-off?

I have heard other people saying that zero-padding does not keep more information, just more empty data. This is because by adding zeros you will not get a reaction from your kernel anyway when part of the information is missing.

I can imagine that zero-padding works if you have big kernels with "scrap values" in the edges and the source of activation centered in a smaller region of the kernel?

I would be happy to read some papers about the effect of down-sampling using pooling contra not using padding, but I can't find much about it. Any good recommendations or thoughts?

Figure: Spatial down-sampling using convolution contra pooling (Researchgate)


texts: What to do if a specific label of a category appears only a few times?

texts:
Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its color, model, etc. The model is the name of a car(e.g: Sportage, Mazda3, etc.) and some of the more famous models such as Sportage appear many times whereas some of the less popular ones might appear only once or twice. In that case, what would be the ideal way to deal with this?

More info:

In my case, I have about 3000 different car models and the first two or three make up about 20% of my data but the rest just appear once or twice in the entire dataset. I have tried one-hot encoding and that did increase my score immensely but it's still not good enough (I know as a matter of fact it could be better).

P.S: I have already looked at the posts regarding a high cardinality and although I do think it's related to my problem, it's still a different issue.

Thank you so much!


texts:
The question is in reference to solution of Titanic survival predictionat kaggle . As many have did the similar kind of feature extraction,

They have converted some of the numerical features (Age, Fares) to Categorical types with Nominal labels (1,2,3) using bands.
Does it help for models to fit efficiently if we have all features in categorical values form?

I have been knowing that if you have categorical values, you should convert them into numerical first, and than to DUMMY Variables. But in the solution it is not done, is dummying not needed?


texts: What are the differences between biased and unbiased learners?

texts:
I'm new in machine learning and I am willing to know better what is the difference between biased and unbiased learners?
Anyone can provide some examples?


texts: What features most contribute for label y?

texts: On a neural network, how can I find out what features most contribute for some label (the dependent variable y)?


texts: How to process natural language queries?

texts:
I'm curious about natural language querying.  Stanford has what looks to be a strong set of software for processing natural language.  I've also seen the Apache OpenNLP library, and the General Architecture for Text Engineering.

There are an incredible amount of uses for natural language processing and that makes the documentation of these projects difficult to quickly absorb.

Can you simplify things for me a bit and at a high level outline the tasks necessary for performing a basic translation of simple questions into SQL?

The first rectangle on my flow chart is a bit of a mystery.

For example, I might want to know:

How many books were sold last month?

And I'd want that translated into

Select count(*)
  from sales
  where
   item_type='book' and
   sales_date >= '5/1/2014' and
   sales_date <= '5/31/2014'


texts: Why is my training accuracy decreasing higher degrees of polynomial features?

texts:
I am new to Machine Learning and started solving the Titanic Survivor problem on Kaggle.

While solving the problem using Logistic Regression I used various models having polynomial features with degree $2,3,4,5,6$ . Theoretically the accuracy on training set should increase with degree however it started decreasing post degree $2$ . The graph is as per below


texts: Accuracy graph of binary classification by CNN

texts: Why in binary classification of images with CNN the loss and accuracy graph are so unstable? I mean accuracy of validation test does not increase smoothly, it goes to 80%, then comes to 60%, then again goes to 84% and so on. Same is the case with train accuracy. Now how do I know that how many epochs is the optimal number?


texts:
I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of the training set and testing set are similar.
And I search it and find a blog which check the similarity of distributions by converting it into a binary classification problem. If it gets a high AUC, the distribution of the training set and the testing set must be different. And the idea he gives as follows:

  If there exists a covariate shift, then upon mixing train and test
  we’ll still be able to classify the origin of each data point (whether
  it is from test or train) with good accuracy.

But I still can't understand why he can check the similarity of these two distributions in this way.
And are there other ways to check the similarity of it?
It will be appreciated, if anyone could help me.


texts: How to fit pretrained yolov4 model to other class?

texts:
I have already trained model for one class, so I have weights and how can i train it for second class? This model detects cars 100% and I need it to detect vehicle plate number, I used yolov4, darknet framework and google colab, please could someone show me as a code? I can send the code file.
I ran training by this line:
!./darknet detector train data/obj.data cfg/yolov-4-obj.cfg yolov4.conv.137 -dont_show -map

In my google disk I have file with wieghts.


texts: does storing file in hdfs parallelize it for Spark?

texts:
For Spark's RDD operations, data must be in shape of RDD or be parallelized using:

ParallelizedData = sc.parallelize(data)

My question is that if I store data in HDFS, does it get parallelized automatically or I should use code above for using data in Spark? Does storing data in HDFS makes it in shape of RDD?


texts: State of the art approaches for Information retrieval tasks based on deep learning

texts:
I am interested in the state of the art approaches for information retrieval (IR) tasks, where you have a single query and a set of documents and the IR model will give you the best matched document.

I have worked on vector space models (tfidf-cosine similarirty ) and LSA.

I have also tried Wordnet, NER, fuzzy matching etc for improving the accuracy.

Now I would like to know how to improve accuracy of IR tasks by applying nueral networks, word embeddingstopic models etc by capturing more context/sematics information


texts: Having trouble understanding None in the summary of my Keras model

texts: The above code is a sample of a CNN model built using Keras. The first layer is a convolutional layer which will receive images of input_shape = (64, 64, 3), thus meaning that the format of the images is in RBG. The output of the first layer is (None, 62, 62, 32), but isn't in supposed to be (62, 62, 32)? Where does the None come from? Also since my images are (64, 64, 3) doesn't it mean that the filter should have 3 dimensions to cater for the color channels of the image?


texts:
I want to extract various amounts and tenure of contracts from different contract documents that we have.

For example :

  Mr xyz, this contact is valid for 3 Months and has to be executed within 1 Month.
  you have to pay \$3000  as  contract fee, \$60 as taxes, \$1200 as security deposit and \$1200 as rent

Expected output : Contract tenure : 3 Months, Amount to pay : \$3060

Please note : I tried NER but that is showing 2 tenured and 2 amounts. However I am looking for a technique by which we can associate amount to contract.


texts: Hindsight experience replay: strategy for sampling goals

texts:
The authors of Hindsight Experience Replay list out several strategies for sampling a set of additional goals $G$ in Section 4.5:

final - corresponds with final state of environment,
future — replay with k random states which come from the same episode as the transition being replayed and were observed after it,
episode — replay with k random states coming from the same episode as the transition
being replayed,
random — replay with k random states encountered so far in the whole training procedure.

My interpretation of the future method is then that we can only select k random states if the current transition being replayed has already happened in the episode, so this is at minimum the second time we've seen the current transition. This seems very unlikely if working in an environment with a large state space (especially with continuous features). Am I missing something obvious in the interpretation of how to implement this strategy?


texts: Estimating coefficients of AR(n) process with deep learning

texts:
Can a deep learning model estimate parameters of an AR(n) process given a time series generated by such process? What kind of deep-learning model would you recommend?
Note: I am very new to machine learning.


texts:
I am trying to perform regression using XGBoost. My dataset has all positive values but some of the predictions are negative.
I read on this link that reducing the number of trees might help the situation.
I reduced the estimators from 700 to 570 and the number of negative predictions decreased but is there any way to remove these negative predictions? When I further tried reducing the estimators to 400, I got worse results with high rmse and more negative predictions.
I looked at the negative predictions to understand where the model went wrong(see below image):

It looks like if I take the absolute value of the negative predictions then that would be sufficient but is this the right method?
My code is:
!pip install xgboost
!pip install scikit-optimize
from xgboost import XGBRegressor

final_XGB=XGBRegressor(random_state=123,gamma= 24.47 ,learning_rate=0.1235,max_depth=10,min_child_weight=0.21509999999999999,
                      n_estimators=570,subsample=0.74,reg_lambda=0.8)

from sklearn.model_selection import cross_validate

cross_val_scores=cross_validate(final_XGB,X_train,y_train,cv=3,scoring=['neg_mean_squared_error','r2'],verbose=1,return_train_score=True,n_jobs=-1 )

cross_val_scores['test_r2'].mean() #returns 0.9595609470775659

EDIT:
A bit more about the dataset. I am trying to predict the number of people that would be present at a given location at a given time period.
In order to predict the count of people, I have taken the

hour of day,
date,
peak(is it a public holiday?),
Weekend(1/0),
Area (area of location in m2)
location name.

The location names were string values (eg: Location1, Location2, etc.) so I transformed them using JamesSteinEncoder.
There are no missing values and I have removed a few outliers from the dataset.
Regarding the relationship between "Actual" and "Error" column, I have plotted the graph below:

Snapshot of final results:


texts: The output of Model from Decision Tree and Random Forests are different?

texts:
I have been made a model using both Decision Tree and Random Forest. But, when I tried to test the model on the same DataFrame the output is different. How is this possible?

The data file from my repo:

#This is the function to help me preparing the dataframe
def process_df_for_ml(df):
    """
    Process a dataframe for model training/prediction use.

    Returns X/y tensors.
    """

    df = df.copy()
    # Map salary to 0,1,2
    df.salary = df.salary.map({"low": 0, "medium": 1, "high": 2})
    # dropping left and sales X for the df, y for the left
    X = df.drop(["left", "sales"], axis=1)
    y = df["left"]
    return (X, y)

I used the decision tree:

from sklearn import tree
from sklearn.model_selection import train_test_split
# Train a decision tree.
X, y = process_df_for_ml(df)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)
clftree = tree.DecisionTreeClassifier(max_depth=3)
clftree.fit(X_train, y_train)

With test_score: 0.96. Afterwards, I test this DecisionTree to the same df and the output I got is [424 rows x 11 columns]

and then I tried to use Random Forest algorithm

X, y = process_df_for_ml(df)

from sklearn.model_selection import train_test_split
# implementing train-test-split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0, stratify=y)

from sklearn.ensemble import RandomForestClassifier
# random forest model creation
rfc = RandomForestClassifier()
rfc.fit(X_train,y_train)
# predictions
rfc_predict = rfc.predict(X_test)

With test_score: 0.99. Afterwards, I test this RandomForest to the same df and the output I got is [11 rows x 11 columns].

How is this possible?
Here is the link to my works:
DecisionTree and RandomForest


texts: How to learn certain Maths to understand machine Learning papers?

texts:
I have done the deeplearning.ai course on deep learning. But I cannot Understand equations like

minGmaxDV(D,G)=Ex∼pdata(x)[logD(x)]+Ez∼pz(z)[log(1−D(G(z)))]

What kind of Maths am I am supposed to learn? I know Calculus basic Multivariable Calculus and Linear Algebra. So should I learn Probability (Advance probability maybe)?


texts: Last cell in recurrent network always the most accurate

texts:
I am using a recurrent network for time series forecasting.

The prediction from the last cell in the network always seems to be the most accurate. For example if I have 20 cells (so my input samples are of length 20), the prediction from the latter cells (e.g. cells 11-20) seem better than the predictions from the early cells (e.g. cells 1-10).

Is this normal?

Lets assume my network only has 3 cells. At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach? If I want good predictions at all timesteps, do I need to using a "moving window" i.e. should I use a rolling sequence of inputs:

For prediction at t=0, input x(t-2, t-1, t=0)
For prediction at t=1, input x(t-1, t=0, t=1)
For prediction at t=2, input x(t=0, t+1, t+2)


texts:
I am a student and I am studying machine learning. I am focusing on deep generative models, and in particular to autoencoders and variational autoencoders (VAE).
I am trying to understand the concept, but I am having some problems.
So far, I have understood that an autoencoder takes an input, for example an image, and wants to reduce this image into a latent space, which should contain the underlying features of the dataset, with an operation of encoding, then, with an operation of decoding, it reconstrunct the image which has lost some information due to the encoding part.
After this, with a loss function, it reconstruct the latent space and so get the latent features.

about the VAE, it uses a probabilistic approch, so we have to learn the meand anv covariance of a gaussian.
So far this is what I have understood.
What I have really unclear is what are we trying to learn with autoencoders and VAE?
I have seen examples where an image goes feom a non smiling to a smiling face, or to a black and white image to a colored image.
But I don't understand the main concept, which is: what does an autoencoder do?
I add here some sources of where I studied so that who needs can see them:

https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694
https://www.youtube.com/watch?v=yFBFl1cLYx8
https://www.youtube.com/watch?v=9zKuYvjFFS8


texts: Gini coefficient vs Gini impurity - decision trees

texts:
The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - these can support our choices when splitting the set of items.

1) 'Gini impurity' - it is a standard decision-tree splitting metric (see in the link above);

2) 'Gini coefficient' - each splitting can be assessed based on the AUC criterion. For each splitting scenario we can build a ROC curve and compute AUC metric. According to Wikipedia AUC=(GiniCoeff+1)/2;

Question is: are both these measures equivalent? On the one hand, I am informed that Gini coefficient should not be confused with Gini impurity. On the other hand, both these measures can be used in doing the same thing - assessing the quality of a decision tree split.


texts: What is the difference between biclustering and clustering?

texts:
After reading the wiki page for biclustering (https://en.m.wikipedia.org/wiki/Biclustering), I am really confused on what is the difference between biclustering and clustering?

Any explanation/comparisons would be really appreciated. Many thanks.


texts: generalisation of k-handed bandit to n-handed player

texts:
Is there a generalisation of $k$-handed bandit to $n$-handed player in a sense that the sample is taken not from just one distribution (one bandit from $k$ bandits) but from n distributons ($n$ bandits) where $n<k$?

In other words instead of predicting best ad banner in terms of click though rate I'd need to predict a set of ad banners that will be shown simultaneously on the page (but only one clicked or not and it does not matter which).

Or yet another way to formulate is to say that the generalized $n$-arm is a vector of bandits arms instead of one banidts arm. The pool of bandits is fixed.


texts: activation function for binarized neural networks

texts:
I am trying to implement a binarized neural network using keras and tensorflow.
Here is my current code:
import tensorflow as tf
from tensorflow import keras
from keras import datasets
from keras.utils.vis_utils import plot_model

from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

def sign(x):
    result = K.sign(x)
    return result

get_custom_objects().update({'custom_activation': Activation(sign)})

digit_data = datasets.mnist
(train_images, train_labels), (test_images, test_labels) = digit_data.load_data()

# import matplotlib.pyplot as plt
# plt.imshow(train_images[0])
# print(train_labels[0])
# print(train_images[0])

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(784, Activation(sign, name='SpecialActivation')),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])

# model = keras.Sequential([
#     keras.layers.Flatten(input_shape=(28, 28)),
#     keras.layers.Dense(128, activation=tf.nn.relu),
#     keras.layers.Dense(10, activation=tf.nn.softmax)
# ])

# plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)

model.compile(optimizer=tf.keras.optimizers.Adam(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=5)
test_loss, test_acc = model.evaluate(test_images, test_labels)
print(test_loss, test_acc)

The activation function I am currently using is a sign function which returns -1 if x < 0 | 0 if x=0 | 1 if x >0
the binarized neural network needs to have a bitwise operation function which returns +1 if x => 0 and -1 if x< 0 (from what i understand)
I'm not sure how to implement that kind of an activation function.


texts: How much data should I allocate for my training and and test sets? (in R)

texts:
I have a matrix of 358.367 data.
Each row is a DNA sequence from the human genome.
I want to build a classification model in R, using XGBoost algorithm and 83 features (dinucleotides, trinucleotides, etc.).

How should I split the data for the train and test set?

For example 70% for the train set and 30% for the test set?
30% for the train set and 70% for the test set?


texts: How can learn machine learning with Statistics background?

texts: I have statistical background now I want to learn machine learning. I have experience in R, SPSS, Stata. Necessary resources and tutorials which are effective not only waste my time are needed for my early research career. I have searched several tutorials but I need effective one.


texts: Vector space model cosine tf-idf for finding similar documents

texts:
Have corpus of over million documents

For a given document want to find similar documents using cosine as in vector space model

$d_1 \cdot d_2 / ( ||d_1|| ||d_2|| )$

All tf have been normalized using augmented frequency, to prevent a bias towards longer documents as in this tf-idf:

$tf(t,d)=0.5+0.5\frac{f(t,d)}{\mathrm{max}\{f(t,d): t\in d\}}$

Have pre-calculated all $||d||$
Have the values for the denominator pre-calculated
So for a given $d1$ need to score over 1 million $d2$
Have a threshold of 0.6 cosine for similarity

I can observe that for a given $||d_1||$ there is a fairly narrow range of $||d_2||$ for cosine $\ge$ 0.6
For example in one search for similar for a cosine of $\ge$ 0.6 and a $||d_1||$ of 7.7631 then $||d_2||$ range from 7.0867 to 8.8339
Where outside the threshold of cosine 0.6 $||d_2||$ range from to 0.7223 to 89.3395
This was with standard tf document normalization
It is looking at a LOT of $||d_2||$ that don't have a chance of being a cosine 0.6 match

Finally the question:
For a give $||d_1||$ and cosine of >= 0.6 how can determine the range of $||d_2||$ that have a chance?
Which $||d_2||$ can I safely eliminate?

I also know the number of terms in $d_1$ and $d_2$ if there is term count range.

Via experimentation
$||d2|| > .8 ||d1||$ and $||d2|| < ||d1|| / .8 $
seems to be safe but hopefully there is range that is proven to be safe

Created some test cases with a very some unique terms, some not so unique, and some common.  Sure enough you can take the most unique term and increase that frequency in the compare.  The numerator will (dot product) go up and so will ||compare|| and will get a cosine very close to 1.

Kind of related and NOT the question.
I am also using the tf-idf to group documents into groups.
The customer base I am selling into are used to near near dup groups.
There I am taking a related approach in I look as the smallest term count and evaluate it against term count up to 3x.  So a term count of 10 looks at 10  thru 30 (4-9 already had their shot at 10).  Here I can afford to miss one have it picked up in another.  I am 10% done and the biggest ratio is 1.8.

Please identify the flaws in this analysis
As pointed out by AN6U5 there is a flaw in this analysis
It is no longer a cosine if the document is normalized on weighted
And as pointed out by Mathew also cannot conclude  d1⋅d2≤d1⋅d1
I am still hoping for something to give me a hard bound but people that seems to know this stuff are telling me no
I don't want to change the question so just ignore this
I will do some analysis and maybe post a separate question on document normalization
For the purpose of this question assume the document is normalized on raw tf
Sorry but I am just not good with what ever markup is used to make the   equations
So in my notation
||d1|| = sqrt(sum(w1 x w1))
d1 dot d2 = sum(w1 X w2)
Assume d1 is the shorter document
The very best d1 dot d2 that can be achieved is d1 dot d1
If d1 is marry 100 paul 20
And d2 is marry 100 paul 20 peter 1
Normalized
d1 is marry 1 paul 1/5
d2 is marry 1 paul 1/5  peter 1/100
Clearly marry and paul have the same idf in both documents
The best possible d1 dot d2 is d1 dot d1
The maximum possible match to d1 is d1
cos = d1 dot d1 / ||d1|| ||d2||
square both sides
cos X cos = (d1 dot d1) X (d1 dot d1) / ( (d1 dot d1) X (d2 dot d2) )
cos X cos = (d1 dot d1) / (d2 dot d2)
take the square root of both side
cos = ||d1|| / ||d2||
is ||d2|| not bounded by the cos?
If I just use ||d2|| >= cos ||d1|| and ||d2|| <= ||d1|| / cos I get the computational speed I need


texts: Is there a standard term for data that are not missing? I.e. is it called non-missing, present, or something else?


texts:
I'm trying to build a recommendation engine for an e-commerce site. By using the common recommendation approach, I'm assuming that each product I recommend has the same value, so all I need to do is optimize the conversion rate probably using a common recommendation algorithm, but when the product's price varies a lot, what I really need to optimize is the following formula for each user:

Value of recomendation = (probability to convert) * (product price)

The bigger problem than choosing the right algorithm and approach is choosing the right metric, so I could compare the different algorithms. For example, if you would like to only optimize the conversion rate, I would use the precision and recall or false/positive metrics.

What metrics and approaches/algorithms are recommended in this case?

Thanks


texts: How would one impute missing values for a Discrete variable?

texts: How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.


texts: How to suggest medical treatments based on injury subjective and objective notes using ML?

texts:
There are 30k+ history injury subjective and objective note records in the dataset, each one with correct corresponding treatments data, the simplified history dataset structure is like:
injury subjective note | GP's objective note | treatments note
I cut my finger, ... | The wound is bleeding, ... | Clean the wound, treat with antibiotic ointment, ...
Fell from cycling, forearm hurts, ... | Bone fracture, ... | Bone X-ray, splint, braces, casts, slings, ...
Now how to using ML/AI to suggest (output) treatments based on new (input) subjective and objective note?
Would like to know:

What algorithms could be best fit?
What tools/platforms are good candidates to choose from?
Any resource links/pages with similar project/purpose that could learn from?


texts: How to fairly rank the best items having two set of grades?

texts:
My problem:

I have a set of items.
Two judges will subjectively rate each item giving a grade from 0 to 10.
I have to select the best 10 items.

How do I rank the items so that the judges are equally important?

Suppose one of them is really rigorous, and grade the items with a lower value, with a median of 5 and a standard deviation of 1.

The other median is 8, but his grades are more spread, with a standard deviation of 2.

Here are examples of the grades distributions:

The second judge has more spread grades, so if I just calculate the mean and rank, a lower grade of Judge Two would eliminate one item. Judge One grades would just order the best items of Judge Two. A favorite of Judge One would hardly get into the favorite rank.

Just to rank the items of each judge would make it more fair, but wouldn't take the grade difference between the items.