texts: Training with a subset of data: relationship between subset size and training metric?

texts:
I have a not-quite linear regression problem which I am investigating. The data set is fairly large, with ~6000 samples and ~2100 features.

By performing 5-fold cross-validation on different sized subsets of the data set, I have found a strong relationship between the fraction of the data set used and the value of both the R-squared metric and the RMSE metric.

EDIT for clarity: for a fraction of 0.01, I am taking 1/100 of the samples (~60) and then performing repeated 5-fold cross-validation as if this were the whole data set. The values shown are means.

It appears that RMSE varies linearly with the reciprocal of the fraction used (1/Frac), and the R-squared regresses very well against a second-order polynomial of 1/Frac.

The raw results are as follows:

Frac   1/Frac  Test R2      Test RMSE
0.01   100     -1.65628     1.85292
0.013  75      -0.71208     1.71476
0.02   50      -0.00786883  1.38874
0.05   20       0.535839    1.06872
0.10   10       0.626532    1.00598
0.20   5        0.702421    0.95058
0.30   3.33     0.745277    0.860082
0.50   2        0.772211    0.86548

Relationships:

My questions are:

Is this a well-known relationship?
Does it have a name (so I can research further)?
Is there a theoretical basis for it?


texts:
My task is to classify free text originated from customer complaints about our product.

I have created a Taxonomy and have around 10 different categories. I've realized that these categories include keywords.

Example:

"Customer doesn't understand how to use the product".

Keywords: understand, knowledge, know, aware.

Record:

Training, Customer doesn't understand how to use the product

I'm using Google Prediction API. When training the model, I would categorize previous text as: "Customer doesn't understand how to use the product" - Training.

How can I add keywords to free text/training data to help the model perform better and provide a better confidence level?

Data in training set:

Training, understand knowledge know aware
Training, Customer doesn't understand how to use the product

Right now, I'm adding Keywords into same training data, but looking for a better suggestion.


texts: ScikitLearn - RandomForestRegressor score different in and out of grid search

texts:
I am using RandomForestRegressor (scikit-learn python package). I am looking for the best values for hyperparameters n_estimators and min_samples_split for fitting my regressor on a train dataset ( X_train, y_train ) :

  param_grid = { 'n_estimators' : range( 10 , 201 ) ,
                 'min_samples_split' : range( 2 , 11 ) }

  rfr = RandomForestRegressor()

  best_score = 0
  best_param = {}

  for param in ParameterGrid( param_grid ):
      rfr.set_params( **g )
      rfr.fit( X_train , y_train )
      score = rfr.score( X_train , y_train )

  if score > best_score :
      best_score = score
      best_param = param

  print( 'best_score : {0}'.format( best_score ) )
  print( 'best_parameters : {0}'.format( best_param ) )

I am not using cross validation on purpose.

The best hyper parameters are in the best_param variable , the corresponding score in the best_score variable.

After that, I set the hyper parameters of my regressor using the best ones :

rfr.set_params( **best_param )

I fit the regressor on my train dataset:

rfr.fit( X_train , y_train )

Finally just to check I obtain the score of the regressor with the best hyper parameters:

rfr.score( X_train , y_train )

The score is not the same as the one calculated in the grid : 0.806 out of grid versus 0.963 in the grid.

I do not understand why. The hyper parameters are the best ones, and the dataset used to calculate these best parameters is the same. Any hints?


texts: What is the relationship between "landmark Detection" and "landmark localization"

texts:
I am reading this paper "Grand Challenge of 106-Point Facial Landmark Localization"

In the context of face recognition "Landmark Detection" is to detect a face by matching landmarks on a face.

"Landmark Localization" is to predict the coordinates of a set of pre-defined key points on human face.

What is the relationship between "Landmark Detection" and "Landmark Localization"?


texts: Deep reinforcement learning with multi-dimensional action

texts:
I am trying to design reinforcement learning algorithm. My action and state space are continuous. Action, which I would like to take can be represented by a matrix, lets say of dimension $n \times n$. And I would like to use reinforcement learning to find optimal entries of this matrix.

So far, I looked into deep reinforce algorithm. However, it is not clear to me how to design network/algorithm, such that it allows for modifying entries of the matrix.

As far as I understood, output of the network should represent probability, with which we take certain action. I wonder, do I need to model each dimension of the action(each element of the matrix) as an independent prob. process? Is it good idea to model it by the same network?


texts: Which algorithm to use to predict the duration of some task

texts:
I have a data set of 36K processes each with 7 features. Each set represents a task and I know how long it took to complete each of the tasks. I want to build a model which will be able to predict the duration of the future tasks.

I've only learned about Decision Tree (DT) and tried to apply it to my problem. The resulting accuracy score is 0.03. I believe DT is not suitable because the time is continuous and DT is meant for categorization.

Which algorithm is suitable for duration prediction?

My environment: Python with sklearn, if that matters.


texts:
I have this problem:

import pandas as pd

stripline = "----------------------------"

rawData = {
    'order number': ['11xa', '11xa', '11xa', '21xb', '31xc'],
    'working area': ['LLA', 'LLE', 'LLS', 'MLA', 'MLE'],
    'time': ['1', '6', '13', '35', '24']
}

df = pd.DataFrame(rawData)
print("original data:")
print(df.head())

print(stripline)

rawData2 = {
    'order number': ['11xa', '21xb', '31xc'],
    'working area': ['LLS', 'MLA', 'MLE'],
    'time': ['20', '35', '24']
}
df2 = pd.DataFrame(rawData2)

print("expected result:")
print("group after order number, sum all times to that order and choose working field with the biggest time")
print(df2.head())

How can I manipulate my dataframe df to get the df2?

I want to sum up all values in the time column that correspond to an order number. I want to use the working field with the highest time and especially I want to keep the rest of the data. The new data frame has three orders, the old one five.


texts: Different Test Set and Training Set Distribution

texts:
I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test set.

How can I do this?


texts: Dimensionality reduction to correlate large number of variables

texts: So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a correlation matrix between these dimensions. I have continuous and discrete columns. I would also like to be able to know which dimensions represent which bundle of single variables in the original dataset. I should add that I have no output variable, I am just trying to find correlation in the data. Is there a way to do this in Python?


texts: How does the R implementation of RandomForest split nodes on categorical data?

texts: The R implementation of RandomForest can take in categorical features as factors and train and predict on these features without encoding. Normally, I use the python implementation from scikit-learn that requires encoding of categorical data before training. I am trying to determine the best encoding strategy for my problem, which has me wondering, how does the R implementation handle categorical data? I would think it would have to be encoded somehow under the hood. If so, what encoding strategy is used?


texts: How to use sklean pipeline to deal with data that read in line by line

texts: The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass operations that I wrote in the future. If I read data line by line, is there any way I can still use sklearn pipeline API?


texts: What kind of classification should I use?

texts:
I'm very new to machine learning (and statistics) and I'm struggling with some basics (I'm using R as my primary environment).

I have a lexicon with words and all of their forms (in different genders, persons - for a Slavic language). So each word with its word forms is one training sample. My training attributes are last character, last two characters, last three characters of each word form.  So each training sample contains from around 100 categorical attributes. I've got 10000 samples. One attribute has around 1700 levels, one around 1200, 5 around 500, others have less than 100 levels.

I have used DBSCAN (and gower distance) to cluster similars words to the same cluster. This step was pretty successful. I get (depending whether is a noun, verb or adjective) from 40-100 clusters.

Now here comes the part where I don't know how to continue. If I get a new (unknown) word I want to predict to which cluster it belongs. Sometimes for this new word I only know its base form, sometimes I also part of its word forms and sometimes I know all of its word forms.

So this should be some kind of classification with multiple classes. Question: which algorithm should I use? I've tried to use multinom (in R) but I get an error of too many weights. After I increased MaxNWts I get an error:

weights:  641676 (633954 variable)
Error: cannot allocate vector of size 396 Kb

Where should I continue? Clean the data? Use a different algorithm?

Also, around 15 attributes have only 1 level or NA. How does this impact to clustering (and classification)? Is it safe to remove those attributes?

Many thanks in advance!


texts:
It's quite a challenging task of aspect extraction in the field of opinion mining if you look at the number of related papers.

But what is an aspect in the field of opinion mining?


texts: Finding the position of an arbitrary object in a static image?

texts:
One common object detection scenario involves finding trained models in an arbitrary scene. For example, we can train a model to understand what a "bicycle" looks like, by providing various images of bicycles we've seen in the past. Then, we can check an arbitrary scene of a busy intersection to determine if it includes anything like our "bicycle."
I'm trying to achieve the exact opposite. In this problem, I have a static scene, like this:

Now, I'm trying to see if there is anything similar in the scene, by providing images like the one below. I may provide infinitely many "test" images, and I don't know anything about their content, ahead of time:

In this case, there should be a near-100% match, since the test image is literally cut from the scene.
In the general case, if there is any reasonable close match in the image, I'd like to know its position, and a confidence interval:
{
  "matches": [
    {
      "id": "first_match",
      "bounds": [
        {
          "x": 345,
          "y": 955
        },
        {
          "x": 370,
          "y": 1000
        }
        {
          "x": 400,
          "y": 800
        }
      ],
      "confidence": "0.1"
    }
  ]
}

Or, represented visually, the match from this example would occur here:

I have looked at a few tools, like Amazon Rekognition. However, Rekognition is primarily focused on solving the inverse problem (given some trained models, do they exist in an arbitrary scene)?
What approaches or tools can I use to solve this problem?


texts: gridsearchcv vs LassoCV/RidgeCV in Python

texts: Is there any advantage to using GridSearchCV over LassoCV/RidgeCV? I don't actually understand the point of the former, when we have the latter. It seems easier to just use the latter functions rather than use a GridSearchCV and having to wrap a model within it.


texts: pandas data frame doesn't show any thing ,when view as data frame in pycharm

texts:
import pandas as pd;
dataSet = pd.read_csv("winequality-red.csv");
dataSet.describe(include = 'all');

When view data set as data frame ,it show empty table.But when printing dataSet,I get following result.

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1599 entries, 0 to 1598
Data columns (total 12 columns):
fixed acidity           1599 non-null float64
volatile acidity        1599 non-null float64
citric acid             1599 non-null float64
residual sugar          1599 non-null float64
chlorides               1599 non-null float64
free sulfur dioxide     1599 non-null float64
total sulfur dioxide    1599 non-null float64
density                 1599 non-null float64
pH                      1599 non-null float64
sulphates               1599 non-null float64
alcohol                 1599 non-null float64
quality                 1599 non-null int64
dtypes: float64(11), int64(1)
memory usage: 150.0 KB
Backend TkAgg is interactive backend. Turning interactive mode on.
<class 'pandas.core.frame.DataFrame'>


texts: Challenge in visualizing large coronavirus clusters in US data

texts:
I'm trying to take the data on large coronavirus clusters in the US and visualize them to show the sizes and the different settings (prisons, healthcare facilities, etc).  I want to show the difference between the different settings.
If the sizes were more similar, I'd try to show a stacked bar chart (with size as horizontal axis and count as vertical axis).  Unfortunately, that's not working well because some clusters are much bigger than others.
The first few lines of my data look like (there are lots of aged care facilities with 50 cases):
size category
50 agedcare
50 agedcare
50 agedcare
50 agedcare
50 agedcare
50 agedcare
50 agedcare

and the bottom looks like  (the prisons and meat packing facilities have huge outbreaks)
931 prisons
981 prisons
1028 prisons
1031 meat
1051 prisons
1065 prisons
1098 meat
1107 prisons
1283 prisons
1362 prisons
1374 prisons
1791 prisons
2439 prisons

Here is a visualization of the smaller sizes
I can do some binning and I get this:

But it's still hard to immediately see that some of these setting types have small outbreaks while others have much larger ones.
Any suggestions on how to visualize would help (I use python primarily if that matters)


texts: Huge performance discrepancies at each run, with the same CNN architecture

texts:
I have set up a CNN architecture with 3 Convolutional layers with pooling (except the last one), a fully connected layer and a logistic regression layer. All the layers, except the last one, have a ReLU activation function to train on the MNIST dataset. I have followed LeNet tutorial

When I train my model, I have some weird behaviours. One time out of five (approximately), my network will early-stop at the second epoch with an error of 89% (so, quite bad) and sometimes, the same network will yield a test performance of ~1.3%, which is great in my opinion. This is totally random and this is weird.

I thought of two reasons on why this might happen :

Weight initializations might not be right. However, I have double-checked the weights initialization for each layer type. While the LogReg layer initializes its weights at 0, the two other layers (FullyConnected & ConvPool) initialize theirs weights according to a normal distribution whose parameters are

np.random.normal(loc = 0, scale = np.sqrt(1.0 / fan_out),size = filter_shape)
There is an error in the SGD step. I double-checked my SGD step with the tutorial's, and I can't spot any discrepancies. Moreover, I don't understand why it would work sometimes, and be broken at others.

Is there any other reasons I might have not considered?


texts: How does the given data gets plotted on a graph

texts:
I come from a programming background and learning the math behind the data science and algorithms now.

I would like to understand the logic behind how a data gets plotted in a graph when using Logistic regression.

Lets consider a Testing data like this,

ID  Age  Grade  Location   Churn
1   24    1        A         1
2   25    1        A1        1
3   28    1        A2        1
4   31    2        A3        0

And this could get plotted like this in Logistic regression like this -

[A mocked image below]

How did these ID's usually gets plotted in the graph. What's the math behind a data like this getting plotted in a graph.

I understand the plane separating the points. But couldn't understand how the points got there in the first place.

This may be basic for experienced people. But it has been a question in my mind for a while. I tried to get any online materials but I couldn't or may be I don't know what to search for.

Also, does this plotting differs in any way for SVM and other models. I guess only the model splitting the points will differ.

Please give me leads in understanding this one.


texts: How do I integrate Github files to Orange for ML?

texts: I am currently trying to make a facial recognition workflow with Orange but since the widgets are still prototypes they are not all available in Orange. I found the Github files for the widgets but don't know how to install this so that I can see and use the widgets in Orange.


texts: Does Kernels always map data points to higher dimensions

texts:
When we use kernels in SVM to linearly sperate non linear data points by mapping it to 'another dimensions', does this suitable 'another dimensions' always be a higher dimension with respect to original dimension of the data points?

And is it true that we can always find a higher dimensions that can linearly seperate data points in a training set?


texts: Containing multicomponent data in rows or columns

texts:
I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of interest so that column contains one 3 character string (i.e. "ATG"). For other rows in the table that trimer column has 2 or 3 trimers of interest so the Trimer column has multiple strings in it (i.e. "ATT, CTG, GAT"). All trimers from one sequence should be thought to have equal weight and importance.

I know I cannot analyze my data in this format. I was wondering whether to split the Trimer column into 3 columns so if a sequence only has one trimer of interest the cells in the other two columns will remain blank. When doing my analysis my worry is that these columns will be seen as different features and will be weighted differently.

I was also thinking to make multiple row entries for the same DNA sequence. But the independent variable is influenced by the combination of trimers.

Any advice on how to change my table or create dummy variables is much appreciated. Thank you!


texts: Predict correct answer among ten answers for a given question

texts:
I have a case study to solve where I am given a dataset of questions and its answers, there are ten answers for a particular question.

It's a classification problem where correct answer is having class_label = 1 and all other nine answers having class_label = 0.

Which deep learning model would fit best for this type of case study and how should I proceed?


texts: Would like to know some Machine Learning applications for Civil Engineering

texts: I've thought about some, like predicting Concrete's strength by it's composition, but would like some ideas on the theme. One other thing I was thinking was traffic, how would I use machine learning to predict something related to streets and roads... Maybe something related to hydraulics on a building. I'm really beginning on machine learning so I'm not so good at knowing how can I use it. Although it's amazing. Thank's a lot guys.


texts: How would you visualize data that comes in the millions of records?

texts:
Developing a application where sometimes when I make a query I get millions or 100 of thousands of records in the results.

My problem is , when I get these huge result sets, how do I visualize (in charts such as bar, radar, line graphs, etc.) this data?

Do I just take a sample of the data and visualize that? Do I just go and plot million records? What is the best way to go about this ?


texts: Why transform embedding dimension in sin-cos positional encoding?

texts:
Positional encoding using sine-cosine functions is often used in transformer models.
Assume that $X \in R^{l\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the embedding size. This positional encoding layer encodes $X$’s position $P \in R^{l\times d}$ and outputs $P + X$
The position $P$ is a 2-D matrix, where $i$ refers to the order in the sentence, and $j$ refers to the position along the embedding vector dimension. In this way, each value in the origin sequence is then maintained using the equations below:
$${P_{i, 2j} = \sin \bigg( \frac{i}{10000^{2j/d}}} \bigg) $$
$${P_{i, 2j+1} = \cos \bigg( \frac{i}{10000^{2j/d}}} \bigg)$$
for $i = 0,..., l-1$ and $j=0,...[(d-1)/2]$
I understand the transormation across the time dimension $i$ but why do we need the transformation across the embedding size dimension $j$? Since we are adding the position, wouldn't sin-cos just on time dimension be sufficient to encode the position?
EDIT
Answer 1 -  Making the embedding vector independent from the "embedding size dimension" would lead to having the same value in all positions, and this would reduce the effective embedding dimensionality to 1.
I still don't understand how the embedding dimensionality will be reduced to 1 if the same positional vector is added. Say we have an input $X$ of zeros with 4 dimensions - $d_0, d_1, d_2, d_3$ and 3 time steps - $t_0, t_1, t_2$
$$
\begin{matrix}
& d_0 & d_1 & d_2 & d_3\\
t_0 & 0 & 0 & 0 & 0\\
t_1 & 0 & 0 & 0 & 0\\
t_2 & 0 & 0 & 0 & 0\\
\end{matrix}
$$
If $d_0$ and $d_2$ are the same vectors $[0, 0, 0]$, and the meaning of position i.e time step is the same, why do they need to have different positional vectors? Why can't $d_0$ and $d_2$ be the same after positional encoding if the input $d_0$ and $d_2$ are the same?
As for the embedding dimensionality reducing to 1, I don't see why that would happen. Isn't the embedding dimensionality dependent on the input matrix $X$. If I add constants to it, the dimensionality will not change, no?
I may be missing something more fundamental here and would like to know where am I going wrong.


texts: Hierarchical Clustering and Variable Selection

texts:
I am using  "Single linkage" hierarchical algorithm to cluster my data points with Gower Distance as my data have both qualitative and quantitative variables.

After applying this for the full model (all variables) I would like to start excluding those variables which are actually the not so important for my data. I was thinking of using principal component analysis (PCA) but I can't because my variables are a mixture of both categorical and continuous. Can someone suggest what is best method to select variables?

Finally I would like to use the Elbow Method to check exactly what is the optimal number of clusters?

Can someone help me with this logic?

I am using R-Studio for my analysis.


texts: How Does Cross-Entropy Work With Softmax Activation Function?

texts: I found online that the derivative of a cross-entropy activation function with a softmax activation is (output - expected), which had me very confused. If for example, the expected value is 1, and since the output is always less than 1 (softmax), doesn't it mean that the derivative is negative, and that if we train our weights with this gradient, the weights will decrease, and so the output will decrease in the next iteration, thus increasing the error? How does it work?


texts: Sklearn: How to adjust data set proportion during training, but not testing

texts:
I'm using sklearn/pandas/numpy.

I have a labeled data set, where the potential outcomes are either True or False. However, the data set has a much higher proportion of True entries. When running through classifiers with k-fold (n=5) cross validation, this appears to bias the classifier towards just saying True.

Using weights, I was able to adjust the sample data set I'm using to have a proportion closer to 1:1, like so (using a pandas csv):

results = csv[['result']]
weights = np.where(results.as_matrix() == True,0.25,1).ravel()
csv_sample = csv.sample(n=60000, weights=weights)

And the results are much more promising! However, I'm wondering if there's a way for me to do cross validation where the TRAINING set is adjusted in this manner, but the TEST set is closer to the actual proportion of data.


texts: How to code an SVM's equation including kernels?

texts:
So I need to code an SVM from bottom up in Python, and I cannot use stuff like libSVM or scikit-learn, for reasons of my own.

Referring to Andrew Ng's excellent notes on Support Vector Machines, I need to specifically implement the second equation on page 20,

$$\max_\alpha W(\alpha) = \sum_{i=1}^m\alpha_i - \frac 1 2 \sum_{i,j=1}^m(y^{(i)} \times y^{(j)} \times \alpha_i \times \alpha_j \times \langle x_i, x_j\rangle)$$

Subject to the given constraints, $\sum_{i=1}^m\alpha_i y_i = 0$, and $0 \le \alpha_i \le C$.

Does someone have the code for this in any language, or can explain the intuition behind coding this? Most SVM implementations I could find either stick to scikit-learn or neglect kernel functions altogether. While I am working in Python, I would try to understand code in any language.


texts: How to encode high dimensional dynamic categorical data?

texts:
Example: Facebook newsfeed ranking. This is typically done by relevance scoring each post using a regression model based on a number of features.
Typical features might include information from the profile of the user, e.g. numerical features such as age, signup timestamp as well as categorial features such as their interests, friends, pages followed, etc.
How would I best encode these categorical features for training a neural network?
One-hot encoding the friends list would result in a 2 billion dimensional vector(one for each facebook user). Not only is this way too large but the number of users changes constantly so I would have to constantly retrain the model whenever a new user signs up.
So, how do they do it?


texts: How to detect Anomalies in raw level data?

texts:
I have a dataset that is growing at about 30 million rows per day.

The data schema is follow:

day_hour_timestamp timestamp,
campaign_id        varchar,
creative_id        int,
uuid               varchar,
device             varchar,
country            varchar,
os                 varchar,
browser            varchar

Each record in the dataset is information about an advertisement impression.

For example, if I visited a particular advertisement at a website on 10/11/2017 13:30. I will be one datapoint in the dataset for 10/11/2017 hour 13:00 along with my device, country, os, browser, etc.

My question is how do detect anomalies within this dataset? Anomalies such as data not normal to the columns, for example if the I receive an OS that is NULL or browser is 'Apple'.

Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?

Since this dataset is pretty big would checking each data point be expensive?


texts: Tips for retraining convolutional neural networks given a drastically different loss surface

texts: For the image dataset I am working with, I need to use B&W version of images (otherwise, I would need to build a network to give false colors to a set of my images, since they have an overpowering color tint). I am attempting to retrain VGG's bottom layers (including the last convolution block) in order to perform this task, but am running into the problem that by rephrasing the space in which the input operates in (and thus also completely changing the loss surface) it is a difficult task to retrain these layers. I've experimented with learning rates from everywhere from 0.01 to 100 (using different decay schedules for each) with no success. Does anyone have any tips with retraining networks for significantly different loss surfaces? Should I ditch my efforts and attempt to train a new network end-to-end (my dataset isn't too large, so this is quite prone to overfitting if I am to use deep learning)? Are there networks built for B&W images that I could instead retrain?


texts: How to find from which row and column the value belong?

texts:
Suppose I created the below data frame

data = {'Height_1': [4.3,6.7,5.4,6.2],
        'Height_2': [5.1, 6.9, 5.1, 5.2],
        'Height_3': [4.9,6.2,6.5,6.4]}

df = pd.DataFrame(data)

Suppose someone comes and asks me

Find the row and column of height 6.9 ?
Find in how many rows and columns height 6.2 is present?

Please help me with what will be the code for this?


texts: How to forecast timeseries based on different events?

texts:
I have a few IoT sensors around my house that over time store some events with timestamps. Each sensor has a unique type e.g. ‘front’ or ‘back’. Let’s call this set X.

Now I have one sensor which value is dependent on X. I have a value for approximately every 5 minutes over the last year. Let’s call this set Y.

Now my goal is to create a model that given a current point and and a few hours of history (X events have influence on Y for longer time), it gives me the forecast of Y for the next 3 hours.

How would one go and create such a model? Any common architectures for such inputs and outputs? I’d like to setup a neural network to allow to find patterns between different IoT sensors.


texts: why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler?

texts:
while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ?

if we should not fit the scaler on x_test then why we need to apply the transformation alone on x_test ?

from sklearn.preprocessing import MinMaxScaler
Scaler=MinMaxScaler()
Scaler.fit(X_train)
x_train=Scaler.transform(x_train)
x_test=Scaler.transform(x_test)


texts: Using simulations to train ML algorithms

texts:
Possibly similar question: Is it ok to collect data using algorithm to train another?

I have a model that accurately describes an underlying physical, complex, system. The model is basically a set of ODEs based on the physics of the system, validated against measurements. When a system perturbation occurs, I can run thousands of simulations to assess if the new system status is secure or not. This is basically a classification procedure (yes/no).

This procedure is very time consuming and has to be performed in real-time (thus, requiring huge computational resources). There are thousands of possible perturbations and infinite number of initial points. The same perturbation from 1 initial point can lead to stable system, while from another, to an unstable.

My question is:

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? What are the considerations when using simulated data to train an algorithm that will then be used online with real data (expect from the obvious that the simulation needs to be very very accurate)? Any references to such examples?

I apologise if this is a basic question. I am new to data science techniques with a more physics/engineering background.


texts:
How can we detect the existence of outliers using mean and median?

Is it really possible to detect the existence of outliers in a set of data from their feature-wise mean and median?

Suppose, I have a data set with eight features in my hand.

I have juxtaposed their means and medians row by row.

            f1       f2        f3        f4        f5        f6        f7        f8
mean      2.5000    0.1868    0.0148    0.2105    0.2088   79.6583    1.0604    0.0091
median    2.5000    0.1826    0.0001    0.0002    0.0000   -0.0000    0.0000   -0.0000

Can we tell anything about the existence of outliers from these information?

Edit:

  How are the data distributed?

Data set is considered to be normally distributed.

  Do they all have the same mean and variance?

I don't know.


texts: Scala vs Java if you're NOT going to use Spark?

texts:
I'm facing some indecision when choosing how to allocate my scarce learning time for the next few months between Scala and Java.

I would like help objectively understanding the practical tradeoffs.

The reason I am interested in Java is that I think some of my production, frequently refreshed, forecasts and analyses at work would run much faster in Java (compared to R or Python) and by becoming more proficient in Java I would enable myself to work on interesting side projects, such as non-Data-Science apps I'd like to develop. Currently I've taken a couple of Java courses, but I need much more education and practice to master it.

The reason I started considering learning Scala is very similar -- I figured that the statistical/ML orientation would make it good for my work as a Data Scientist, and since it is based on Java I would be getting practice at work that helps me with my side-interest in Java, even though there are a few major differences such as functional vs. imperative and traits vs. interfaces.

It seems like a lot of the advantages of Scala revolve around integration with Spark. I am thinking that this should be the tipping point for my decision, because my team is not currently using Spark and I don't have a good enough reason to request it. However, I thought I should ask here so that I don't waste too much time if Scala is still a better choice.

For the purposes of this question please ignore alternatives such as Python, R, Julia, etc (I've eliminated those from consideration for other reasons, such as already being sufficiently familiar with them for my use cases).


texts: How to find what events cluster together?

texts:
I have a data set that looks at a five-year timespan of peoples' lives and indicates if specific events have occurred (Divorce, Birth of Child, Health Shock, etc.).

# Sample data in R
n <- 100
set.seed(1337)
exampleFrame <- as.data.frame(cbind(1:n,replicate(7, sample(0:1,n, TRUE))))
colnames(exampleFrame) <- c('Person',paste(rep("Event",7),1:7, sep = ''))

First thing I did was simply show a correlation matrix.

cor(exampleFrame)

And it gives an idea of what items follow each other, so next I ran a logistic regression to see how events are likely to improve odds of occurring together.

glm(Event1 ~ . - Person,family = binomial(link = 'logit'), data = exampleFrame)

And this is interesting, but I feel like I'm missing something more compelling. Am I chasing a phantom? Does anyone know a better way to demonstrate which life events occur together in this five-year span?


texts:
I want to create a simple object detection tool. So basically an image will be provided to the tool and from that, it has to detect the number of objects.

For example

An image of a dining table which has certain items present on it such as plates, cups, forks, spoons, bottles etc.

The tool has to only identify the number of objects irrespective of the type of object. After identifying it should return the position of the object with its size so that I can draw a border over it.

I don't want to use any library or API present such as Tenser Flow, OpenCV etc.

If the process is very difficult to be created without using an API then the number of/type of objects which it will count as an object can also be limited but since this project will be for my educational/learning purpose can anyone help me understand the logic using which this can be achieved? For eg, it may ignore a napkin present in the table to be counted as an object.


texts: Where to get models with weights instead of only weights? What's the purpose of .h5 files?

texts:
I have downloaded .h5 files from qubvel/resnet and qubvel/efficientnet. I was trying to use some models as a backbone for my model but I'm getting the following error:
ValueError: No model found in the config file.

As explained here this is because the .h5 file contains only weights, not a model.
So those .h5 files are only weights. What's the purpose of having only weights without architecture?
I was trying to do following code:
resnet18_path_to_file = "models/resnet18.h5"
resnet18 = tf.keras.models.load_model(resnet18_path_to_file)
resnet18.compile()

inputs = resnet18.input
outputs = resnet18.layers[-2].output

return tf.keras.models.Model(inputs=inputs, outputs=outputs, name="custom_resnet18")


texts: Is it a good thing if my model overfits in TensorFlow?

texts:
Very conceptual question:

I have a TensorFlow model that works well. I have isolated about 70 features worth of data and when training, my validation accuracy stays around 32% while my training accuracy is up to 89%! However, when I use only 24 of those 70 features, I get a validation accuracy of 38% and a training accuracy of 53%. I am using regularization, heavy dropout, and early stopping.

Is this a good problem to have? Should I keep the 70 features while boosting overfitting parameters? What should I do about this? Maybe it's a good idea to keep using only the 24 best features?

Is it generally O.K. to shed some features by just discarding them? Definitely some emotional attachment to my features here, but is it common for an ML process to simply discard these features as opposed to working with them to achieve higher accuracy?


texts: Creating a metric based on some features

texts:
I want to create a new metric based on some features but dont know how to start. I basically want to create a "job satisfaction level" metric based on some features. The features could be work hours, shift, If working on weekend and so on. I dont know how to start. In ideal world, I want to comp up with weights for each of these features and compute a final value and then put the final value in a job satisfaction level bucket. I then want to use this metric in my training model. Is there any methodology to do so? Lets assume I have different warehouses with different values for those features and I want to compute a "job content" or "job satisfaction" metric based on the features I mentioned above for all of these locations. Then I want to use this new computed metric with my other features for an employee resignation prediction. Any help is appreciated.

Thanks


texts: MinMaxScaler when LSTM predictions fall outside of training range?

texts:
I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set may have values between 1-200, but in the most recent time steps, the values are toward the high end of that range.

How can I handle future time-steps that should be predicted to land above 200 (since activation function outputs values between 0-1)? I guess my question is, is there a way to manually (or through some “rule of thumb”) specify a maximum value for the training set? So in this case, would it be possible to use MinMaxScaler to use 0-500 as the min-max values to transform to 0-1, that way values in my test set or predictions can be inverse_transformed accordingly? Is this an out-of-the-box scikit-learn feature? What’s the most efficient way to implement this, especially if this has to be done independently for multiple features?


texts: What does the training time for a Neural Network include?

texts:
I recently developed a DNN model and I want to know what exactly is training time and what all steps are included in it? For ex I carried out the following steps
1) Determined best Network Architecture by 5 fold CV ( using Back-propagation)
2) Once I got the Best architecture I used unsupervised learning to get initial weights using unlabled data
3) I applied back-propagation to fine tune the network with initial weights given by step 2 (It ran for 500 epochs)

What is the training time in this procedure? Does it include all the steps or only steps 2 or 3? Or in general terms does the training time include time to run the code for the best architecture or it also includes the time taken to determine the best network architecture?


texts: I am seeing, in all the notebooks that I found, that Max Pooling is never used in the first layer of a CNN. Why this? Is it a convention among data scientist to do not use max pooling in the first layer? Or is it an error to use in the first layer?


texts: How to train a supervised sequence classifier like CRF, if we have to extract start date and end date from a user query in python

texts:
I have to build a chatbot, in python in which a user can apply for a leave. I want to extract start date and end date from a users query. I did some research on couple of algorithms and found CRF Entity Extractor as the best one. I now want to see a similar implemented solution in python, which I can use it as a reference. I would like to see an end to end solution right from training dataset to predicting start and end dates from query. Kindly help.

For example:

1.query: "I want to take leave from 2nd April to 5th April."
Predicted dates: "02-04-2020" and "05-04-2020"

1.query: "I want to take leave on next Monday"
Predicted dates: "09-03-2020"


texts: Yolo ground truth to boudning box matching mechanism

texts: How does Yolo algorithm query the ground truth file and match it to the prediction, do I need to annotate the objects in a certain order, or is there a built in mechanism to match a prediction to the corresponding ground truth (if so what is that mechanism)


texts: Discriminator of a Conditional GAN with continuous labels

texts:
OK, let's say we have well-labeled images with non-discrete labels such as brightness or size or something and we want to generate images based on it.  If it were done with a discrete label it could be done like:

def forward(self, inputs, label):
    self.batch = inputs.size(0)
    h = self.res1(inputs)
    h = self.attn(h)

        ...

    h = self.res5(h)
    h = torch.sum((F.leaky_relu(h,0.2)).view(self.batch,-1,4*4), dim=2)
    outputs = self.fc(h)

    if label is not None:
        embed = self.embedding(label)
        outputs += torch.sum(embed*h,dim=1,keepdim=True)

The embedding can be made to match any shape to be added to a hidden layer and summing the embeddings to the latent, which forces the discriminator to recognize the class that's it's discriminating to make a better judgment on the discrimination.  That's cool and all, but this method uses embeddings which is the discrete method.  How about a continuous label?  I can't really find a method for doing this except for some semisupervised methods whereas I have exact labels.  Is anyone able to help me?


texts: What limits the long-term dependencies learned by an LSTM?

texts: LSTMs and GRUs are meant to learn longer-term dependencies than basic RNN architectures that do not have gated architectures. However, both LSTMs and GRUs are still typically trained with truncated backpropagation through time. Naively, it seems that the choice of truncation length will limit how many time-steps the network can "remember". Is this the case, or can an LSTM trained with BPTT truncated to n steps learn dependencies from more than n steps in the past?


texts: Autoregressive Moving Average ARMA with statsmodels

texts:
Can I please get direction on what is wrong in the code? All forecasts provide output except the ones listed bellow.

so basically doesn't pick ['UserDefinedData4'] and ['ItemId'] separate ... only when both are  in group_level ...

this link is with the program:
program location

elif forecast_type == 4:
    # forecast weekly order quantity at item level with the dummy variable 'Quarter' using AR(8)
    mape = forecast(dataset, 'W', ['ItemId'], 'Quarter', 8, output_file)
elif forecast_type == 5:
    # forecast weekly order quantity at item level with the dummy variable 'Month' using AR(8)
    mape = forecast(dataset, 'W', ['ItemId'], 'Month', 8, output_file)
elif forecast_type == 6:
    # forecast monthly order quantity at item level with the dummy variable 'Quarter' using AR(2)
    mape = forecast(dataset, 'M', ['ItemId'], 'Quarter', 2, output_file)
elif forecast_type == 7:
    # forecast weekly order quantity at customer level with the dummy variable 'Month' using AR(8)
    mape = forecast(dataset, 'W', ['UserDefinedData4'], 'Month', 8, output_file)
elif forecast_type == 8:
    # forecast weekly order quantity at customer level with the dummy variable 'Quarter' using AR(8)
    mape = forecast(dataset, 'W', ['UserDefinedData4'], 'Quarter', 8, output_file)
elif forecast_type == 9:
    # forecast monthly order quantity at customer level with the dummy variable 'Quarter' using AR(2)
    mape = forecast(dataset, 'M', ['UserDefinedData4'], 'Quarter', 2, output_file)

Thank you for help.


texts: How to use a dataset with only one category of data

texts:
I am performing a classification task, to try to detect an object. A picture of the environment is taken, candidates are generated of this possible object using vision algorithms, and once isolated, these candidates will be passed through a CNN for the final decision on whether the object has been detected or not. I am attempting to use transfer learning on InceptionV3 but am having difficulty training it, as I only have one set/class of images.
The dilemma is that I only have one class of data and when I pass it through the network, I get a 100% accuracy (because there is nothing to compare it to). How should I overcome this? Should I find more categories online to add to my dataset? What should these categories be?
Just to clarify, as an example, I have class "cat".
Not "cat" and "dog".
Not "cat" and "no cat".
Just "cat". That is what my dataset consists of at the moment.


texts: Multiple (>1500) timelines- which model to use?

texts:
I'm new to Data Science (<6 months). I'm currently working with American economic data from the BLS.
I only have quarterly numbers for about 20 years, so my timeline data isn't very wide- but I do have timelines for about 1,500 different cities/states. Most of the research on time-series data focuses on a single very wide timeline and trying to predict future data points. I'm instead trying to predict how many quarters it will take for a dip to recover.
Edit- some clarity on what I'm trying to do here:
Here is a small sample of the type of data I'm working with:
idx Q12000      Q22000      Q32000      Q42000      Q12001      Q22001      Q32001     Q42001      Q12002      Q22002      Q32002      Q42002

0   4085280.0   4114911.0   4108089.0   4111713.0   4055699.0   4076430.0   4043219.0  4039370.0   4201158.0   4243119.0   4231823.0   4254681.0
1   21226.0     21566.0     21804.0     22072.0     21924.0     23232.0     22748.0    22258.0     22614.0     22204.0     22500.0     22660.0
2   96400.0     102000.0    98604.0     97086.0     96354.0     103054.0    97824.0    95958.0     115938.0    123064.0    120406.0    120648.0
3   23820.0     24116.0     24186.0     23726.0     23504.0     23574.0     23162.0    23078.0     22306.0     22334.0     22152.0     22080.0
4   7838.0      7906.0      7714.0      7676.0      7480.0      7520.0      7102.0     6722.0      8324.0      8166.0      8208.0      8326.0

Is this appropriate for an RNN or CNN? I've not seen any examples of them being used on this type of problem- but I'm also not sure what this type of problem is called, so maybe my searching isn't hitting the right keywords.
Edit: Some clarity on what I'm trying to do:
I'm looking at economic data for the past two recessions. Specifically, the number of jobs in the economy (see the third chart at this blog post calculatedriskblog.com/2012/05/…). But I have specific job number breakdowns by industry (as defined by the BLS)- so I have one time-series for each industry. For my target, I'm trying to predict how long it will take for an industry to achieve its peak job numbers before a recession. I know that's confusing, so here is a chart that hopefully will make it more clear: https://www.dropbox.com/s/jyztzjjcslwrkye/Screenshot%20from%202020-12-30%2018-12-45.png?dl=0
My target is the distance between the peak and the recovery.


texts:
I am trying to develop a custom MLP for MNIST Dataset with 2 hidden fully connected layers
Mnist: 28281
FC1: input 28*28, output 512
FC2: input 512, output 128
Classifier: FC: input 128, output 10
I am adding batch normalization,relu,BN and dropout layer
Code:
model = Sequential()

model.add(Dense(512, activation = "relu", input_shape = (input_dim,), kernel_initializer = RandomNormal(mean = 0.0, stddev = 0.039, seed = None)))
model.add(BatchNormalization())
model.add(Dropout(0,5))

model.add(Dense(128, activation = "relu", input_shape = (input_dim,), kernel_initializer = RandomNormal(mean = 0.0, stddev = 0.039, seed = None)))
model.add(BatchNormalization())
model.add(Dropout(0.5))

model.add(Dense(10, activation = "relu", input_shape = (input_dim,), kernel_initializer = RandomNormal(mean = 0.0, stddev = 0.039, seed = None)))
model.add(BatchNormalization())
model.add(Dropout(0.5))

model.add(Dense(output_dim, activation = "softmax"))

model.summary()

Is something wrong by which I am defining the dense layers?


texts: Using a trained classifier in an Android app

texts:
As the title suggests, I'm attempting to train some different classifiers into an android app. The main question I have is how to represent the different models in a neat and effective way, from python to Java (Android Studio).

Background:
I will attempt to connect 3 bluetooth bio-marker sensors through an app in order to perform a medical classification on heart disease risk groups. I'm fairly experienced with the machine learning packages in Python, mainly Scikit learn. I want to try different classifiers (I'm thinking of Gaussian Process, neural net and support vector machine) to evaluate which to implement. When one is chosen I want to transfer the trained model into Android studio.

Is there a good way to represent models in scikit learn? What about non-parametric models such as GP?


texts: how to create multiple plot from a panda Dataframe

texts:
I want to plot multiple plots.
The data is stored in a pandas dataframe and each row should be a seperate plot.
Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be plotted.

import pandas as pd
import numpy as np

df = pd.read_csv('./Result_set_edited.csv')
df = df.drop("ZRD_ID", axis=1).drop("TAG", axis=1)

x = df.iloc[[0]]
print(df.head())

returns:

     W01     W02     W03    W04    W05  ...     W20    W21    W22    W23    W24
0  72616  156076  141025  72629  72631  ...       0      0      0      0      0
1  67114  171650  139920  67291  67292  ...  172924  93511  72445  72445  72445
2  66893  161919  134041  66913  66911  ...  166244  86672  67114  67120  67124
3  66603  171297  134227  66615  66631  ...  166078  86622  66871  66877  66879
4  66759  167198  133523  67126  67128  ...  163999  74525  66562  66568  66574

To start easier, since I am really new to this I thought of plotting the first line alone first.

since the columns are named 'W01', 'W02', ... , 'W24' I thought I coould use them as labels for the x-axis.
I just didn't find a way to do so, since its the header of the df I guess.
So i created a new array and tried to plot it with the first row of my Dataframe:

y = np.arange(0,24,1)
y.reshape(1,24)
print(y)
print(df.iloc[[0]].values)
plt.plot(y, x)
plt.show()

when trying to plot my values I get the following Error:

  ValueError: x and y must have same first dimension, but have shapes
  (24,) and (1, 24)

thanksfor the help on how to fix the Error for plotting the first line.

PS: I would appreciate some hints on how to improve my question, since it is my first one.

Cheers


texts: Visualization for comparing the results of two different models

texts: I have two different mathematical models I use to predict the dollar amount in claims a group will incur over the course of a year. I want to visually juxtapose how accurate those two models are on historical data (i.e. I know what actual claims for each group are) with group-level granularity. I haven't done much with data visualization in the past, so I was wondering if there are any pre-existing data-visualization paradigms that this kind of thing might fit into. Thanks.


texts: Softmax vs Sigmoid in RBM/Auto Encoder final layer

texts:
I'm creating a deep network with 3 hidden layers for classification of the MNIST dataset. The network architecture is something like:

784 input nodes -> 150 hidden nodes (layer 1) -> 125 hidden nodes (layer 2) -> 100 hidden nodes (layer 3) -> 10 output nodes (1 for each possible digit)

For the output layer, I need to decide on an activation function. I've previously used the sigmoid activation function, but it seems many people are using softmax in this particular implementation. I'd like to know which is best suited for the task and why.

Many thanks


texts: Can I run Orange widgets from normal Python scripts?

texts:
I'd like to do normal machine learning with Python and Sklearn, but occasionally use the nice graphical widgets or Orange for presentation.

Is this possible?


texts: replace values based on Number of duplicate rows are occured

texts:
I have a dataframe ,that looks like this
       site  Active
0     deals  Active
1     deals  Active
2     deals  Active
3  discount  Active
4  discount  Active

i don't want to drop the duplicate items, but i want to change the Active columns value based on Site column,for example Active  has to change inactive based on  duplicate item in site column,last duplicate item has to Active, other than that Inactive
Expected
       site    Active
0     deals  InActive
1     deals  InActive
2     deals    Active
3  discount  InActive
4  discount    Active


texts: Statistical method to find the value which preserves the most information inside "most" of data points. (resize images to a common height)

texts:
So I have this data of around 88K images and I found out some interesting properties for my images.
print(np.median(width),np.mean(width),scipy.stats.mode(width))
print(np.median(height),np.mean(height),scipy.stats.mode(height))

>>
1280.0 1266.8129869839922 ModeResult(mode=array([1280]), count=array([84584]))
377.0 438.3157888861602 ModeResult(mode=array([125]), count=array([3113]))

So I am resizing all of the images to width of 1280 because it'll preserve the images for most of the images when scaling up or down as all of three are same.
But what I want to know is that what should I do for the height so that it preserves most of the information. Or to rephrase, to which height I should resize my image so that I can preserve most of the information. In, my opinion, Scaling down is better than scaling up.
for q in [0.35,0.55,0.75,0.95,] :
    print(np.quantile(height,q))
>> 274.0
414.0
562.0
1057.0

Is there any statistical method so that I can find the appropriate range?
My height data is positively skewed and looks like:

Black line is the scipy.stats.norm


texts: How do you distinguish between conversational text and possible news article?

texts:
Context

When you receive messages in group chats, how do you detect if that message belongs to conversational dialogue or if it is a 'news' article (could be fake or real) that they are sharing?

Examples

Conversational dialogue: "Does anyone wanna have dinner today? I am pretty free this evening and I don't have dinner at home. Please let me know by 8pm!!"

'News' article: "New Japan’s growth has been helped by YouTube, which has made New Japan’s matches more accessible to an audience outside of Asia, said Dave Meltzer, publisher of the Wrestling Observer Newsletter, which has followed the sport since 1983. Capitalizing on this rise, New Japan launched an online streaming service — similar to the W.W.E. Network — in December 2014."

Question

Would you use rule-based matching or a classifier for this problem? (Assuming you already have a classifier for detecting if the news article is fake or real)


texts: If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

texts:
Right now I am trying to better understand how Bayesian modeling works with just the basics.  I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling use something called the "Gibbs sampling algorithm", which is a Markov Chain Monte Carlo Method algorithm.

I know that, if I am going to do anything with Markov Chains, then I have to test a data or parameter violates the assumption of memoryless.  However, I am uncertain what exactly do I need to test for memorylessness.  Is it a parameter, the dataset, the response variable, a particular residual or error distribution.... what is it that has to be memoryless if you use Gibbs Sampling?

(P.P.S.) I did find a paper on that test's memoryless for Bayesian models but its really confusing.  I am a little familiar with how to test memorylessness for basic data, but I am lost on more complex algorithms like Gibbs Sampling.


texts: (Pre)training an Embedding Layer via prediction (as an alternative to similarity) - does that makes sense?

texts:
Similarity is the go-to way to train embeddings - use a similarity matrix (eg dot product) between the embeddings of two inputs, train to increase it for connected inputs and decrease it for inputs that are not connected.
Is it possible - and does it makes sense from a practical perspective - to train an embedding layer by connecting it to a fully connected layer (and a subsequent softmax layer) and directly predicting a target value?
Would this allow for the creation of a meaningful embedding space that could be used for further applications (eg., use it to transform data to an embedded vector as part of the input for another neural network).
For example:
class emb_pretrain_module(nn.Module):
    def __init__(self, emb_layer, output_size):
        super(emb_pretrain_module, self).__init__()

        self.emb_layer = emb_layer
        self.fc1 = nn.Linear(emb_layer.embedding_dim , output_size )
        self.fc2 = nn.Linear(output_size , output_size)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        out = self.emb_layer(x)
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.softmax(out)
        return out


texts: How to train a model using a daunting huge training dataset

texts:
I have an extremely huge dataset and I'm wondering me how could be the right way to set an experiment to use this data to train a model.
I understand that I can use data-reduction to, for instance, drop out some variables. In despite data-reduction can actually reduces the data amount, as I can see this technique is intended to improve the model training effectiveness, not to deal with the practical issues that comes out from the data amount.
One of my ideas is to suffle the whole data first and then split the data into 'small' chunks. Once I have, let's say' $N$ chuncks, I can train the same model using each chunck as follows:
initialize(M);
for(n in N) {
  D = load_chunck(n);
  M = train(M, D);
}

Although this approach can be effective to fit the experiment to computational resources at hand, I'm afraid that training the model this way can affect the model's quality by including bias from the latter chuncks. In addition, N is now a new hyperparameter to be set.
Another alternative I can see is by using statistical sampling:
D = retrieve_sampling(sampling_size);
if (D is good)
   M = train(D);

I'm wondering me if there are other ways to do it then the ones I've cited here.


texts: Missing population values in census data

texts:
I have population data from Census.gov:

Total US population by age by year from 1940 through 2010

Depending on the range of decades, the data is missing discrete population values for ages greater than a certain age.  Instead an aggregate amount is provided that represents all ages greater than the cutoff.

Specifically it follows this pattern:

1940 to 1979:  Discrete data from 0 to 84 and aggregate for ages 85
and greater
1980 to 1999:  Discrete data from 0 to 99 and aggregate for ages 100 and greater
2000 to 2010:  Discrete data from 0 to 84 and aggregate for ages 85 and greater

The desired outcome is to have discrete data points for each age and year from 0-99 and then an aggregated lump sum figure for ages 100 and greater.

Therefore I want to input the missing discrete population values for ages 85 to 100 for years 1940 through 1979 and years 2000 through 2010.

And I want to use the actual discrete population values for ages 85 to 100 for years 1980 through 1989 to achieve that outcome.

Some Observations:

The pattern of missing values is MNAR (Missing Not At Random) - these were systematically omitted but the aggregate value representing the missing detail is provided
Population data for this time frame is deterministic:  Population levels rise linearly each year;  The duty cycle of a human body is finite and constraints and limits are well known.

Looking at the data we can see that each of the three subsets of years have very similar patterns.  More variation in younger ages and variation flattens out for ages greater than 60

Then if we focus on the years 1980 through 1989 we can fit a nice curve for ages 0 through 100 with a Multiple-R-Squared of .979.

Then if we narrow the focus to ages 60 to 100 and even narrower to ages 85 to 100 the Multiple-R-Squared increases to .9996.

Now if we flip our focus and look at the increasing levels of population we can observe that these relationships are linear.  Population rises at a steady rate year over year.

Total population 1940 through 2010:

Ages 85 through 100 for years 1980 through 1999.  Each age is linear.  Each age has a slightly lower rate of increase (lesser slope).

My question

This is where I could use some guidance to move forward:

When imputing discrete missing population values by age and year, how do I combine the fitted curve that models changes in population when age increases with the linear regression that models changes in population year over year?

Does one or more documented methods naturally apply to the problem as I have described it?
For example:  KNN, PCA, BPCA, Mean, MICE, other?

If there is recommended method can you point me to available R or Python packages and documentation that describes the mechanics of applying a given approach?


texts:
To design a neural network based predictor for f(x) = sin(x) I designed the following network. Input ----> Dense(num_neurons=1) ---> Relu() ----> Dense(num_neurons=100) ---> Relu() ---> Dense(num_neurons=1) ----> Relu()--> Output.

The choice of loss was L2 and optimizer was standard gradient descent. The predictor was trained on X_train = numpy.arange(0.0, 314.1, 0.1) and Y_train = numpy.sin(X_train). It is subsequently tested on X_test= numpy.arange(-10.0, 10.0, 0.001) and Y_test = numpy.sin(X_test) The predictor however performs badly on the test data. What could have gone wrong?

The training data is too small to train the network. Adding more data especially when X < 0

Relu activations used in the network are wrong. Changing the activation in the output node from Relu to Linear should help network learn the function better.

Network is not deep enough. Adding more hidden units/layers to the network will help the network generalize

L2 loss is not a good metric to measure loss when regressing for sin(x)

Gradient descent is not the best optimizer for this problem. Adam optimizer would be a better option.

This type of neural network architecture can fundamentally not learn a function like sin(x).


texts:
I have serveral text files.

These files has been analysed through some analytical tool and provided main features

There each feature extracted has one repetition

I know to use predictive modeling in NLP using word2vec, Bag-of-words, Glove, etc.

these method depend on the word repetition

Example

Text 1 : in Melbourne Supermarket the month of January customers bought more of beef minced meat, Milk law fat 2 liters. also customers were buying cream cheese 250 grams. there was high demand on unsalted butter 200 gram ...

Text 2 : things were a bit different this month for sydney Supermarket customers bought more of orange juice 1 ltr, apple juice 1 ltr and mango juice 1 lts, also Milk law fat 2 liters was on high demand. from confectionary kitkat box 12 pack and Snickers 12 pack were hogh in demand...

The analytics tool gives me these results

Text1 Analysis : Product 3038, Product 6783, Product 5799, , Product 4836, , Product 4882
Text2 Analysis : Product 9398, Product 9391, Product 8349, , Product 8118, , Product 6783

and the predictive results are

Products recommended to be promoted

Text 1 : Recommend Chedder cheese 200 gram, chicken breast 1kg
Text 2 : Recommend Chocolate Milk 1 ltr, Smoothy Apple and Mango 1 ltr.

So how can I approach this problem?

Shall I use word2vec, Bag-of-words, Glove?


texts: Can we make two separate models vs one for classification?

texts:
Suppose I have a binary classification problem and my data is imbalanced, I can build a classification model using any of the algorithms and use an oversampling or undersampling technique to handle the class imbalance.

What If I were to make two separate models independently, each model trained only for one class of data. Will this be a right approach?


texts:
Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), which allow to reduce the dimension of a data set to plot it in 2 dimension. It seems fast and efficient.

In the main paper authors even provide some beautifull plots. As for exemple :

I have been unable to reproduce this kind of graphics in R with ggplot2 (computer crashing because of too many points, overlaping points...). How would one go about building a similar plot ?


texts: Reinforcement Learning model always gives different output

texts:
I am trying to build a reinforcement learning model for hardware capacity optimisation. The state of the model would input like CPU capacity utilisation, memory utilisation. The model is supposed to predict what should be the CPU, memory etc I need to provision for my environment. The model uses DQN at its core and reward mechanism is based on the current capacity used.
The challenge I have is that every time I run with same input state and reward mechanism, I am getting different combination of hardware to be provisioned.
Is it possible that RL might give different output for same set of inputs and reward?


texts:
I am looking for a framework or chart that classifies all the different types of data analysis/machine learning. I want to use this classification to organize my knowledge/areas to learn.

Examples of what I mean by types of data analysis/ML include:
 - Geographical Maps
 - Image Recognition (CV, OCR...)
 - Signal Processing
 - Text/Speech Comprehension

Is there a chart/framework classifying all those domains ?


texts: Ways to reconstruct shuffled pixels of a video file?

texts:
Suppose that you have a video file which pixel order has been shuffled once. That is, a random order have been defined once and applied to all frames.

Does it exist some known approach for retrieving the initial order of pixels?

I have some ideas around retrieving the initial topology by placing pixels which values are correlated in space and time closer together. I wonder if this has been studied and if efficient algorithms were published.

Also this problem can be thought of as a way to project to a 2D matrix a set of values varying in time in order to be able to apply computer vision techniques (like CNN), with the assumption that these values are indeed somehow correlated.


texts:
Suppose I want (the machine) to learn the ">" function, and my training set is a collection of pairs of numbers $(n_1, n_2)$ with the output True if $n_1 > n_2$ and False otherwise. Which method is the most likely to work (actually, my secret goal is a form of the Netflix challenge - I have a set of $n$-tuples of real and boolean values, and I want to rank them based on some training rank data).

This has surely been studied a lot...


texts: Logistic regression with unbalanced data, scoring based only on rare class

texts:
I have a dataset off app. 600.000 data points in which 0.2% (1.200 samples) is labelled as signifying a rare event. I want to use logistic regression to help me predict this rare event, but even when I apply weighting, the classification accuracy is poor.
I know that I can rebalance the dataset, but the problem with that is that lots of other weird stuff is going on that signifies various events but which is not indicative of the exact type of event I’m trying to predict. Therefore, rebalancing would itself be a large and complicated task in trying to get representative weirdness into the cropped dataset.
I have found that I get great predictive performance by using a home-made logistic regression classifier that scores based on the error measure below, and then using scipy.optimimize.minimize with method='Powell' to tune the parameters to minimize this measure:
e = 1 - tp/(tp+fp+fn)

for the rare class only, ignoring true negatives.
This lets me include the full dataset (and ignore weights and rebalancing considerations), and it makes it clear in the scoring when the classifier starts erroneously including other types of events that shouldn’t be caught by this classifier (i.e. increasing number of false positives).
The problem is that my home-grown classifier/trainer is naturally much slower to train than the sklearn version, and I’d like to build on mature optimized packages instead. So my question is: Is there a Python implementation of a logistic regression classifier that lets me perform fitting based only on the rare class as described above? It seems that setting the rare class weight arbitrarily high does not make it perform like my home-grown method that explicitly ignores the common class.


texts: In a Time Series Problem, is it possible to forecast quantities by learning the patterns of other items? What are my options?

texts:
Suppose I own a store that sells a variety of apples and I have the following stats each month.

Report Date
Type of Apple (TA)
Quantity Available(QA)
Quantity Sold in the Past 30 days(QS30)
Quantity Shipping In (QSI)
Quantity Needed to Order(QN)

Lets make the following assumptions/givens:

There are three types of apples: red apples, green apples and yellow apples.
T(1) denotes the first month and T(60) denotes the 60th month.
QA @ T(i + 1) = QA@T(i) + QSI@T(i) - QS30@T(i+1)

I can provide all the data from T1-60 for each apple.

I can also provide all the data for T61 besides QN for each apple.

My goal is predict QN at T61, or the 61st month for each apple.

If I am only concerned about Red Apples, I can just use ARIMAX with data pertaining to only red apples or VAR right?

But what if I suddenly introduce a new apple type such as orange apples and only have a history from T1-4?

Given I need to use T1-4 for the orange apples...
Is it possible to use other data from the red, green and yellow to aid in properly calculating a QN value for orange apples at T5?


texts: I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me.  Could someone explain the difference to me using examples?


texts: What is the minimum size of the test set?

texts:
The mean of a population of binary values can be sampled with about 1000 samples at 95% confidence, and 3000 samples at 99% confidence.

Assuming a binary classification problem, why is the 80/20% rule always used, and not the fact that with a few thousand samples the mean accuracy can be estimated with > 95% confidence?


texts: Training vs test data set for supervised learning in real life scenario

texts:
In the tutorials, I have noticed only similar data has been used with models training and prediction.

I was wondering how cases where you can't find training data that is similar to your final use case(test data)?

What happens if I am building a recommendation system for house rentals but the closest training data I can find is for movies. And it has fewer features than I would have in the house scenario.

How common are scenarios like this and how are they generally handled?


texts: Need help regarding my thesis project related to Data Analysis

texts:
I am working on a project with a company that manufactures Injection molding machines. I am to perform data analysis on some selected parameters as a part of my thesis to tell the company if from these parameters some ideas can be generated to suggest any changes in the health of the machine.

I have used Pandas for data cleaning and plotted the graphs using Matplotlib. I collected the data daily from the machine for two different input materials and have a dataset for each material for 13 days. Now the problem I am facing is how to analyze the data.

Sure I can plot various kinds of plot, bar plots, scatter plots, histograms, box-plots in matplotlib, but since, I am writing my Master's thesis, I am unsure if just plotting and discussing the data would suffice my degree without any mathematical calculations?

Another idea I got from my Math Prof was to use DOE, but I don't seem to understand what and how I could use DOE in my case. I was thinking to apply a regression model from scikit-learn and maybe tell the company during my presentation about the possibility of some changes in the future in some of the monitored parameters.

So far, I have seen no significant changes in the values. Or am I missing something? Or am I doing the analysis in the wrong direction? I am really confused. I am new to this field. Any suggestions and help would be greatly appreciated.

Thanks in advance.


texts:
$$X \mid(Y=k) \sim N\left(\mu_{k}, \Sigma\right) \text { for } k=1,0.$$
I have found, $$Err(C^*(X)) =  \int_{-\infty}^{x^*} P(X=x|Y=1) \,dx$$ where $x^* = \frac{\mu_1+\mu_0}2$. But I need to express $Err(C^*(X))$ as the integral of univariate distribution $\Phi(u)=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{u} e^{-t^{2} / 2} dt$. Find an expression of $Err(C^*(X))$ using only $\mu_{1}$, $\mu_{0}$,$\Sigma$ and $\Phi(u)$.


texts: Usage of Doc2Vec as feature extractor for text classification of websites with political articles

texts: I have gathered political articles from polish websites for my engineering thesis. The main goal is to try to predict the website that input text belongs to. So for this few websites I want to create one doc2vec model, train it, and then use vector representation of these documents (one doc2vec for content, one doc2vec for headers) to serve as an input for some Deep Neural Network. Does it make sense to use the doc2vec as a feature extractor in this case?


texts:
the list of Reuters dataset labels are provided with the dataset and also available in varous online resources. Here is an example.
But I couldn't find what each of these labels actually means.
I was able to find out 'acq' is short for 'Acquisitions'. But yet don't know these ones:
dlr, gnp, bop
Is there a description availbale about what these labels represent?


texts: Making Prediction on logistic regression using SAS

texts:
proc logistic data= train descending;
   class Emp_status Gender Marital_status;
   model Default = Checking_amount  Term  Credit_score  Gender  Marital_status
   Car_loan  Personal_loan  Home_loan  Education_loan  Emp_status
   Amount  Saving_amount  Emp_duration  Age  No_of_credit_acc;
run;

As you can see, I already made a logistic regression on train dataset. However, how can I make the prediction on the test dataset? I checked online, but there are not many sources to implement this.


texts: I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What mathematics tools should I use?


texts: What regression model can handle tiny amounts of data?

texts:
I'm trying to use machine learning to predict properties of a material during a crash test, but each data point requires physically crashing an expensive toy car, so I can only gather a few hundred data points. The end goal is to use the trained model to figure out how to optimize material strength given the parameters I can tweak in the material, which I'm using as inputs.

What type of model would be able to:

Handle very few data points (minimizing number of crashes necessary)
Accurately learn a likely very complex relationship between the inputs (design choices) and the output (performance in crash)
Be able to then be explored to figure out, in as few crashes as possible, which changes to the inputs should be made to maximize the output


texts: Using packages such as sklearn vs building ML algorithm from scratch

texts:
I have been using different machine learning algorithms throughout various projects at university, and attended some inspirational lectures where industrial companies show and present how they use machine learning, data mining, etc. in their work. I myself mostly use Python, and have previously used libraries such as sklearn. My problem is, that I have a huge difficulty understanding the role of built in algorithms vs making them completely from scratch with pure coding and math - i.e. using theoretical machine learning tools to actually do the work yourself. I understand doing everything yourself can be constrained by time/money/resources. Also sometimes it doesn't make sense to reinvent stuff that has been vastly optimized by others before you.

I keep feeling that using sklearn's built in random forest classifier or using xgboost in python is kind of cheating. I am only preparing the data, cleaning it to get the right formats, maybe do some feature engineering og initial plottings and statistical analysis. The problem is that when all that is done, we simply feed the data to this pre-made algorithm and it does everything behind the scene, and just spits out predictions. I feel that i am not doing anything, and not using all the knowledge i have learned in the data exploration analysis. Neither am i using any of the patterns that i found in the data. Still i hear from big companies that they use xgboost and sklearn - and I can see it actively being used in Kaggle competitions.

Almost every website i find only provides examples using these built in libraries, and do not go through any deeper math or statistics at all. I really enjoy working with machine learning - but i have this strong feeling that im completely missing the "professional" approach of doing things. I know there is a lot of books on theoretical machine learning - but still almost everyone online seems to just use pre-made algorithms. I have been struggling with this understanding for about a year now. The validity of these pre-made algorithms in serious industrial/business/academic use is still not clear to me.

EDIT: To be more specific. My question is: How are these libraries/tools viewed in professional/industrial/academical context in comparison to actually building a model yourself. Are they just a "quick and easy" alternative way to start learning machine learning and data mining for students and amateurs, or are they in fact more powerful (than i at least know) and should not be seen as an alternative, but a viable solution for professionals?

The motivation for my single question above, can be ellaborated by explaining the questions i ask myself. The very questions that began this confusion for me. Is it cheating to use these models? In which situations would you use a pre-built library, and when to avoid it? How do i merge (or use) the knowledge gained from the scientific data analysis i did before modelling, and these pre-built classifiers.


texts: Amazon Cloud Image istance most suited for R data mining

texts: I'm new to the field of machine learning, I always used my laptop for regular statistical analysis with no performance problems. Though lately I started programming with caret and I find myself stuck with hours optimizing models and resampling datasets. I saw EC2 istances but I can't understand the difference between the different classes, I know that generally the one with highest numbers of cpus and RAM are most performant but what is the best type of istance for R programming (for example p one vs c one)? Then I'll chose the right amount of memory for my applications but I'm wondering if one subset is more suited than the others


texts: Can Reinforcement learning be applied for time series forecasting?


texts: Detect photos of boats which have the Europe's flag installed on it

texts:
My task is a binary classification: is there a boat in the picture that has the Europe flag installed on it?
In other words, my classifier should first detect if there is a boat in the picture or not, and if there is a boat, is find the flag, and if there is a flag, check if the flag is Europe's flag.
Accordingly, all the following should be identified as false:

no boat in the picture;
no flag in the picture;
the flag is not on the boat (e.g., the flag is installed on a deck);
the flag is not Europe's flag (e.g., Japan's flag).

I have data to train a model to detect boats and flags separately, and I have a ton of pictures of boats with Europ's flag installed on them.
Any suggestions on how best I can layout a machine learning architecture to make the above-mentioned classification?


texts: How to convert a TensorFlow tensor to a NumPy array within tf.Dataset.map() while multithreading/multiprocessing?

texts:
I use TensorFlow 1.12 in eager execution. I have a very expensive function which I map onto this dataset using tf.Dataset.map(). Inside this function — which I developed by simply for-looping over the dataset in eager execution — I convert the tensors to NumPy arrays using EagerTensor.numpy(). I now wish to multithread this whole map procedure, using tf.Dataset.map(myfunction, num_cores=30). However, now myfunction() isnt being executed eagerly anymore. I tried two remedies: 1) wrapping myfunction() using tf.py_function, in which case map() isnt being multithreaded anymore, and 2) opening a tf.Session inside myfunction() and getting my NumPy array using .eval() instead of .numpy(), in which case I need to supply input tensors (I dont understand why):

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'arg0' with dtype string
     [[node arg0 (defined at /home/.../semantic_fpn.py:101)  = Placeholder[dtype=DT_STRING, shape=<unknown>]]```

Is there an easy why for me to loop over my dataset with ```myfunction()``` (and its NumPy conversion) while multithreading/multiprocessing?


texts: Understanding clusters after applying PCA then K-means

texts:
I have a dataset grouped by customer level, and the rows are sum_mexico, sum_uk, ... etc to indicate if the customer has spent money at stores in those countries..similarily counts for these as well. I end up with 200 columns.
I would like to cluster this data to observe the spending habits and see if i can group them by these features. I'm unsure what clustering method would be best but i would like to provide meaningful results to business.
I've read about using PCA then k-means. E.G. you can carry out K-means on your Principcal componenets. What i don't understand is how i can then interpret the results. e.g. say i have the below situation from https://365datascience.com/tutorials/python-tutorials/pca-k-means/:

i can visually see 4 clusters, but how can i get the characteristics of each cluster.. What can i say about cluster 1 for example, are they charactertized by high spend in mexico ? (assuming i got these results myself using my own data and steps from link!) My end goal is to understand what charcterizes each cluster e.g. it may be that certain customer IDS have high spends in spain and wine etc.


texts: What are some unsupervised clustering methods that are fairly easy to explain and scale well?

texts:
I have been tasked to identify some clusters in some data and then match these with real world counterparts to find the accuracy. To give a contrived example, say I have some data about profiles for vehicles like cars, buses and trains. I want to be able to identify these  from an unlabelled dataset of mostly numerical data using clustering techniques that ideally scale well, and aren’t too “black box” i.e. easy to explain to non-technical audiences.

I’ve thought about using kmeans and dbscan. Kmeans is fairly easy to explain but I need to choose the number of clusters as opposed to dbscan. However, both are fairly computationally intensive, and I’m looking to implement something that works real time after say an API call.

What’s the best way of going about this?


texts: Avoid hardware limitation while competing in Kaggle?

texts:
I've learned machine learning via textbooks and examples, which don't delve into the engineering challenges of working with "big-ish" data like Kaggle's.

As a specific example, I'm working on the New York taxi trip challenge. It's a regression task for ~ 2 million rows and 20 columns.

My 4GB-RAM laptop can barely do EDA with pandas and matplotlib in Jupyter Notebook. However, when I try to build a random forest with 1000 trees, it hangs (e.g. Kernel restart error in Jupyter Notebook).

To combat this, I set up a 16GB-RAM desktop. I then ssh in, start a browser-less Jupyter Notebook kernel, and connect my local Notebook to that kernel. However, I still max out that machine.

At this point, I'm guessing that I need to run my model training code as a script.

Will this help with my machine hanging?
What's the workflow to store the model results and use it later for prediction? Do you use a Makefile to keep this reproducible?
Doing this also sacrifices the interactivity of Jupyter Notebook -- is there a workflow that maintains the interactivity?

My current toolkit is RStudio, Jupyter Notebook, Emacs, but am willing to pick up new things.


texts: Would writing a decision tree algorithm in Pytorch or Tensorflow be faster than with Numpy?

texts: Since these libraries can turn CPU arrays into GPU tensors, could you parallelize (and therefore accelerate) the calculations for a decision tree? I am considering making a decision tree class written in Tensorflow/Pytorch for a school project, but I want to be certain that it makes sense.


texts: Machine learning algorithms for time series analysis

texts:
I am wondering which ML algoirthms (supervised) are commonly used for TS analysis.
Which ones have used and found to be succesful for your projects?


texts: In YOLO training, what if two objects' centers fall in the same grid?

texts:
As I know, YOLO predicts one classification result (as well as some bounding boxes) for each grid. But when training yolo, what if two or more objects' centers fall in the same grid? How to choose the training target among these objects?
Isn't it unable to train all the objects in the same grid? Since there's only one output for classification in a grid, then there can be only one object as the ground truth.
Wonder how YOLO deals with this problem.


texts:
I have written a variable length sequence to seqeunce autoencoder in keras using this tutorial as a guideline: https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html.  The idea to apply it to the problem of time series forecasting came from the winning entry of this kaggle competition: https://www.kaggle.com/c/web-traffic-time-series-forecasting

The model is trained on 1831 different time series of products but they all come from the same product category.  Each series has a minimum length of 156 data points and a maximum of 208 (So 3-4 years of historical data).  The data points in each series are upscaled daily data aggregated to the Monday of each week.  Some of the series are intermittent some are not

The forecast horizon for each time series is 26 points into the future.  The data is log(x + 1) transformed and then normalised by subtracting the mean of the dataset and then dividing by the standard deviation of the dataset.

The features fed into the model are:

The time series
Lags of the the time series using 13, 26, 13, & 52
A yearly autocorrelation scaler per series which is tiled to be the length of each time series
A quarterly autocorrelation scaler per series which is tiled to be the length of each time series
A product popularity scaler per series which is tiled to the length of each time series.

The accuracy of the model is evaluated with two separate measures MASE and a variant of SMAPE:

Both of these metrics should be as close to zero as possible.  The forecasts need to be made per series and thus I determine overall success by taking the median of all the MASE scores and calculating one SMAPE3 score by taking all the forecast and actual values and computing them in one sum for all the series instead of taking an average per series.

My model has a single encoder layer with 64 neurons and a single decoder layer also with 64 neurons.  It uses Adam optimiser using all keras defaults and the mean absolute error as the loss function, as I believe this most closely reflects the error metrics it is evaluated with eventually.  The model doesn't perform as well as I want it to and I am looking for ways to imporve it.  Currently as measured on my validation set I have the two following scores:

Vol SMAPE3: 0.46 and MASE median: 0.75

The model is trained over 100 epochs with a batch size of 256.  Below is the loss and some example plots of series and predictions the model makes:

To me the model doesn't seem to capture the variation in the series very well and almost looks like a linear learning.  It appears to only really capture the level of the different series well.  To improve it I would like to try and add the ability for the model to capture more of the variations in the series and predictions rather than the smooth kind of predictions it gives at the moment.  Are there any suggestions for how to achieve this?


texts: Do I need to include a squared and linear variable in a random forest to achieve a parabolic effect?

texts:
I'm using a random forest model. One of my independent variables almost certainly has a parabolic effect on the dependent variable. In a linear regression, I would include the variable as both linear and squared in order to capture this effect. Should I do the same in a random forest?

By extension, same question for variables that have logistic effects (age, for example)?


texts: tensorflow: is there a way to specify XLA_GPU with tensorflow?

texts:
following code is used to specify device on which tf node is running on

with tf.device('/gpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

i have already known:
this post, tensorflow doc and xla demo

what i want to know is:

is there a way to specify XLA_GPU as the device on which tf node is running on

with tf.device('/XLA_GPU:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

executing the code above gives ValueError: Unknown attribute: 'device' in '/job:localhost/replica:0/task:0/device:/XLA_GPU:0'

this is 100% reproducible on google colab.


texts: What are the different models I can use to find the gender affinity of businesses using yelp dataset-- https://www.kaggle.com/yelp-dataset/yelp-dataset . I need to find Probablity (Male buying from a merchant) and Probablity (Female buying from a merchant) where both probablities add to 1 . There is no information about gender so I can use the Genderize api to find gender using names of users. This is an unsupervised problem


texts: Time Series Machine Learning Feature Selection Problem

texts:
I have to solve a time series model that can take one of two shapes. It can probably take more but here are the two I'm going to ask about. If you have other ideas they are of course welcome.

First Possible Model -

  X(Dependent variable 'Spending') = X(lag1)...X(lagN) + X(Dummy
  variable when US) + X(Dummy Variable when Mexico) ... + Error term

Or Make a separate model for each Country like -

  X(Total Spending in the US only) = X(lag1)...X(lagN) + Error term

  X(Total Spending in the Mexico only) = X(lag1)...X(lagN) + Error term

  … for Each country

Mathematically I can't decide what approach is better. I will use an F-Statistic, dickey fuller statistic to check the auto regression for stationarity and then compare the two models but I wanted to see what others thought of the theory and if you should ever include the dummy variables.

I'm looking more for an answer that includes mathematical reasoning.


texts: Data Engineering Stack - collect, transform and visualize geospatial data

texts:
I'm making a side project, where I collect geospatial data by web scrapping and from OSM API. I've started with simple Java application, however, I would like to make it as a data flow, purely for learning purposes.

Unfortunately, my knowledge about tools, and mostly connecting them, is, well, low.

What is my goal?
As a final result I want to visualize scrapped geospatial points on the map with the roads connecting them(from OSM).

Current flow:
In standalone Java application I'm scrapping the data for geospatial points. There is a client consuming the OSM API for needed data.

What I think it might be useful:
Use Apache Spark for collecting and transforming the data. Then use somehow GeoSpark, or Geotrellis, and Zeppelin to visualize the data. I was also thinking about using ES + Kibana for geodata, but it looks like the Zeppelin is enough.
I feel comfortable to work with Java, then Scala.

What do you think? Are there any better tools I can use? Did I miss anything?


texts:
I am learning how to plot data. I am trying to obtain this grafic result on my dataset. The thing is, I can't do it. The cool plot is from a kaggle's kernel, which code is:

ggplot(cocoa,aes(x=pct, y=rating)) +geom_point(aes(colour=factor(location))) +
  theme( legend.position="bottom",legend.key.width=unit(0.2,"cm"),legend.key.height=unit(0.2,"cm")) +
  xlab("Cocoa Percent(%) ") + ylab("Chocolate Bar Rating") +
  ggtitle("Scattor plot of Cocoa Percent vs Chocolate Bar Rating")

When I try to install "ggplot" on RStudio, it says Error in install.packages : object"ggplot" not found. So I tryied copying the text in plot() function. Obviously, it gives me errors on the arguments.
The only thing I managed to do is this:

plot(data=chocolate, Rating~CocoaPerc, xlab="Rating", ylab="Cocoa Percentage", col="brown",
     main="Valutazione e Percentuale di cacao")

Can anybody help me achieving the cool result? Or can you advise me any other "tricks" or guides about data plotting and grafic study of datasets? Thanks


texts: Document matching with more priority to certain features than others

texts: I am working on recommendation systems wherein I need to match the similarity of 2 users. Now, I know that I can use Tfidf vectorizer to calculate the the cosine similarity score between them. But, now suppose I have some features where I have different priorities for those features. So, for each feature there will be a different priority and the one with with higher priority will be checked first. So, when I get cosine similarity based on that feature, I will move on to the next feature and so on. How can I achieve this?


texts: Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

texts:
I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is heavily distributed on only 3-6 topics and all the rest are close to zero ($K$~$\mathcal{O}(100)$). What I mean by this, is that the 3-6 highest contributing topics for all documents is orders of magnitude (about 6 orders) greater than the rest of the topic contributions.

If I use the Jensen-Shannon distance to compute the similarity between documents, I need to store all values of the topic distribution as non-zero, even the very small values of the non contributing topics, because Jensen-Shannon divides by each discrete value in the distribution. This requires a lot of storage and is inefficient.

If, however, I store the topic distributions of each document as a sparse matrix (the 3-6 highest contributing topics are non-zero and the rest are zero) where each row is a unique document and each column is a topic, then this uses far less space. But I can no longer use the Jensen-Shannon metric, because we would be dividing by 0. In this case:

Can I use the euclidean distance between documents topic distributions to compare similarity between documents?

Using the euclidean distance would require far less storage and is extremely fast to compute.

I appreciate that Jensen-Shannon is one of the "correct" metrics to compare discrete probability distributions, as well as the Bhattacharyya distance and Hellinger distance. But ultimately, the output of LDA is a discrete topic distribution for each doucment - each document is a vector (or point) in a $K$ dimensional space. By this argument, is it valid to use the euclidean distance to calcualte documents similarities? Is there something blatantly wrong with this method?

I have tested the euclidean distance to compare documents, and yielded good results, which works well for my industrial application. But I want to know the academics behind such a method. Thanks in advance!


texts: Non-parametric approach to healthcare dataset?

texts:
I have a Healthcare dataset. I have been told to look at non-parametric approach to solve certain questions related to the dataset. I am little bit confused about non-parametric approach.

Do they mean density plot based approach (such as looking at the histogram)?
I know this is a vague question to ask here. However, I don't have access to anybody else whom I can ask and hence I am asking for some input from others in this forum.

Any response/thought would be appreciated.

Thanks and regards.


texts: Adding the input layer - units with a decimal

texts:
I took the course Machine Learning A-Z from Udemy and am trying to apply what I learned in the tutorials.  Theye taught us in the "Adding the input layer" portion of an ANN that the units is based off of the input_dim.  Normally the "units = (input_dim + 1) / 2".  In the dataset that I am working with my input_dim=754.  (754 + 1) / 2 = 377.5.  Should I use 377.5 or should I round up or down to a whole number?

#Adding the input layer and the first hidden layer
classifier.add(Dense(units=377.5, kernel_initializer='uniform', activation='relu', input_dim=754))