texts: Using Generative Adversarial Network to generate Single Image

texts:
Most generative adversarial networks learn the distribution of the dataset and then generate a sample of 10's to 100's of images with similar distributions. I am curious if there is any research regenerating a single image.

I have looked into Super Resolution GAN's and they somewhat model this. They try to estimate a highly resolved image from its lower resolution counterpart. Could that be used to recreate an image?

Thanks for the help.


texts: What to do when feature engineering and parameter tuning don't add to the base model performance

texts:
I've been working on using LogisticRegression from scikit to try the Titanic Kaggle comp.

I've found something interesting, and that is that no amount of feature engineering and paramater tuning is changing my base model by more than a percentage point one way or the other.

At this point I'm completely self taught, so I figure one of two things is happening (maybe both):

1. I'm doing logistic regression all wrong

2. logistic regression isn't the right choice for the problem.

Are either of these two things true? My notebooks are below. They are a bit long, but if you are bored during quarantine, I'd appreciate feedback:

notebook 1: cleaning, feature engineering and comparing base models against one another.

notebook 2: hyper-parameter tuning with GridSearchCV


texts: What is the distribution of the goal feature in this dataset

texts:
https://www.kaggle.com/kemical/kickstarter-projects
I was checking the distribution of the "goal" feature in this 2016 KickStarter data set. I figure goal would be more normally distributed but it wasn't. I checked the Kick Starter website for published statistics. It seems the sampling was random as certain categories  appear at the frequency Kick Starter Published but the data didn't meet the required assumptions for a T test. I check the Kaggle site for a summary of the goal data and the distrubtion looks the same but I'm very skeptical. I made a qq plot to check normality but this really can't be right.
What type of distribution is the column 'goal' in this data set? If the distribution should look like this why? Maybe I broke something or put in the wrong code and I wanted to be sure. My gut is telling me to take a serious look at the goal column.


texts: What the good general regression technqiue for a problem with 50 independent varaibles

texts:
I am a newbie to data science and statistics. I came across this problem, which has 50 independent variables and one dependent variable and trying to identify the good regression technique to start with. The following is the flow chart that I executed:

Data Exploration -> Correlational matrix -> dimension reduction -> PCA (Dimension reduction) -> Basic Linear Regression technique.

Can someone guide me, if there is any other better technique or procedure.


texts: Hello, anyone able to direct me to a "cheat sheet" of Neural Network equations with legends?

texts:
Can anyone here can direct me to a site that provides a cheat sheet of equations for Neural Networks with a legend for the notation?

It can be on any and all aspects of NN, be it forward or back propagation, activation function, CNN, RNN etc.

I have some reading material with the equations and I have worked through them.  I am looking for a site that breaks it down by notation by legend.

For example, without posting a specific equation, a landing page/cheat sheet that identifies each element in the equation, such as: b= bias
w= weight  a=activation the signification of what numbers are vectors in the equation etc.

I work through equations but find most literature lacking in descriptions within the equation, which makes it difficult for someone who is trying to refresh their linear algebra and calculus.

Thank you.


texts: What is the best technique to transform documents into vectors?

texts: What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?


texts: General question on the approach to optimise numbers

texts: I have a huge huge model in SQL that nobody knows what it is doing. This model spits out some numbers and those numbers should be optimised to match another batch of 'correct' numbers as much as possible. So I get one set of numbers form my black box model that change according to how different parameters change inside that model - there are around 400 parameters that can be changed to get different numbers out of it (no, I don't want to even touch it, less trying to understand how it works) and I have another 'correct' set of numbers that I get for those parameters from a person in charge of it. What would be the best approach to 'optimise' my black box numbers to match the correct ones? The black box numbers will relatively match the correct numbers depending on how parameters are set up but I need to make them more correct. I was thinking maybe Bayesian optimisation in Python or GridSearch but not sure if that is the best approach. Any ideas would be appreciated.


texts: Hyperparameter tuning with Bayesian-Optimization

texts:
I'm using LightGBM for the regression problem and here is my code.

    def bayesion_opt_lgbm(X, y, init_iter = 5, n_iter = 10, random_seed = 32, seed= 100, num_iterations = 50,
                          dtrain = lgb.Dataset(data = X_train, label = y_train)):
      def lgb_score(y_preds, dtrain):
        labels = dtrain.get_labels()
        return 'r2', r2_score(labels, y_preds), True

      def hyp_lgb(num_leaves, feature_fraction, bagging_fraction, max_depth, min_split_gain, min_child_weight):
          params = {'application': 'regression',
                    'num_iterations': 'num_iterations',
                    'early_stopping_round': 50,
                    'learning_rate': 0.05,
                    'metric': 'lgb_r2_score'}
          params['num_leaves'] = int(round(num_leaves))
          params['feature_fraction'] = max(min(feature_fraction, 1), 0)
          params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)
          params['max_depth'] = int(round(max_depth))
          params['min_split_gain'] = min_split_gain
          params['min_child_weight'] = min_child_weight
          cv_results = lgb.cv(params,
                              train_set = dtrain,
                              nfold = 5,
                              stratified = False,
                              seed = seed,
                              categorical_feature = [],
                              verbose_eval = None,
                              feval = lgb_r2_score)
          print(cv_results)
          return np.max(cv_results['r2-mean'])
      bounds = {'num_leaves': (80,100),
                'feature_fraction': (0.1, 0.9),
                'bagging_fraction': (0.8, 1),
                'max_depth': (5,10,15,20),
                'min_split_gain': (0.001, 0.01),
                'min_child_weight': (10,20)
                }

      optimizer = BayesianOptimization(f = hyp_lgb, pbounds = bounds, random_state = 32)

      optimizer.maximaze(init_points= init_iter, n_iter = n_iter)
    bayesion_opt_lgbm(X_train, y_train)

When I run my code, I get an error something like that, Please help me where am i missing
TypeError                                 Traceback (most recent call last)
TypeError: float() argument must be a string or a number, not 'tuple'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-57-86f7d803c78d> in <module>()
     40   #Optimize
     41   optimizer.maximaze(init_points= init_iter, n_iter = n_iter)
---> 42 bayesion_opt_lgbm(X_train, y_train)
     43

2 frames
/usr/local/lib/python3.6/dist-packages/bayes_opt/target_space.py in __init__(self, target_func, pbounds, random_state)
     47         self._bounds = np.array(
     48             [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])],
---> 49             dtype=np.float
     50         )
     51

ValueError: setting an array element with a sequence.


texts: Can I continue a machine learning process after shutdown?

texts:
I use Mask RCNN for object detection and instance segmentation.
I am new to neural networks, this is my first project.

I use jupyter notebook. Can I continue learning after shutdown?

I would like start learning from 6 epoch. How can I do it?
One epoch ~ 1h. I don't want to waste time, and continue this process. Is it possible?


texts: Converting continuous to categorical variable

texts:
What method must be chosen for converting a continuous variable(socio-economic ratio) into a categorical variable, the quantiles are as follows:

df['ses'].quantile([0,0.25,0.5,0.75,1])

0.00     0.000
0.25     1.070
0.50     1.979
0.75     3.341
1.00    11.889


texts: I am looking for a paper detailing the very basics of deep learning. Ideally like the Andrew Ng course for deep learning. Do you know where I can find this ?


texts: Is there any Package for PCA for data having Categorical variable ?


texts: Tensorflow Model not returning a Distribution object when having DistributionLambda as last layer in a multitasking model

texts:
I am building a TF CNN model that takes a picture as input and has 3 outputs (multitask learning). On one of the output layers, I would like to output a distribution object, tfp.distributions.Normal, from which I can take the mean and sttdv to figure out confidence intervals as in this example Case2.
The problem is that my model does not output a tensorflow_probability.python.util.deferred_tensor.Normal type of object as in the model. Therefore, in the output I cannot call .mean() or .sttdv(). Instead it only outputs a numpy value. What am I doing wrong? How can I make my model output a distribution type object from which I can call .mean() and .sttdv()?
base_model
...
output1 = Dense(16, activation='relu')(base_model)
output1 = Dense(8, activation='relu')(output1)
output1 = Dense(1, activation='sigmoid')(output1)

...
output2 = Dense(32, activation='relu')(base_model)
output2 = Dense(16, activation='relu')(output2)
output2 = Dense(5, activation='softmax')(output2)

...
dist_output3 = Dense(128, activation='relu')(base_model)
dist_output3 = Dense(128, activation='relu')(output3)
dist_output3 = Dense(2)(output3)
dist_output3 = tfp.layers.DistributionLambda(lambda t: tfd.Normal(loc=t[..., :1],
                                                               scale=1e-3 + tf.math.softplus(0.05 *t[...,1:])))(dist_output3)

model = Model(inputs=base_model.layers[0].input,
              outputs=[output1, output2, dist_output3])

negloglik = lambda y, p_y: - p_y.log_prob(y)

model.compile(optimizer = 'adam',
              loss =['binary_crossentropy', 'categorical_crossentropy', negloglik],
              metrics=['accuracy', 'mse'])

history = model.fit_generator(
    train_generator,
    validation_data=validate_generator,
    epochs=100,
)

...
predictions = model(test_generator)
len(predictions)
>>> 3

predictions[0][0]
>>> array([0.08734298723])

predictions[1][0]
>>> array([0.26107603, 0.1662454 , 0.24182138, 0.09926344, 0.23159383])

predictions[2][0]
>>> array([0.9827457034]) <-- I am expecting the output of this to be something like array([0.9827457034, 0.246534654]) as in array([<mean>, <std>])


texts: Convolutional Neural Network for outlier detection

texts: Lets say I have the cifar 10 dataset and I manipulate it in such a way that I reduce the number of frogs in the dataset from 6000 to say 600. I now consider frog as an outlier and train a two class classification problem on this modified cifar 10 dataset. All the other 9 classes as one group and the frog as a outlier. What architecture is best to detect this frog outlier? Where can I find reading materials to learn about these?


texts: Tensorflow uses more memory, the more epochs it completes

texts:
I created a genetic algorithm "optimizer" for Tensorflow but it is written in python. I know TensorFlow was not designed like this and I need to rather create the optimizer in C++ using their API's but I found out about it only after I already programmed the optimizer and I don't really have the time to do research on how to create a tf.train.Optimizer(). My GA "optimizer" is not really an optimizer as it is more a tf.Variable.assign() call. See code below

The "optimizer" updated a series of weight matrix variables after the evolve takes place. Each variable represents a weight matrix between two layers. If it is a network with inputs, one hidden layer and outputs then there will be two variables. Weights between input to hidden then hidden to output. Although this is a 2d matrix but my weights act as an individual in my population so I have a 3d matrix. 1 dimension represents the population amount and the 2d matrix as the weigths. Weight updates are then done by crossover or mutation which consists of a lot of multiplication, matmul and summation operations. My guess is the variables I create for the new layers is not garbage collected. I am guessing tensorflow keeps the variables alive as I don't ever call a train step provided by the tf.train.Optimzer classes since I created my own.

Here is my code to run my GA optimizer

for i in range(epochs):
    sess.run(fitness_per_individual, feed_dict={x: train_data, y: train_labels},
                 options=run_options,
                 run_metadata=run_metadata)
    evolved_layers = evolve_layer(nn_layers, fitness_per_individual, population)
    for tensor_idx in range(len(nn_layers)):
        layer = nn_layers[tensor_idx]
        evolved_layer = evolved_layers[tensor_idx]
        layer = layer.assign(evolved_layer)

        # I don't have to feed my training data again I just do it here to keep tensorflow happy.
        sess.run(layer, feed_dict={x: train_data, y: train_labels},
                 options=run_options,
                 run_metadata=run_metadata)

Is there a way I can dispose of variable matrices created by tf.multiply or is there no way around it without creating a tf.train.Optimizer. Also is there a good practice guideline I can read up for TensorFlow.


texts: How to load this data from .dat into dataframe using python

texts:
1 1:31080.410200 2:2.871828 3:5.862267 4:7.100850 5:8.283706 6:-5.427875 7:-6.667087 8:-8.888233 9:28898.943400

Can someone please tell me how to load this data into a dataframe from .dat file. The data is given such that attribute number:value. I want only the values into  the dataframe


texts:
I have a Dataframe which looks as shown below

I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as
sns.set_style("darkgrid")
plt.plot(wr['a'][:100])
plt.show()

but the plot looks shabby,

wr.set_index(['Date_x'],inplace=True)
wr['a'][:100].plot()
wr['b'][:100].plot()

I am looking to have something like this

Any Help is Appreciated.


texts: Method for finding top-k cosine similarity based closest item on large dataset

texts:
I have a dataset with 40 million item, where each item is 400-dimension double vector. What I want to do is to find top-k (small k, about 3~10) most similar items to an arbitrary given input vector. Similarity measure is cosine similarity, since this dataset is based on word2vec representation.

However, this data is so huge, so it couldn't fit at main memory (I'm now working in single machine). The goal I want to achieve is to find top-k similar items as fast as possible, with small memory(~5G) that could fit in RAM.

Any suggestions for this problem? I've already tried PCA, but I observed that this data projected in low-dimension wasn't working very well..


texts: Parameters used in GridSearchCV Slow Down in skLearn Tutorial

texts:
I'm not sure why grid.fit(X,y) is correct, rather than grid.fit(X_2d, y_2d)

In this tutorial on RBF SVM Parameters, we are using GridSearchCV to find the optimum hyperparameters for an SVM.

They have the following code:

# Dataset for decision function visualization: we only keep the first
two
# features in X and sub-sample the dataset to keep only 2 classes and
# make it a binary classification problem.

X_2d = X[:, :2]
X_2d = X_2d[y > 0]
y_2d = y[y > 0]
y_2d -= 1

...

param_grid = dict(gamma=gamma_range, C=C_range)

# GridSearchCV will search the parameter space for the best parameters to use, minimizing the score function
cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)

grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv)

# ==================== CODE I'M INTERESTED IN  ==================>
# ===== SWITCH `grid.fit(X,y)` with grid.fit(X_2d, y_2d) ========>
grid.fit(X, y)
# ==================== ^^^^^^^^^^^^^^^^^^^  =============>

print("The best parameters are %s with a score of %0.2f"
      % (grid.best_params_, grid.best_score_))

where X_2d and y_2d are a subset of X and y.

Just some information to show what X and y are:

print(X.shape)    #(150,4)
print(y.shape)    #(150,)
print(X_2d.shape) #(100,2)
print(y_2d.shape) #(100,)
print(type(X))    #<class 'numpy.ndarray'>
print(type(y))    #<class 'numpy.ndarray'>
print(type(X_2d)) #<class 'numpy.ndarray'>
print(type(y_2d)) #<class 'numpy.ndarray'>

Why does changing the code above to grid.fit(X_2d, y_2d) not work? I'm not sure if it's taking a very long time, OR that it's incorrect. My Jupyter Notebook is just sitting, whereas grid.fit(X, y) only takes seconds.

My original thought was that we would want to fit to the actual data set we're running on which are X_2d and y_2d, rather than X and y


texts: Choosing the right corpus to build a TF IDF Vectorizer used for comparing the similarity of two strings

texts:
I am working on evaluating when a pair of string objects can be considered equal (e.g. given that we are talking about journals, is "international journal of air and water pollution" the same of "air and water pollution"?) and I was wondering what is the proper corpus to use to build a TF IDF vectorizer.

I am currently using as corpus all the distinct values of these strings belonging to the same type (in this example all the strings referring to journals). Is this a valid approach? Why? What could be other valid approaches?


texts: TensorFlow: number of channels of conv1d filter

texts:
I want to apply a ConvNet on my one dimensional data retrieved from 13 sensors. So, each of my samples consists of 13 channels (of 51 values)

I am using 'conv1d' to apply a ConvNet on my data. The network works nicely, but I wonder how 'conv1d' determines the number of channels for it's filters... To my knowledge, a filter should have an equal number of channels as its input data, which makes it a $5x13$ filter. I set the filter to have a width of 5, but don't need to set the number of channels anywhere.

My question is: how does layer 'conv1' determine it's number of channels?

Below is a portion of my code:

# We have 13 1D channels of 51 points each

# Note that we've indicated -1 for batch size, which specifies that this dimension should be dynamically computed
# based on the number of input values in features["x"], holding the size of all other dimensions constant.

input_layer = tf.reshape(features["x"], [-1, 51, 13])

# Convolutional Layer #1
# Shouldn't this filter also need to number of channels?
This should match the input number of channels
conv1 = tf.layers.conv1d(inputs=input_layer, filters=32, kernel_size=5, padding="same", activation=tf.nn.relu)

# Pooling Layer #1
pool1 = tf.layers.max_pooling1d(inputs=conv1, pool_size=2, strides=2)


texts: Feature selection or Dimension reduction in unsupervised learning

texts:
I'm trying to do Embedded clustering using kmeans.

This is customer data, so it involves a lot of sentences, so I'm using the universal sentence encoder before clustering.

But I should be doing a feature selection or dimensionality reduction before embedding the features.

I want to know if there is a method to do feature selection or dimensionality reduction in unsupervised learning. This could be very helpful as the clustering is giving a mixed result as of now and I have a strong feeling that this could be because of the unwanted attribute in the data.

I have read all the resources which only gave options to be done on the supervised learning.

Any help is appreciated!

Thanks
Arav


texts:
I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.

Can R + Hadoop overcome R's memory constraints in any case?

The answer might be clear as the accepted answer to
this question implies but for me it isn't. To be more precise:

Can I use R + Hadoop to fit a model to a really big data set at once. I mean that the entire data is needed for the computation without any independent sub processes which could be parallelized in some way?

I do not see how this can work when using a computer cluster for the computation. In case it is possible: How does it work?


texts: On Python is the `i` variable defined inherently?

texts:
I am new to this whole landscape, please need some hand holding at first, I can't find a beginner section. Can someone please tell me if i is defined inherently in Python. Can i test for this: what type of object is i?

squares=['red', 'yellow', 'green', 'purple', 'blue']

for i, square in enumerate(squares):
    print(i, square)

Square i is not even defined? There is no programming error.


texts:
I am using Scikit's MLPRegressor for a timeseries prediction task.

My data is scaled between 0 and 1 using the MinMaxScaler and my model is initialized using the following parameters:

MLPRegressor(solver='lbfgs', hidden_layer_sizes=50,
                           max_iter=10000, shuffle=False, random_state=9876,
                           activation='relu')

I am expecting output between 0 and 1 but getting values outside the bound (both negative values as well as > 1).

Non-normalized data has the same problem, I get predictions out of range!

Any idea where I could be wrong?

UPDATE:
Based on the answers below I played a bit with modifying the output activation layers and got some interesting results that I thought worth sharing. There are three scenarios, hope the captions convey the message clearly:
Legend:

Black solid line = Training Epoch

Red solid line = Test Epoch

Cyan dashed line = Network prediction over the entire data set

Output when the network is trained using 'relu' activation layer but output_activation_ set to 'logistic

Output when the network is trained using 'relu' activation layer and output_activation_ set explicitly to 'relu'

Output when the network is trained using 'relu' activation layer and output_activation is left alone


texts: I have a very large csv file which Apple Numbers won't open. I can open it with TextEdit, but each row is so long it forms multiple lines in the document and makes the document difficult to understand. Is there a tool for opening a csv and exploring different rows and columns?


texts: Structured Support Vector Machine (Joint Feature Map)

texts:
I'm studying Structured Support Vector Machine. (https://en.wikipedia.org/wiki/Structured_support_vector_machine)

The theory's clear, but I need a tangible example to make everything more concrete.

Could you provide me with a real example of SSVM, specifying me the structured output space and the joint feature map Ψ(x,y)?

Happy Holidays


texts: Does it make sense to use a tfidf matrix for a model which expects to see new text?

texts: I'm training a model to classify tweets right now. Most of the text classification examples I have seen convert the tweets into tf-idf document term matrices as input for the model. However, this model should be able to identify newly collected tweets without retraining. Does it make sense to use tf-idf in this context? What is the correct way to turn tweets into feature vectors in this task?


texts:
I am building a binary classifier from a set of feature vectors some of which are categorical like Yes or No (two options). I am replacing them with 1 and 0 and since there is strong imbalance between 1 and 0 my model learns basically on those features. I am afraid that this particular feature might be an artifact - samples that have 0 can also sometimes be classified as 1. What to do in this situation? Should I drop the feature completely?

Update:

Let me explain in detail the problem since the answers mostly focus on imbalanced classes. I have a dataset that comprises of around 30 features and binary classes {0,1}. Features are mostly numerical (continuous), but there are also binary categorical features like YES/NO, MALE/FEMALE etc.

One aspect of this dataset is imbalanced classes (there are more ones than zeros) and the other aspect is that one categorical feature, lets say $x$ (YES/NO) is not very balanced also. In fact if you make predictions solely based on $x$ like: $x = 1 \rightarrow 1, x=0 \rightarrow 0$ you will perform better than a naive model, which predicts only $0$ (remember imbalanced classes).

Now, my dilemma is what to do in that case? Should I remove that variable completely from modelling or maybe use some techniques of bias removal?


texts: How to get different length output using keras?

texts:
I am working on a time-series kind of problem. I have (timesteps, features) and the length of output will be (timesteps,). All outputs as float numbers. But the problem is I have training samples with different timesteps.
Suppose one example has 1500 timesteps then it will output 1500 dimensional vector. Another example has 1000 timesteps and it will have 1000 dimensional output.
I am looking for a way to train such a model using keras.


texts: Why are my Decision Tree Leafs not pure?

texts:
I'm making a using DecisionTreeClassifier from SKlearn (v0.21.3) with its default settings, using Python. I do not want regularize it in any way, I want it to overfit as much as possible.

When drawing the tree out I saw that some of the leafs were not pure. Is this normal? Was the tree not able to separate the samples?

   ...
   model = DecisionTreeClassifier(criterion="entropy")
   model = modell.fit(X, y)
   ...


texts: For outliers treatment, clipping, winsorizing or removing?

texts:
I came across two different techniques for treating outliers winsorization, clipping and removing:

Winsorizing: Consider the data set consisting of:
{92, 19, 101, 58, 1053, 91, 26, 78, 10, 13, −40, 101, 86, 85, 15, 89, 89, 28, −5, 41}       (N = 20, mean = 101.5)
The data below the 5th percentile lies between −40 and −5, while the data above the 95th percentile lies between 101 and 1053. (Values shown in bold.) Then a 90% winsorization would result in the following:
{92, 19, 101, 58, 101, 91, 26, 78, 10, 13, −5, 101, 86, 85, 15, 89, 89, 28, −5, 41}       (N = 20, mean = 55.65)
Clipping:Given an interval, values outside the interval are clipped to the interval edges. For example, if an interval of [0, 1] is specified, values smaller than 0 become 0, and values larger than 1 become 1.
Removing: Just taking them out.

My questions are:

In which cases should I use which one?
If I always do winsorization (seems the best in my opinion) when I am loosing important information?
Is this model dependent (for decision trees, for linear...) or the same strategy can be applied to all of them


texts: out of frame landmark detection with CNN

texts:
a robust landmark detector must cope with occlusion (the landmark surrounding are occluded by another object) the detection can still be performed but it should be stressed that the landmark is occluded. below you can notice that the occluded landmarks are presented in red

in my opinion, there is a more challenging problem that I have not found a lot about in the literature which is the case where the landmark area is out of frame.
imagine that you face the case below where the right eye extremity is out of frame but you model will spit any way the same number of the detected landmark.

how can one manage this in the training step? how could we inform our model that the landmark is out of frame?


texts: Predict quantitative response variable from quantitative and qualitative variables

texts: I have to predict a quantitative response variable Y from p quantitative and q qualitative variables. There's around 100,000 rows in the data frame. What predictive methods can I use for that purpose?


texts: Is there an NLP corpus that contains common medical terms?

texts:
I am trying to use the NLTK library to extract keywords denoting medical symptoms from medical reports of patients. For example, I have a medical report as follows:

s:a 33 year old female crystallographer presents with mild spells of
vertigo, mild headaches particularly at the back of the head and in
the morning x 2 weeks. pt also reports chronic mild occasional
lightheadedness. o:Height 160 cm, Weight 53.8 kg, Temperature 37.3 C,
Pulse 76, SystolicBP 146, DiastolicBP 93, Respiration 15, Heart = 2/6
systolic murmur at base of heart, Chest = clear to auscultation B/L,
no rales or wheezing, Extremities = no edema or clubbing, Heart =
normal S1, S2, RRR a:Hypertension p:performed E/M Level 2 (established
patient) - Completed, and prescribed Hydrochlorothiazide - 50 mg po
qd, and ordered Cholesterol.

Here, I would like to find all the keywords or bigrams that represent medical symptoms. In the text above, these keywords are 'mild spells of vertigo', 'mild headaches', 'lightheadedness' etc.
For this, I need some kind of collection of terms that represent symptoms, so that I can detect similar terms in the medical reports I have. Is there any NLTK corpus associated with medical terminology? If I find a list of words that denote medical symptoms, I can tokenise and lemmatise the words I have detected in the medical reports and compare them to the words found in the corpus.
Thank you.


texts: Kaggle Titanic Survival Table an example of Naive Bayes?

texts:
is the survival table classification method on the Kaggle Titanic dataset an example of an implementation of Naive Bayes ? I am asking because I am reading up on Naive Bayes and the basic idea is as follows:
"Find out the probability of the previously unseen instance
belonging to each class, then simply pick the most probable class"

The survival table
(http://www.markhneedham.com/blog/tag/kaggle/)
seems like an evaluation of the possibilities of survival given possible combinations of values of the chosen features and I'm wondering if it could be an example of Naive Bayes in another name. Can someone shed light on this ?


texts: Time Series - Values recorded every 10 minutes - Fill missing values

texts:
I have a time series data from a sensor that records value periodically - sometimes - every 10 minute period, other times every 5 minute period etc. I have to find out anomalies in real time (as and when data comes) based on breach of a static threshold value.

Question is related to the minutes fall within the period and do not have any recordings.

Let's assume today the sensor is recording values every 10 minutes (APPROXIMATELY) as shown below.
What is the best possible way to handle the data in between the periodic interval. In below example, should I do a back fill with value 12 for all the points from 13:02 to 13:09?

2019-04-25T13:01:00  10,
2019-04-25T13:02:00  NAN,
2019-04-25T13:03:00  NAN,
2019-04-25T13:04:00  NAN,
2019-04-25T13:05:00  NAN,
2019-04-25T13:06:00  NAN,
2019-04-25T13:07:00  NAN,
2019-04-25T13:08:00  NAN,
2019-04-25T13:09:00  NAN,
2019-04-25T13:10:00  12,
2019-04-25T13:11:00  NAN,
2019-04-25T13:12:00  NAN,
2019-04-25T13:13:00  NAN,
2019-04-25T13:14:00  NAN,
2019-04-25T13:15:00  NAN,
2019-04-25T13:16:00  NAN,
2019-04-25T13:17:00  NAN,
2019-04-25T13:18:00  NAN,
2019-04-25T13:19:00  11,
2019-04-25T13:20:00  NAN


texts: Is there a way to aggregate frequency distribution for many runs?

texts:
In the following plot

I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is 30 but I'm only showing 5 runs.
My question is is there a way to aggregate all the frequency distributions for all the fitness values among the 30 runs? so that I can only show a single plot for all the runs.


texts: What is the best possible method/methods to determine best possible branch(rule) in a decision tree plot for the positive cases only?

texts:
I have a dataset, which I am using for loan prediction. Thus, it is pretty much clear that my dataset is imbalanced. I have used Decision Tree to plot the tree structure. Now, I want to find the optimal decision rule or the branch which shows the highest conversion rate. Can anyone please show me a path and guide me to achieve the same?

from sklearn.tree import DecisionTreeClassifier
dtc = DecisionTreeClassifier(criterion="entropy")
dtc.fit(X, y)

# Import tools needed for visualization
from sklearn.tree import export_graphviz
import pydot
# Pull out one tree from the forest
# Export the image to a dot file
export_graphviz(dtc, out_file = 'tree.dot', feature_names = X.columns, rounded = True, precision = 1)
# Use dot file to create a graph
(graph, ) = pydot.graph_from_dot_file('tree.dot')
# Write graph to a png file
graph.write_png('tree.png')


texts: Using Scorer Object for Classifier Score Method for scikit-learn

texts:
I have written my custom scorer object which is necessary for my problem and which I've called "p_value_scoring_object".

For the function sklearn.cross_validation.cross_val_score one of the parameters is "scoring", which allows to use this scorer object.

However, this option is not available for the score method of a classifier. Is sklearn just lacking that feature, or is there a way around it?

from sklearn.datasets import load_iris
from sklearn.cross_validation import cross_val_score
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state=0)
iris = load_iris()
cross_val_score(clf, iris.data, iris.target, cv=10,scoring=p_value_scoring_object)

This works. However, this doesn't:

clf.fit(iris.data,iris.target)
clf.score(iris.data,iris.target,scoring=p_value_scoring_object)


texts:
I am trying to preprocess my dataset and needs some suggestion on it.

The training data shape is : (166573, 14)

The distribution of features :

As you can see, only the first 4 columns go to different max values. Rest of the columns have either 1 or 0 value (max: 1, min: 0)

**Null Handling: **

I have dropped claims_daysaway column as most of the values are NULL and replace tier's NaN values with its mean value.

Scaling features:

I have scaled features where max value varies and left others untouched.

X['scaled_distance']= sc.fit_transform(X['distance'].values.reshape(-1,1))
X['scaled_visit_count'] = sc.fit_transform(X['visit_count'].values.reshape(-1,1))
X['scaled_tier'] = sc.fit_transform(X['tier'].values.reshape(-1,1))

Is this right approach? or should I scale all features?


texts: Can I not call my project a "Machine Learning" project if it doesn't involve a prediction at the end?

texts:
I'm doing an analysis on some data involved with the NBA right now, and I'm nearing the end of the project but I need to decide if I want to include an official prediction for future data or not.

From what I've seen online, the steps to a Machine Learning project is Gathering Data, Data Prep, Observe Data, Train Model, Evaluate Model, then Predict (on unknown data).

The first 5 steps seem to all the same as a data analysis but a "data analysis project" would focus a lot more on the observe data part and derive conclusions from that. What I did was follow all the steps, derive conclusions from the data and graphs I've observed, I ran some models on the data as well to prove some correlation between the data, and to prove that a trained model can predict data.

However, at the end I don't think a prediction of the future is necessary for my project. If I don't include this "predicting" part of my analysis, can I not technically call that a "Machine Learning" project? Because all I did was look at and analyze data and graphs and derive a conclusion from that rather than concluding with a "with our trained model we can predict these trends"?

This is a big vague so let me know if I can clarify anything, thanks!


texts: How can I use Machine learning for inter-relationship between Features?

texts:
Machine learning is used mostly for prediction and there are numerous algorithms and packages for this.

How can I use machine learning for studying inter-relationships between features? What are major packages and functions for this? Are there any packages for graphics in this area? Can artificial neural networks also be used for this purpose? If so, any particular type is specifically suited for this?

I do not want to limit to any particular language like Python or R.


texts: How do we implement a custom loss that backpropagates with PyTorch?

texts:
In a neural network code written in PyTorch, we have defined and used this custom loss, that should replicate the behavior of the Cross Entropy loss:

def my_loss(output, target):
    global classes

    v = torch.empty(batchSize)
    xi = torch.empty(batchSize)

    for j in range(0, batchSize):
        v[j] = 0
        for k in range(0, len(classes)):
            v[j] += math.exp(output[j][k])

    for j in range(0, batchSize):
        xi[j] = -math.log( math.exp( output[j][target[j]] ) / v[j] )

    loss = torch.mean(xi)
    print(loss)
    loss.requires_grad = True
    return loss

but it doesn't converge to accetable accuracies.


texts: Is it possible the model be better on a few epochs rather than hundreds of epochs?

texts:
I have very interesting experience in my CNN binary image classification. Do you think the result is by chance or there is a logic behind it?
I used InceptionV3 transfer with softmax (I know you will say why not ReLU) but it is what I did.
I trained on 100 epochs. But the result was terrible. from the training process I noticed in the 12-th epochs the result is excellent (both train accuracy and validation accuracy ).  So I trained the model on 12-th epochs. and suprisely the result on the test data was also excellent.
Does it mean on a few epoch the result is better?


texts: Forecasting non-negative sparse time-series data

texts:
I have a time-series dataset (daily frequency) representing the sales of a product to a customer over time. The sales is represented as the following:

$$[0, 0, 0, 0, 24, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 17, 0, 0, 0, 0, 9, 0, ...]$$

in which each number represents the sales of the product in a day.

The problem is that time-series forecast methods (ARMA, HoltWinters) work well for "continuous" and "smooth" data, but is not producing good results in this case.

I want to make a forecast of that series, with attention to 2 points: (1) assuring non-negative values and (2) sparse/ non-continuous data. Anyone knows how to approach this problem? What methods/ technique?

Thanks!


texts: Input contains NaN, infinity or a value too large for dtype('float32'). Without ruining Dataset

texts:
I have a datasheet that is all in Binary but sometimes there are missing cases. In one of the inputs, there is a blank, I don't want to completely remove the whole column because some of the rows have values that maybe could be useful.

Example: Column 9 has all data except for Row 15 (random row) which is blank. How can I make my decision Tree without having to remove the whole column or make the data bad (By Averaging).

I'm not sure if averaging in the situation is will strengthen the data because it might correlate to a different output.


texts: How to get a confidence score for predictions?

texts: In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?


texts: Difference between sklearn make_pipeline and imblearn make_pipeline

texts: Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.


texts: Class wise opinion for multi-label sentiment classification

texts:
I'm trying to build a model which separates positive and negative classes and assigns the label.
I have a multi-label review dataset for example:

No
Review
Label

1
Phone is good but charger is not working
[Phone, Charger]

2
Phone is not working but earphone is working fine
[Phone, Earphone]

How do I classify the opinion on this like below
Expected:

No
Review
Label
Positive
Negative

1
Phone is good but charger is not working
[Phone, charger]
[Phone]
[charger]

2
Both Phone and earphone are not working
[Phone, Earphone]
[]
[Phone, Earphone]

Please direct me on how to approach


texts: How to get correlation between two categorical variable and a categorical variable and continuous variable?

texts:
I am building a regression model and I need to calculate the below to check for correlations

Correlation between 2 Multi level categorical variables
Correlation between a Multi level categorical variable and
continuous variable
VIF(variance inflation factor) for a Multi
level categorical variables

I believe its wrong to use Pearson correlation coefficient for the above scenarios because Pearson only works for 2 continuous variables.

Please answer the below questions

Which correlation coefficient works best for the above cases ?
VIF calculation only works for continuous data so what is the
alternative?
What are the assumptions I need to check before I use the correlation coefficient you suggest?
How to implement them in SAS & R?


texts: Standardizing binary decision with other scales (Like 1-5)

texts:
In the company I work for there are 2 different evaluation metrics for a song:

Yes / No (Equivalent to like/dislike)
1-5 Scale

Customers can use both to rank songs they like. I would like to create a model that predicts the next possible songs you would like. Currently, I'm ignoring the Binary data. I wonder if there's a good way of utilizing the Binary data as tagged data [And not as a feature].
I've thought about two possible solutions:

Calculate the 1-5 Scale rating of each user and then take the (mean + std) as 'yes' and (mean - std) as 'no'.
Calculate the percentile of 1-5 Scale rating of each user and then take p75 as 'yes' and p25 as 'no'.

In case user doesn't have 1-5 scales I simply ignore him.
I guess that are better ways? (Maybe more empirically correct ways?)
Thanks.


texts:
I am working a data-set with more than 100,000 records. This is how the data looks like:

email_id    cust_id campaign_name
123         4567     World of Zoro
123         4567     Boho XYz
123         4567     Guess ABC
234         5678     Anniversary X
234         5678     World of Zoro
234         5678     Fathers day
234         5678     Mothers day
345         7890     Clearance event
345         7890     Fathers day
345         7890     Mothers day
345         7890     Boho XYZ
345         7890     Guess ABC
345         7890     Sale

I am trying to understand the campaign sequence and looking for the next possible campaign for the customers.

Assume I have processed my data and stored it in 'camp'.

With Word2Vec-

from gensim.models import Word2Vec

model = Word2Vec(sentences=camp, size=100, window=4, min_count=5, workers=4, sg=0)

The problem with this model is that it accepts tokens and spits out text-tokens with probabilities in return when looking for similarities.

Word2Vec accepts this form of input-

['World','of','Zoro','Boho','XYZ','Guess','ABC','Anniversary','X'...]

And gives this form of output -

model.wv.most_similar('Zoro')
[Guess,0.98],[XYZ,0.97]

Since I want to predict campaign sequence which occurs more frequently in combination with target word, I was wondering if there is anyway I can give below input to the model and get the campaign name in the output

My input to be as -

[['World of Zoro','Boho XYZ','Guess ABC'],['Anniversary X','World of
Zoro','Fathers day','Mothers day'],['Clearance event','Fathers day','Mothers
day','Boho XYZ','Guess ABC','Sale']]

Output -

model.wv.most_similar('World of Zoro')
[Sale,0.98],[Mothers day,0.97]

I am also not sure if there is any functionality within the Word2Vec or any similar algorithms which can help finding the next possible campaign for individual users.


texts:
I need to build parse tree for some source code (on Python or any program language that describe by CFG).

So, I have source code on some programming language and BNF this language.

Can anybody give some advice how can I build parse tree in this case?
Preferably, with tools for Python.


texts:
I'd like to know what would you do in this specific and unrealistic case appliying Knn when k = 1, k = 2 and k = 3.

Class 1 individuals: [1,1] [2,2] [2,4] [2,5]
Class 2 individuals: [3,1] [4,1] [4,2]
individual to classify: [2,1]

Plot:

I don't know if there would be any criteria rather than the generation order to choose the class it belongs.


texts: What is the correct way to compute lift in lift charts

texts:
How is "lift" computed? i was reading about "Gain and lift charts" in data science.

I picked the following example  from https://www.listendata.com/2014/08/excel-template-gain-and-lift-charts.html

I am clear on how the gain values are computed. Not clear about lift values are computed? (last column in table)


texts: Real world example of Generative model & Discriminative model

texts:
After going over the math/stats behind the Generative model &  Discriminative model, I still have no intuition about it. Does any one have a good real-world example or use
 case for Generative model and  Discriminative model ? Thanks!


texts: Checkerboard artefacts vs distinct objects in GANs

texts:
I found a very good solution for getting rid of checkerboard artefacts in GANs:
https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/190
Instead of using Transposed Convolution, use bilinear upsampling
                      nn.Upsample(scale_factor = 2, mode='bilinear'),
                      nn.ReflectionPad2d(1),
                      nn.Conv2d(ngf * mult, int(ngf * mult / 2),
                                         kernel_size=3, stride=1, padding=0),

I replaced padding=0 with padding=1 to keep the same size of the image. The tradeoff, though, is that the Generator doesn't learn to evolve distinct objects as with Transposed Convolution. For example, on Street View House Number (SVHN) dataset, the one on the left was made with Transposed Convolution, the one on the right with Upsampling:

I haven't found any good explanations on the difference in results so far.


texts:
I normalize time series data using a sliding window to be fed into a Neural Net. I normalize each window independently using Z-score. (Because on the entire dataset proven to be useless. Because on later on batch level is random due to shuffling.)
Z-score normalize data as a pattern, as a deviation from the mean measured in the portion of the standard deviation. I apply the function on every time window (and not on batch). It would return the same for:

Window: [1, 5, 2] Mean: 2.66 STDev: 2.081 Result[-0.800, 1.120, -0.320]
Window: [10, 50, 20] Mean 26.66 STDev: 20.81 Result[-0.800, 1.120, -0320]

This is not desired. The magnitude of the input carries information that is useful and I need to preserve it while applying normalization benefits. My best idea is:

  Z-Score(i) / (1 / Mean(Window(n)))

Where (i) is the item in Window(n)
and (n) represents the Nth sliding window

This results an output that keeps the magnitude:

[-2.135, 2.890, -0.854]
[-21.35, 29.89, -8.540]

This output keeps the Z-score benefits while preserves the magnitude. Can you help me out if this a fair approach?


texts: why should i do target encoding within cv loop?

texts:
i wish to use target encoding, using the category encoders sklearn library. I don't really understand why it is necessary to include this as a step in a sklearn pipeline WITHIN the cross validation loop?
e.g. this example here does so Target encoding with KFold cross-validation - how to transform test set?
my methodoloigy is similar to the one in the link except i do not use any smoothing. my dataset is quite large around 300-500k.
however looking at my result below does this not indicate over fitting?


texts: Multidimensional Dynamic Time Warping Implementation in Python - confirm?

texts:
I believe that I implemented MDTW in python here but I don't know if I did it correctly. The results seem intuitive. Can someone look at this code and tell me if you see anything wrong?

A lot of the papers I read on the subject kinda go over my head, but this just takes the mse of all column vectors in matrix s1 and s2 at each time point inside a given window w instead of single dimensional dynamic time warping which just takes the mse error of each value between two vectors and finds the smallest one.

I use panda's Panel, which is essentially a 3D version of a DataFrame.

Also, after I have this, I want to do clustering with some multidimensional time series. Thoughts on which clustering algorithm to run? Kmeans? Hierarchical? I'll start by building a dendrogram at least.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

def MDTWDistance(s1, s2, window=10, num_columns=1):
    DTW={}

    w = max(window, abs(len(s1)-len(s2)))

    for i in range(-1,len(s1)):
        for j in range(-1,len(s2)):
            DTW[(i, j)] = float('inf')
    DTW[(-1, -1)] = 0

    for i in range(len(s1)):
        for j in range(max(0, i-w), min(len(s2), i+w)):
            #print "Finding Distance of", s1.loc[i], s2.loc[j]
            dist= mdist(s1.loc[i], s2.loc[j], num_columns)
            #print "Dist", dist
            #print i, j, dist
            DTW[(i, j)] = dist + min(DTW[(i-1, j)],DTW[(i, j-1)], DTW[(i-1, j-1)])

    return np.sqrt(DTW[len(s1)-1, len(s2)-1])

def mdist(a, b, num_col):
    dist = 0
    for col in range(num_col):
        #print "Finding Distance of", a[col], b[col]
        dist = dist + (a[col]-b[col])**2
    return dist

x=np.linspace(0,50,100)
ts1=pd.Series(3.1*np.sin(x/1.5)+3.5)
ts2=pd.Series(2.2*np.sin(x/3.5+2.4)+3.2)
ts3=pd.Series(0.04*x+8.0)
ts4=pd.Series(0.048*x+8.6)
ts5=pd.Series(-0.17*x+4.1)
ts6=pd.Series(-0.14*x+4.5)

ts1.plot()
ts2.plot()
ts3.plot()
ts4.plot()
ts5.plot()
ts6.plot()

plt.ylim(-4,12)
plt.legend(['ts1','ts2','ts3','ts4','ts5','ts6'])
plt.show()

timeSeries = pd.Panel({0:pd.DataFrame(np.transpose([ts1, ts2])),
                 1:pd.DataFrame(np.transpose([ts3, ts4])),
                   2:pd.DataFrame(np.transpose([ts5, ts6]))
                  })

print "0 and 1:",MDTWDistance(timeSeries[0], timeSeries[1],window=10, num_columns=2)
print "0 and 2:",MDTWDistance(timeSeries[0], timeSeries[2],window=10, num_columns=2)
print "1 and 2:",MDTWDistance(timeSeries[1], timeSeries[2],window=10, num_columns=2)

The output is as follow:

0 and 1: 89.354619036
0 and 2: 58.8268328591
1 and 2: 133.434513377

With the graph:

EDIT: I found the distance using an MDTW package in R with the following code:

import rpy2.robjects.numpy2ri
from rpy2.robjects.packages import importr
rpy2.robjects.numpy2ri.activate()

# Set up our R namespaces
R = rpy2.robjects.r
DTW = importr('dtw')

# Generate our data
idx = np.linspace(0, 2*np.pi, 100)
template = np.cos(idx)
query = np.sin(idx) + np.array(R.runif(100))/10

for i,j in ([0,1], [0,2], [1,2]):
    print "From", i, "to", j, R.dtw(np.transpose(np.array(timeSeries[i])), np.transpose(np.array(timeSeries[j])), keep=True).rx('distance')[0][0]

which output:

From 0 to 1 186.623310713
From 0 to 2 119.769089068
From 1 to 2 272.849560995

So these numbers are about double the distances I have... so I'm not far off, but there's something wrong with what I'm doing. Halp.


texts: With unbalanced class, do I have to use under sampling on my validation/testing datasets?

texts:
I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.

The thing is that, as you may know, the dataset is very unbalanced : Around 1300 negative examples (non click) for 1 positive example (click).

This is what I do:

Load the data
Split the dataset into 3 datasets :
    A = Training (60%)
    B = Validating (20%)
    C = Testing (20%)
For each dataset (A, B, C), do an under-sampling on each negative class in order to have a ratio of 5 (5 negative example for 1 positive example). This give me 3 new datasets which are more balanced:
    A’
    B’
    C’

Then I train my model with the dataset A’ and logistic regression.

My question are:

Which dataset do I have to use for validation ? B or B’ ?
Which dataset do I have to use for testing ? C or C’
Which metrics are the most relevant to evaluate my model? F1Score seems to be a well used metric. But here due to the unbalanced class (if I use the datasets B and C), the precision is low (under 0.20) and the F1Score is very influenced by low recall/precision. Would that be more accurate to use aucPR or aucROC ?
If I want to plot the learning curve, which metrics should I use ? (knowing that the %error isn’t relevant if I use the B’ dataset for validating)

Thanks in advance for your time !

Regards.


texts:
I'm new to machine learning but am interested in decoding applications like the Higgs Boson challenge on Kaggle. I just wondered, what are the best algorithms or approaches for classifying such a small signal in large amounts of noise? I think supervised and unsupervised are equally fine to use for my project, but hearing about your thoughts in general would help a lot!

I wanted to start with the Higgs Boson challenge to explore machine learning methods for improving discovery significance of scientific data. The experiment and data are available at https://www.kaggle.com/c/higgs-boson . The code used by the winner was released here: https://www.kaggle.com/c/higgs-boson/forums/t/10425/code-release .

I know the basic theory of supervised and unsupervised learning, and have studied and used Linear Regression and SVM. I've read about neural networks also but on a basic approach level. I'm just learning about deep learning coming from the natural sciences, to improve data analysis, and found SVM very useful but otherwise don't know more advanced techniques.


texts: How to delete entire row if values in a column are NaN

texts:
I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this:

Age  Height  Weight  Gender
12    5'7     NaN     M
NaN   5'8     160     M
32    5'5     165     NaN
21    NaN     155     F
55    5'10    170     NaN

I want to remove all the rows where 'Gender' has NaN values. The output i'd like:

Age  Height  Weight Gender
12   5'7     NaN     M
NaN  5'8     160     M
21   NaN     155     F

Thanks in advance!


texts: When should I use StandardScaler and when MinMaxScaler?

texts:
I have a feature vector with One-Hot-Encoded features and with continous features.

How can I decide now, which data I shall scale with StandardScaler and which data scale with MinMaxScaler? I think I do not have to scale the one-hot-encoded anyway because they are already between 0 and 1.

(I use afterwards a MLPClassifier)


texts: Scaling features in artificial neural networks

texts:
So it is a well known thing that it is a good idea to scale features/training samples in the training set, so that the values do not differ too much in the absolute sense. For example we want to train a neural network to learn a simple quadratic function y = x*x.

The training data, say, looks like x = [1, 2, 3, 4, 5, 6] for input data and y = [1, 4, 9, 16, 25, 36] for target data.

Obviously, one would scale both the input and the output data by something like x := x/6 and y := y/36, so all the training values live in the [0:1] range.

My question is, what if I then want to predict the output value for the input 7?
If I feed 7 / 6 into the network and then rescale the output by doing output := output * 36, I will not get the correct result. How to resolve such an issue?


texts:
Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our weights. This completes processing 1 minibatch.

I read these posts

[1]
[2],
about padding entries in a minibatch so they have same length
and about preserving the cell state but the following is still unclear to me:

Question part a:

How a minibatch entity would look like for LSTM?
Say, I want it to reproduce Shakespeare, letter by letter (30 characters to choose from).

I launch LSTM, let it predict for 200 characters of a poem, then perform back propagation. (hence, my LSTM works with 200 timesteps). Does this mean my minibatch consist of 1 example whose length is 200?

Question part b:

If I wanted to launch 63 other minibatches in parallel, would I just pick 63 extra poems?  (Edit: Original answer doesn't mention this explicitly, but we don't train minibatches in parallel. We train on 1 minibatch, but train its examples in parallel)

Question part C:

If I wanted each minibatch to consist of 10 different examples, what would such examples be, and how would they be different from 'what I perceive as a minibatch'?


texts: How to normalize data of a different nature?

texts:
I am working a price prediction LTSM model for the stock market. I am using multiple features: Open, Close, High and I would like to add the Volume.

The 3 first features are of the same nature but the volume presents much higher values.

What would be the safest way to keep consistency during the normalization process?


texts: Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

texts:
I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata.

They could very well be relational records in a table, except for the fact that they have one array field with IDs pointing to child records.

What implementation looks more solid?

JSON objects on a PostgreSQL database (just one large table with one column, namely the JSON object)
JSON objects on a MongoDB
Explode the JSON objects into columns and use arrays on PostgreSQL

I want to maximize performance in joins, so I can massage the data and explore it until I find interesting analyses, at which point I think it will be better to transform the data into a form specific to each analysis.


texts: How to compare performance of Cosine Similarity and Manhatten Distance?

texts:
I'm doing clustering of documents by applying k-Means on the word-vectors. To measure the cluster quality, I calculate David Bouldin Index for different k's. I tried two different distance measures, Cosine Similarity and Manhatten Distance, and get quite different values:

Cosine Similarity: ~0.8 to ~0.6
Manhattan Distance: ~0.3 to ~0.2

Can these values be compared directly? (Does Manhattan really performs a lot better here?) Or is there another way to compare the clustering results of the two different measures?


texts: What are limitations of decision tree approaches to data analysis?

texts:
The major limitations of decision tree approaches to data analysis that I know of are:

Provide less information on the relationship between the predictors and the   response.
Biased toward predictors with more variance or levels.
Can have issues with highly collinear predictors.
Can have poor prediction accuracy for responses with low sample sizes.

Are there any others? Are they robust to traditional statistical assumptions such as homogeneity, normality, independence?


texts: Bagging vs Dropout in Deep Neural Networks

texts: Bagging is the generation of multiple predictors that works as ensamble as a single predictor. Dropout is a technique that teach to a neural networks to average all possible subnetworks. Looking at the most important Kaggle's competitions seem that this two techniques are used together very often. I can't see any theoretical difference besides the actual implementation. Who can explain me why we should use both of them in any real application? and why performance improve when we use both of them?


texts: I have a dataset (~52k rows) with a column containing just pure sentences (upper and lowercase, with punctuation and stop words) in each row. What can I do to represent this data in a meaningful way for my model. I tried using a CountVectorizer but it's giving me a dataframe with way too many columns and the model is having a hard time using it. What other options do I have?


texts: Choosing a model for dataset with categorical variables

texts:
I have a question about the type of model which I should use for a dataset I have.

The dataset has a total of 7 independent variables and 1 dependent variable which I need to predict.

Out of the 7 input variables, 6 of them are categorical and 1 is a date column. Now I have encoded the categorical columns using label encoding and converted them into numerical values. Now I’ve used a simple linear regression model on this dataset and achieved a normalized RMSE value of 0.11.

If I want to improve this accuracy, how do I go about doing it? How can I derive upon the kind of models that I can use considering the data set I have?

The data is mostly about forecast revenues for each product group. That's why I have those categorical columns which represent the hierarchy of the product groups.


texts: Error when using MAX in Apache Pig (Hadoop)

texts:
I am trying to calculate maximum values for different groups in a relation in Pig. The relation has three columns patientid, featureid and featurevalue (all int).
I group the relation based on featureid and want to calculate the max feature value of each group, heres the code:

grpd = GROUP features BY featureid;
DUMP grpd;
temp = FOREACH grpd GENERATE $0 as featureid, MAX($1.featurevalue) as val;

Its  giving me  Invalid scalar projection: grpd Exception. I read on different forums that MAX takes in a "bag" format for such functions, but when I take the dump of grpd, it shows me a bag format. Here's a small part of the output from the dump:

(5662,{(22579,5662,1)})
(5663,{(28331,5663,1),(2624,5663,1)})
(5664,{(27591,5664,1)})
(5665,{(30217,5665,1),(31526,5665,1)})
(5666,{(27783,5666,1),(30983,5666,1),(32424,5666,1),(28064,5666,1),(28932,5666,1)})
(5667,{(31257,5667,1),(27281,5667,1)})
(5669,{(31041,5669,1)})

Whats the issue ?


texts: Which deep learning network would be suitable for classifying this kind of text data?

texts:
I have some experience with images and have played around with image classification using CNN's but have limited knowledge when it comes to text data.
The input that I currently want to classify is written as:
hjkhghkgfghjkhghkgfghfefdefdcdefghjkjh-hjhgfe
fdcd-dd-fdc-dad-ad-dfe-cde-dggf-ghd-gg-bcd
hjkhghkgfghjkhghkgfghfefdefdcdefghjkjh-gh-gfed
dh-hg-gf-gh-dh-hg-gf-gh-hkhg-kh-hg-gf-gh-hkhg-kh-hg-gf-ghh-hgfg-dfd-dc-fgf-gh

I have over 2000 rows of this data, that needs to be classified. I know that for regular text data RNN networks and LSTM cells have been known t be very effective. Using RNN+LSTM good results can be achieved by pre-processing the data using the usual approaches such as stemming, lemmitization, stop word filtering, tokenization etc. But the same cannot be applied to the text data I have.
Would RNN and LSTM still work on my data? If not which networks do you guys suggest I explore for such a task?


texts: Multi target classification for different types of target variables

texts:
I am new to machine learning and I got this task in my university. I have a dataset with over 100 columns and two target variables: $target1$ is categorical i.e. $0$ or $1$ and $target2$ is continuous i.e. values in range $0 \space to \space 100$.

How can I predict this type of problem?

I tried using multi-output classification from sklearn using the Random forest as an ensembler and it is predicting nicely for continuous target variable but not for categorical target variable.


texts: Dot product of two matrices in NLP how can i get this error be solved

texts:
from sklearn.metrics.pairwise import linear_kernel
sim_matrix = linear_kernel(tfidf_matrix, tfidf_matrix)

when I try to get dot product I am getting this  errro
MemoryError                               Traceback (most recent call last)
<ipython-input-19-2c4d43d4a89e> in <module>
      1 from sklearn.metrics.pairwise import linear_kernel
----> 2 sim_matrix = linear_kernel(tfidf_matrix, tfidf_matrix)

~\anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in linear_kernel(X, Y, dense_output)
   1002     """
   1003     X, Y = check_pairwise_arrays(X, Y)
-> 1004     return safe_sparse_dot(X, Y.T, dense_output=dense_output)
   1005
   1006

~\anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
     70                           FutureWarning)
     71         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 72         return f(**kwargs)
     73     return inner_f
     74

~\anaconda3\lib\site-packages\sklearn\utils\extmath.py in safe_sparse_dot(a, b, dense_output)
    151             ret = np.dot(a, b)
    152     else:
--> 153         ret = a @ b
    154
    155     if (sparse.issparse(a) and sparse.issparse(b)

~\anaconda3\lib\site-packages\scipy\sparse\base.py in __matmul__(self, other)
    558             raise ValueError("Scalar operands are not allowed, "
    559                              "use '*' instead")
--> 560         return self.__mul__(other)
    561
    562     def __rmatmul__(self, other):

~\anaconda3\lib\site-packages\scipy\sparse\base.py in __mul__(self, other)
    478             if self.shape[1] != other.shape[0]:
    479                 raise ValueError('dimension mismatch')
--> 480             return self._mul_sparse_matrix(other)
    481
    482         # If it's a list or whatever, treat it like a matrix

~\anaconda3\lib\site-packages\scipy\sparse\compressed.py in _mul_sparse_matrix(self, other)
    514
    515         indptr = np.empty(major_axis + 1, dtype=idx_dtype)
--> 516         indices = np.empty(nnz, dtype=idx_dtype)
    517         data = np.empty(nnz, dtype=upcast(self.dtype, other.dtype))
    518

MemoryError: Unable to allocate 2.04 GiB for an array with shape (546860044,) and data type int32


texts: What does a conservative technique mean in the context of neural networks?

texts:
I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph

  LRP is a conservative technique, meaning the magnitude of any output y is conserved through the backpropagation process and is equal to the sum of the relevance map R of the input layer. This property holds for any consecutive layers j and k, and by transitivity for the input and output layer.

What does this even mean? I can't understand what the author is trying to say. I understand what backpropagation is but not what is being said related to backpropagation!


texts:
I have sales training data from 2019-06 to 2020-06 and I have to predict sales from 2020-06 to today. The pandemic had a great impact on sales. Are there methods to make my ML forecasting models more robust? Here are some ideas:

Add a binary variable, pandemic, being 1 when date > 2020-03 and 0 otherwise.
Alternatively, define two variables hard_pandemic and soft_pandemic describing respectively the data between March/April 2020 and data after May 2020.
Weigh the pandemic data more, since they describe better the current situation.

Are there drawbacks to these methods? Are there alternative ways to improve this?


texts: Could my AI classifier be extended to categorical data?

texts:
I created a new AI classifier.
Physics Based Classifier


texts: How to best visualize or capture time interval between lab measurements?

texts:
I have a table like as shown below
subject_id   lab_test_id    lab_test_date       value    difference

"10005606"  "20364112"  "2143-12-06 02:32:00"   "1.3"   "13:10:00"
"10005606"  "20364112"  "2143-12-06 15:42:00"   "1.3"   "02:02:00"
"10005606"  "20364112"  "2143-12-06 17:44:00"   "1.3"   "10:11:00"
"10005606"  "20364112"  "2143-12-07 03:55:00"   "1.3"      NULL
"10005866"  "20364112"  "2149-10-01 19:30:00"   "2.1"   "07:42:00"
"10005866"  "20364112"  "2149-10-02 03:12:00"   "2.2"   "08:51:00"
"10005866"  "20364112"  "2149-10-02 12:03:00"   "2.1"   "04:59:00"
"10005866"  "20364112"  "2149-10-02 17:02:00"   "1.6"     NULL

the difference column indicates the time difference between each of the lab tests
Now, the doctor likes to know how often these measurements are done for each patient?
How can I best convey this information? Should I capture the average difference between lab tests?
For example, for patient = 10005606, it is 8 hours. Meaning adding all different values and divide by total number of records. 25 hours, 30 mins (approx)/3 records = 8 hours approx
Is there any other better way to represent this using median or any other measure etc?
Can guide me with this?


texts: Interpretation of scikit-learn one class svm scores

texts:
How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way to tell when one sample is "more anomalous" than other? The predict() and decision_function() functions have sign information, yet the score_samples function does not have this.
from sklearn.svm import OneClassSVM

X = [[0], [0.44], [0.45], [0.46], [1]]
clf = OneClassSVM(gamma='auto').fit(X)

clf.predict(X)
# array([-1,  1,  1,  1, -1])

clf.score_samples(X)
# array([1.7798..., 2.0547..., 2.0556..., 2.0561..., 1.7332...])
```


texts:
I am not able to figure out what kind of machine learning is this:
Training set: consists of sentences with object labels for object phrases
Example:
"This is a black chair. It is next to a large bed."

Phrases 'This', 'black chair', 'it' were annotated with the label "chair" and phrase 'large bed' was annotated with the label 'bed'. There are 18 available labels that can be assigned.
For an unannotated sentence, I want to predict the labels for each phrase in the sentence.
For example:
There is a study table in the corner of the room, behind it is a small chair.

I would like the model to predict a label (available 18 labels) for each phrase that represents an object in the above sentence.
Expected output:
'study table', 'it' -----> label 'table'

'small chair' -----------> label 'chair'

Is this a classification, regression, or a different kind of problem?


texts: Using the cosine activation function seems to perform very badly

texts:
I have created a neural network to classify the MNIST handwritten numbers dataset. It is using softmax as the activation function for the output layer and various other functions for the hidden layer.

My implementation with the help of this question seems to be passing the gradient checks for all activation functions but when it comes to the actual run with my training data for an exemplary run of 10 iterations I get an accuracy of about 87% if I use sigmoid or tanh as the activation function for the hidden layer, but if I use cosine it returns an accuracy of 9%. Training the network with more iterations (100, 200, 500) does not have any effect either and in fact my minimization function does not manage to move below 2.18xxx for the cost function no matter how many epochs pass.

Is there some pre-processing step that I need to perform before using cosine if not why is it that this activation function works so badly?


texts:
I have sentences telling me to who a shop is opened to:

"cats, dogs or birds" (1)
"young dogs with collar" (2)
"old cats or yellow birds" (3) etc...

I would like to design an algorithm that will change this sentences to a tree representation of the logic in it:

(1) = (cat) or ((dog) or (bird))
(2) = (young) and (dog) and (collar)
(3) = ((cat) and (old)) or ((bird) and (yellow))

What do you think will work the best? LSTM maybe? How can I have this representation as a result?


texts: From unsupervised to supervised in fraud detection

texts:
I have a question. I am working on the fraud detection domain. And I have data from imports to the country. As you can get from the title, I have unsupervised data. I do not know that the record is fraudulent. But I have domain knowledge what fraudulent behavior might have and (smuggling vs) I want to randomize some records and flag them as fraudulent behavior (meaningful randomness not all the features). With this information, I want to train my neural network to distinguish fraudulent behaviors. The approach I have in mind so far:

Train, Test split
In train data, create fraud data with randomizing algorithm (das,
Schneider 2007 APD)
Train Neural network on train data
In test data create fraud data with randomizing algorithm - this
time with different algorithm ( R. Triepels, H. Daniels, and A.
Feelders, “Data-driven fraud detection in international shipping,”
Expert Systems with Applications, vol. 99, pp. 193–202, 2018.)
test the model

I found some similar approaches (from unsupervised to supervised fraud detection, (Ghosh, Anup K., and Aaron Schwartzbard. "A Study in Using Neural Networks for Anomaly and Misuse Detection." USENIX security symposium. Vol. 99. 1999.) But I want to see more papers or a thesis on this topic. Anyone have advice or comment on this topic

Thanks


texts: Can the number of epochs influence overfitting?

texts:
I am using a convolution neural network ,CNN. At a specific epoch, I only save the best CNN model weights based on improved validation accuracy over previous epochs.

Does increasing the number of epochs also increase over-fitting for CNNs and deep learning in general?


texts: I'm trying to do a time series model without a datetime field in python. Is this possible?

texts:
I have a dataset with data like this:
Day       Revenue
1          1.2
2          1.5
3          1.1
4          1.34

I want to do a time series model on it, but am getting this error:
ValueError: view limit minimum -35.45 is less than 1 and is an invalid Matplotlib date value. This often happens if you pass a non-datetime value to an axis that has datetime units
When I plt, it assigns all of the date to 1/1/1970. I understand why, because it's not a date time field. Out of curiosity, I tried converting the Day column to a datetime, but it assigned every day to 1/1/70. Is there a way to either convert the column to a datetime field and have it assign a new date starting with a specific date (say 1/1/2017, 1/2/2017, etc) or is there a work around when you just have the day counts (1,2,3,4)?


texts: Should I use epochs > 1 when training data is unlimited?

texts: If I have virtually endless training data (it's synthesized) is there still purpose in having epochs? I.e. training on the same samples multiple times?


texts: What is the best way to visualize the relationship two categorical variables

texts:
I am currently working on an ambulance dataset and one of my tasks is to find when a patient was misdiagnosed by the call dispatcher.

I have two codes; a dispatch code(what the dispatcher believes is wrong with the patient) and a hospital code(what a doctor decides the actual diagnosis is in hospital)

I'm looking for a way of visualizing the relationship between the two codes i.e. given ambulance code x what is the probability of each ambulance code being the outcome.

This can be computed using SQL pretty easily but I'm looking for a way to visualize/cluster it would be great. Any help would be appreciated.

EDIT:
Some good feedback in the comments

firstly the dimensionality:
Dispatch code can take 1 of 1722 unique values

hospital codes can take 1 of 1058 unique values

The two codes are entirely different an example would be as follows

Dispatcher 17D03:Unconscious
Hospital R41:Other symptoms and signs involving cognitive functions and awareness

My interest is in visualizing the relationship. So for example given a dispatcher code of unconscious what are the most common hospital codes?

Again this is fairly easy to calculate numerically but a visualization would make it easier to explain to my stake holders.


texts: Network overfitting after adding batch normalization

texts:
After I added BN between my model's hidden layers, my neural network started to overfit faster and because of this it didn't find optimal solution for the problem.
My theory is BN speeded up the training and made neural network converge to training global minimum faster before exploring mininum which fits validation set the best. But BN should also regularise model a little bit which doesn't happen in this case.
I have FFNN which is designed to predict football (soccer) outcomes and what bets are most profitable to take. Because I need odds in some metrics I had to create wrapper for categorical loss and crossentropy but it's the same as keras' ones underneath. Model is setup in code like this:
factor = 0.002
rate = 0.4

model = tf.keras.models.Sequential()
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(rate))
model.add(keras.layers.Dense(256, activation='relu',
                             kernel_regularizer=l2(factor),
                             kernel_initializer=tf.keras.initializers.he_normal()))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(rate))
model.add(keras.layers.Dense(128, activation='relu',
                             kernel_regularizer=l2(factor),
                             kernel_initializer=tf.keras.initializers.he_normal()))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(rate))
model.add(keras.layers.Dense(64, activation='relu',
                             kernel_regularizer=l2(factor),
                             kernel_initializer=tf.keras.initializers.he_normal()))
model.add(keras.layers.Dense(3, activation='softmax', kernel_initializer=tf.keras.initializers.he_normal()))
model.compile(loss=categorical_crossentropy_with_bets,
                  optimizer=keras.optimizers.Adam(learning_rate=0.0003),
                  metrics=[categorical_acc_with_bets])

And here's loss and accuracy of network with BN:

and here's without bn between hidden layers:


texts: What's the difference between data products and intelligent systems?

texts: Basically, both are software systems that are based on data and algorithms.


texts: How to choose between variations of EfficientDet

texts:
In the Tensorflow 2 Object Detection API model ZOO, there is 7 versions of the EfficientDet:

Being new to deep learning, I don't understand very well what is the difference and how to choose the best one for my application (detecting buildings from satellite imagery). I haven't been able to find much info on this online.
Question: How do these differ and how does one decide which one to choose?


texts: Effect of embedding dimension in performance of embedding algorithms

texts:
Consider a simple Matrix Factorization or any embedding algorithm. It is obvious that a very small embedding dimension does not contain enough information so the algorithm will have low performance and enlarging the dimension will improve the performance. However, as I tried various algorithms, enlarging the dimension more than a threshold decreases the performance of the algorithm. Why does this happen?

I know using larger embedding while you can get the same result with smaller ones does not make sense, however, I am just curious to find out more about that.


texts: What feature engineering is necessary with tree based algorithms?

texts:
I understand data hygiene, which is probably the most basic feature engineering. That is making sure all your data is properly loaded, making sure N/As are treated as a special value rather than a number between -1 and 1, and tagging your categorical values properly.

In the past I've done plenty of linear regression analysis. So feature engineering mainly concerned with:

Getting features into the correct scale using log, exponent, power transformations
Multiplying features: if you have height and width, multiply to make area
Selecting features: remove features based on P value

But, for LightGBM (and Random Forest) it seems like the scale of the features doesn't matter because orderable items are ordered and then randomly bisected. Interactions of features don't matter because one of the weak classifiers should find it if it is important. And feature selection isn't important because if the effect is weak then those classifiers will be attenuated.

So, assuming you can't find more data to bring in, what feature engineering should be done with decision tree models?


texts:
I need to build a time series model with explanatory variables, and ARIMAX seems to be the one that comes up most frequently in practice, based on my survey of related work.

I know ARX solves a similar problem, but I'm having trouble wrapping my mind around the practical differences in what an ARX representation of my data would have compared with an ARIMAX approach.

I know ARX lacks the moving average component, but I'm curious whether anyone can point me toward some best practice for choosing one approach over the other.

Are there certain characteristics in my data I should look for to make an informed decision?


texts: SKLEARN SGDClassifier prediction accuracy hint?

texts:
There is a function predict but is it possible to also hint how much is the predicted category probable? Like prediction of category 1 with 90% confidence, or 2 with 30% confidence etc.
Without this I don't know how much is the prediction reliable.
I am using it together with TfidfVectorizer. I classify the texts into categories (like Business, Sport...)
text_clf = Pipeline([
      ('tfidf', TfidfVectorizer(max_df=0.35, min_df=1, max_features=n_features, stop_words=stop_words, ngram_range=(1,3))),
      ('clf', SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, random_state=random_state, max_iter=5, tol=None)),
])

Edit:
It seems to me there is a method .predict_prob() for that, but I need to change the loss function to modified_huber which changes the general accuracy. Otherwise the method does what I expected, it assigns the probabilities of predicted class (it gives probability to all classes possible).
If there is any other solution you can tell.
Big problem now is it decreased accuracy from .67 to .5 when i changed loss function from hinge to modified_huber


texts:
I am reading a book on machine learning for undergraduate. I am actually confused on linear regression flexibility as the say:

Occasionally, linear regression will fail to recover a good solution
for a data set. While this may be because our data doesn’t actually
have predictive power, it might also just indicate that our data is
provided in a format unsuitable for linear regression.

I read some questions here related to predictive power and I noticed that it's all about the model produced. What do we mean when we talk about the predictive power of a dataset?
I think of it as there is no relationship between features (as linear regression is intented to learn the relationship between the inputs $X$ and the output $Y$). But I am not convinced of this answer yet.


texts: What are recommended ways\tools for processing large data from Excel Files?

texts:
A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time data to the millisecond. The processing also has to be efficient in view of the large data volume. What is the recommended methodology and tool to handle this?

I've seen others convert excel to CSV, and then confining their analysis to the CSV itself. Is this method advantageous in the sense that CSV is faster to read in, and can be processed by a larger toolset? Is there any powerful tool\library that can do batch conversion, and even extract custom formatting data?

Thanks and Regards


texts: How do ML model retain/store the learning(s)?

texts:
In other words, if the model after training and testing is ready for making future predictions, it must be storing the learning(s) somewhere in memory or disk or cache (or I really don't know).

So, where (how) is it exactly stored?

Hope I asked my doubt clearly for experts to advise.


texts: The wiki page for bootstrapping says that you use it in the case where the underlying distribution is unknown. Why is bootstrapping, or sampling with replacement, better than just calculating the variance and other properties from the data directly?


texts: Feature selection with information gain (KL divergence) and mutual information yields different results

texts:
I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees, i.e. the Kullback-Leibler divergence.

My data (class and features) is all binary.

All sources I could find state, that MI and IG are basically "two sides of the same coin", i.e. that one can be tranformed into the oher via mathematical manipulation. (For example [source 1, source 2])

Yet, when I rank my features using the two measures they do not result in the same ranking order. But if the two measures are equivalent, shouldn't the ranking be the same?

Can someone help me understand why the rankings are different?


texts: Difference between advantages of Experience Replay in DQN2013 paper

texts:
I've been re-reading the Playing Atari with Deep Reinforcement Learning (2013) paper. It lists three advantages of experience replay:

  This approach has several advantages over standard online Q-learning
  [23]. First, each step of experience is potentially used in many
  weight updates, which allows for greater data efficiency. Second,
  learning directly from consecutive samples is inefficient, due to the
  strong correlations between the samples; randomizing the samples
  breaks these correlations and therefore reduces the variance of the
  updates. Third, when learning on-policy the current parameters
  determine the next data sample that the parameters are trained on. For
  example, if the maximizing action is to move left then the training
  samples will be dominated by samples from the left-hand side; if the
  maximizing action then switches to the right then the training
  distribution will also switch. It is easy to see how unwanted feedback
  loops may arise and the parameters could get stuck in a poor local
  minimum, or even diverge catastrophically [25]. By using experience
  replay the behavior distribution is averaged over many of its previous
  states, smoothing out learning and avoiding oscillations or divergence
  in the parameters. Note that when learning by experience replay, it is
  necessary to learn off-policy (because our current parameters are
  different to those used to generate the sample), which motivates the
  choice of Q-learning.

I am confused on how the second and third advantages differ. Isn't the third advantage just another case of breaking correlation?

Thank you in advance for your help!


texts: Converting RDD to spark data frames in python and then accessing a particular values of columns

texts:
I want to access values of a particular column from a data sets that I've read from a csv file. The datasets are stored in pyspark RDD which I want to be converted into the DataFrame.
I am using the below code :

from pyspark.sql import SQLContext
sqlc=SQLContext(sc)
df=sc.textFile(r'D:\Home\train.csv')
df=sqlc.createDataFrame(df)

but it is showing error:

Can not infer schema for type: <class 'str'>

First 2 rows of df are :

['"id","product_uid","product_title","search_term","relevance"',
 '2,100001,"Simpson Strong-Tie 12-Gauge Angle","angle bracket",3']

I think the first row is creating this problem. Moreover I want to create data frame which stores the values from 2nd row to last.(Not the first row because it will be the header). How can I achieve this ? I've searched for it but could not find any solution. Thanks in advance.


texts: Calculation and visualization of islands of influence

texts:
I recently saw the graph of correlated technologies from the SO Developer Survey, which was pretty; reproduced below:

I have a weighted graph that represents the amount of communication between pairs of people. I'd like to create a similar visualization as above. I want to be able to tease out "islands" of people with high influence among each other, but little influence elsewhere.

To make this concrete, let's say I have four animals: dog, wolf, cat, lion. The canines talk a lot amongst themselves; the felines do too, but members of one group only rarely talk to members of the other group. So my influence graph (which is symmetric, hence the upper triangular matrix) looks like:

+------+-----+------+-----+------+
|      | Dog | Wolf | Cat | Lion |
+------+-----+------+-----+------+
| Dog  | ∞   | 100  | 1   | 1    |
| Wolf | -   | ∞    | 3   | 2    |
| Cat  | -   | -    | ∞   | 130  |
| Lion | -   | -    | -   | ∞    |
+------+-----+------+-----+------+

I know how to create graphs whose edges are visually represented by the weight between two nodes, but I am not sure how to account or calculate islands of influence. Are there general visualization techniques to apply to a data set like this, and also for figuring out islands of influence?

Thanks in advance!


texts:
How can I estimate

$y_{i,t}=β_{i}x_{i,t}+ϵ_{i,t} \\
β_{i}=γ_{i}z_{i}+η_{i}$

in R ? Moreover, if I have splitted my data set to a train set which will contain 80% of all the i's and then I want to forecast the $y_{i,t}$ for the rest 20% i's  $\forall t$, how can I do it in R ?

Thank you in advance


texts: Nearest Neighbors on mixed data types in high dimensions

texts:
I would like to be able to use nearest neighbors to attempt to find the most similar samples to a subclass of samples (think treated vs untreated) in a dataset with continuous, categorical, and text features.

Toy data set:

import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, QuantileTransformer
from sklearn.neighbors import NearestNeighbors
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.decomposition import TruncatedSVD

np.set_printoptions(suppress=True)

a = [20, 100, 10000, 500]
b = [1, 2, 3, 2]
c = ['dog', 'cat', 'foo', 'cat']
d = ['apple apple fruit',
     'mercedes bmw chevrolet',
     'monster dragon snake',
     'mercedes chevrolet bmw buick']

z = [a,b,c,d]

names = {0: 'col1', 1:'col2', 2:'col3', 3:'col4'}

X = pd.DataFrame(z).T
X = X.rename(names, axis='columns')
X

Will create:

   col1    col2 col3    col4
0   20      1   dog     apple apple fruit
1   100     2   cat     mercedes bmw chevrolet
2   10000   3   foo     monster dragon snake
3   500     2   cat     mercedes chevrolet bmw buick

As we can see, samples 1 and 3 are the most related. They have many of the same vocabulary, share two labels (col2 + col3) and considering the distribution of col1 they are fairly close together. We transform them into a feature array and ask for nearest neighbors like so:

numeric = ['col1']
numeric_transformer = Pipeline(steps=[('scaler', QuantileTransformer())])

cat = ['col2', 'col3']
cat_transformer = Pipeline(steps=[('onehot', OneHotEncoder())])

text = ['col4']
text_transformer = Pipeline(steps=[('tfidf', TfidfVectorizer()),
                                   ('svd', TruncatedSVD(n_components=2))])

prep = ColumnTransformer(transformers=[('num', numeric_transformer, numeric),
                                       ('cat', cat_transformer, cat),
                                       ('text', text_transformer, 'col4')], sparse_threshold=0)

X_transformed = prep.fit_transform(X)

nn = NearestNeighbors(n_neighbors=2)
nn.fit(X_transformed)

d, i = nn.kneighbors(X_transformed[1].reshape(1,-1))
i

Correctly returns array([[1, 3]], dtype=int64) with the 1 indicating the self match and the 3 being the nearest neighbor.

But on a real world dataset, across 100+ dimensions, should I be using a custom distance function? for nearest neighbors with sklearn, if the column was one of the ones that was text, we could use cosine distance, another distance across the high dimensional one hot encoded variables, and another distance calculation with real continuous transformed variables (col1).

But is this....hacky? Is there some way to deal with the heterogenous data for nearest neighbors search without this? I worry that my decisions as to how to weight each of the three types of variables make the end results very subjective and open to criticism.


texts: differences between LSQR and FTRL when working with very sparse data

texts:
I have a 2M instances dataset with millions of very very sparse dummy variables created using the hashing trick = hash(orig_feature_name + orig_feature_value)=1. Note that the data is sparse both on rows (every instance has only a limited <100 features=1) and on columns (most features are relevant only to very few instances < 1%)

I discovered that in such sparse scenarios the follow-the-regularized-leader FTRL proximal gradient descent is very popular:
paper, reference implementation.

But I'm not sure why shouldn't I prefer a batch gradient descent algorithm? FTRL for all its merits is still an online-learning algorithm that sees one instance at a time.

So what are the advantages and disadvantages of using FTRL vs. a well known sparse least squares algorithm such as LSQR (paper, reference implementation)?

My intuition is that if possible to use all the data for each iteration, we should do it, but I'm not sure...


texts: How to scale an array of signed integers to range from 0 to 1?

texts: I'm using Brain to train a neural network on a feature set that includes both positive and negative values. But Brain requires input values between 0 and 1. What's the best way to normalize my data?


texts: Where in practice can we see following information propagation model

texts:
I am not certain, if this is the right place to ask the following question. I am looking for some practical scenarios in social networks where the following information propagation model can arise:

Basically, I have a source node and some information propagating radially from it and each recipient receives the information from a single sender.


texts: Problem regarding calculating correlation approach?

texts: There are 3 columns say product_id, product_type, and price_drop. product_id is unique while a product can belong to say 100 classes that information is given by product_type. price_drop column value is 1 when a drop in the price of the product occurred else 0. What I have done is, I one-hot encoded the product_type column created the dummy variable for them. Using each dummy variable I calculated their correlation with the price_drop column. I wanted to see the correlation between each product type and a drop in price. Is this approach is correct?


texts: Do CNN convolution and pooling layers get backpropogated?

texts:
I can't find a simple answer to this by Googling which leads me to think the answer is no, but I want to be sure...

In a feed forward network, all of the layers of weights get backpropogated, but what happens in a convolutional neural network on the backprop step? Is it only the feedforward part of the network (after the convolution and pooling layers) that gets backpropped? Which would mean that the convolutional layers are a form of static feature extraction...


texts: Calculate marginal probability distributions of a dataset

texts: I have a dataset with 'n' features and a label(binary) corresponding each entry. I am using a predicitive model to predict these labels using those 'n' features. Now, I wish to know the marginal probability distributions of my dataset. Is there any way to do so? I have read one calculates marginal probability distributions of x wrt y, how do people do it in such cases? Do they calculate label wrt one of the feature?


texts: How does class_weight work in Decision Tree?

texts: I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that there are a lot of algorithms Cost Sensitive Decision Tree. So what exactly does class_weight do in Decision Tree?


texts: Delete/Drop only the rows which has all values as NaN in pandas

texts:
I have a Dataframe, i need to drop the rows which has all the values as NaN.

ID      Age    Gender
601     21       M
501     NaN      F
NaN     NaN      NaN

The resulting data frame should look like.

Id     Age    Gender
601     21      M
501    NaN      F

I used df.drop(axis = 0), this will delete the rows if there is even one NaN value in row. Is there a way to do as required?


texts: Ways of filtering erronous email addresses using NLP?

texts:
Background:

I have a database of user information, in which they registered through a website.

Objective:

I would like to filter out erroneous emails, not by if it is malformed (i.e. it's missing an @-sign), but rather by "weird strings" in the "local-part" of the address. So examples of erroneous email address are things like:

zzzzzzzzzzzzzzzzz@gmail.com
asdfasdfasdf@gmail.com
yourenotgettingmyrealemail@gmail.com
123@yahoo.com
test@test.com

I know most of these require some "human" interpretation to figure that they're probably not real, but I was wondering if there are any algorithms that can help me out.


texts: Smoothing or averaging plots to better represent trends and their variations

texts:
My picture looks like this. It shows some percent variation but, due to fluctuations, curves are not very smooth at all.

I would like to draw a smoother image, without altering too much the data (it is okay if the graph is not a perfect representation of the data).
I am using matplotlib to draw the data, and my code looks like this:

x = df["Value"].shift(period)+1
y = 100*df["Value"].diff(period)/(period*x)

Basically, it draws the variation from the total over a period. What is the suggested way to alter the points (possibly drawing them alongside) the original data, so that the graphs do not look so jagged?

I tried just drawing every other third point, doing:

w = x[2::3]
z = y[2::3]

but I am sure there are better ways to do this. Again, this is for a nice representation only and I am mostly interested in showing a trend or any variation of it, not in showing the exact data points.