texts:
Imagine that I've this dataset (just a sample)

A   B   C
1   23  1000
2   52  5000
3   12  500
4   10  450

I'm trying to assign each row to a clustering based on C value. Like this:

A   B   C   CLUSTER
1   23  1000    2
2   52  5000    1
3   12  500     3
4   10  450     3

For that I'm using K-Means algorithm using Spark:

import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors

val data = sc.textFile("/user/cloudera/TESTE1")
val parsedData = data.map(s => Vectors.dense(s.split(',').map(_.toDouble))).cache()

val numClusters = 4
val numIterations = 20
val clusters = KMeans.train(parsedData, numClusters, numIterations)

val WSSSE = clusters.computeCost(parsedData)
println("Within Set Sum of Squared Errors = " + WSSSE)

clusters.save(sc, "/user/cloudera/KMeansModel")
val sameModel = KMeansModel.load(sc, "/user/cloudera/KMeansModel")

But this script extracts me a .gz.parquet file and when I try to see what type of information this file contains, using:

sqlContext.read.parquet("/user/cloudera/KMeansModel/data/part-r-00000-f551ea29-54db-45be-8cba-d06a97d6d9f2.gz.parquet").show

I'm getting this:

+---+--------------------+
| id|               point|
+---+--------------------+
|  0|[9.39601519885208...|
|  1|[9.80112351958380...|
|  2|[9.63822872186722...|
|  3|[9.44194658832542...|
+---+--------------------+

How can I get the table that I put above? Basically I just want to extract the same fields and add a column with the cluster calculated by K-Mwans to each row...

Many thanks!


texts: How to add incorporate meta data into text classification?

texts:
I have a collection of statements which I need to classify into 5 classes. Each statement have meta data in different columns:

Author|Editor| date of release| statement | Class

How can one use the meta data to improve the text classification task?


texts: I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.


texts: Backpropagation Mathematics with Sigmoid Output Activation and Cross Entropy Loss

texts:
I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly.
I am using Cross-Entropy Loss as my Loss function:
Where:
Now, I have a 1 hidden layer network architecture so I am trying to update my 2nd weight matrix:
Chain Rule derivation to Update 2nd Weight Matrix:
Where  This is the output of my hidden layer before I apply the sigmoid activation. A1 is the hidden layer activation matrix.
This is what I get so far for each part of the chain rule I derived above:

Finally with everything put together my derived partial derivative is:

This means sigmoid - y is a scalar value (the error from my training example) and it is multiplied by the prior layer's activation matrix (A1). Does this derivation seem correct? Sorry for formatting. I don't know LaTex yet.


texts: Multi-label text classification. 3-4 labels per text, 100 labels overall

texts:
My first pet ML project, so please pardon if I phrase something incorrectly. Recently I had IMDB sentiment analysis binary classification practice on Tensorflow site. Now I am keen to do multiple label classification of the abstracts in the newspapers.
I have prepared a Pandas dataframe with 2 key columns [sectors_array] and [text]. It is 10 0000 rows. There are 100 sectors overall. [sectors_array] column has 1 to 4 sectors (Same article can fall into multiple categories). Text is a string up to 500 chars. Text contains a piece of article and sectors categorize the text, i.e whether it is food, sport, cinema, politics etc.
So far I cleansed text column to remove single characters, urls, punctuation and did removal of stop words and Lemmatisation for nltk tokenized text.
For multiple sectors I added a hundred of columns with 1/0 flags for each sector.
Now what could be my direction from there in order to classify the new text to existing sector(s)?
Would this big number of sectors present an issue for categorisation?
Which library should be best for the task? Tensorflow/Spacy?
Ideally I would want to present up to most probable 4 sectors for the piece of text.


texts: Using random forest when not all the observations have data

texts:
I have this given database:

I would like to predict column "y" using the columns "index_1","index_2","index_3" using the random forest classifier.

as you can see, column "size: does not have values for each observation.

My question is : Can I still use random forest classifier when I don't have data for all the observations, and if yes, is it ok?should I give value (e.g "noData") to the empty cells? will it harm the prediction? or maybe no need?


texts: I try to understand my time series data for my ANN for a regression problem. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. This I repeat with certain time intervals. Do I have than 10.000 different features or only one feature as input dimension for my lstm neural network ? I want to shape my data for a lstm encoder decoder.


texts: CNN-Regression with a variable number of outputs

texts:
I want to predict several variables describing an object on an image. I can use CNN Regression to do that. But how can I do that when the number of objects on the image differs from one image to another ?
Here is an example : I have an image representing several arrows. For each arrow, I want to predict the position of the arrow tip (x1, y1) and the position of the arrow base (x2, y2). So I want to predict 4 values per arrow. For example, on the image below, there are 5 arrows so I want to predict 5x4 = 20 values. I want my algorithm to work on images having a variable number of arrows. So the number of output positions should be different for each image.

The only method that I know to predict a variable number of outputs from an image is object detection, with algorithms such as YOLO. These algorithms generally output a bounding box for each object on the image. So in my example, I could use YOLO to isolate a sub-image of each arrow, and then use a CNN-Regression model to predict the tip and base positions on each sub-image. But it takes 2 steps (so, 2 different models), and I don't think it would work well when the arrows overlap, as in my example : if the first model (object detection) outputs a sub-image with several arrow tips, the second model (CNN regression) can't know which tip/base it should look at to make a prediction.
Do you know other methods for this type of problem ?


texts:
There are a lot of ml-algorithms which cannot directly deal with categorical variables.
A very common solution is to apply binary (dummy-) coding to still properly handle the categorical nature of the data.

Very often e.g. in sk-learn or apache-spark the actual dummy-coder can only handle numeric values. So label-encoding needs to be performed beforehand.

In a real live ml-scenario, the fitted model will encounter new and formerly not known data. Usually, such a label-encoder (string-indexer) for spark has the option to either skip (ignore) a row of data which contains any unknown value or to throw an error. If multiple values require coding this can lead to a big loss of "new" data.

Are there any approaches which "tolerate" up to x new values per row and still properly evaluate the fitted pipeline?

An example for spark string-indexing + dummy-coding is shown below.

val df = spark.createDataFrame(Seq(
  (0, "a"),
  (1, "b"),
  (2, "c"),
  (3, "a"),
  (4, "a"),
  (5, "c")
)).toDF("id", "category")

val indexer = new StringIndexer()
  .setInputCol("category")
  .setOutputCol("categoryIndex")
  .fit(df)
val indexed = indexer.transform(df)

val encoder = new OneHotEncoder()
  .setInputCol("categoryIndex")
  .setOutputCol("categoryVec")
val encoded = encoder.transform(indexed)
encoded.select("id", "categoryVec").show()

  http://spark.apache.org/docs/latest/ml-features.html#onehotencoder


texts: What is the advantage of keeping batch size a power of 2?

texts:
While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.

This answer claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?


texts: What's the best way to validate a rare event detection model during training?

texts:
When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early stopping etc., or a validation set representative of reality? If an unbalanced, realistic validation set is used it may have to be huge to contain only a few positive event examples, so I'm wondering how this is typically dealt with.

In the given example of alarm sound detection, false negatives are obviously costly, but I imagine false positives still have an equal cost, because the event is so rare in reality that even a very low false positive rate could still correspond to low precision. Also, to me anomaly detection doesn't seem very applicable in this example because of the open set nature of the problem, where the "normal state" of the audio stream isn't clearly defined (i.e. there could be many unforeseen noises/sounds aside from alarms).

If anyone has insight in this area I'd greatly appreciate it!


texts: What's the input dimension for transformer decoder during TRAINING?

texts:
For example, translate English sentence A to French sentence B.
During training with ith word in B, all previous words before B will be fed to decoder, whose length will change for different i. How this is handled so that it can fit into a fixed dimension in the final linear layer during TRAINING?


texts:
what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5)

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

Where there are $N$ actions to choose from;

$s$ is the incoming state (the input vector)
$a$ is the action taken? (the chosen action)
$a'$ I don't know what it represents in this context
$\theta$ represents the weights of the convolutional layers
$\alpha$ are the weights of the "Advantage stream" which outputs a vector
$\beta$ are the weights of the Value stream (which outputs a scalar)

Why not to simply use $a$ everywhere, why is $a'$ used in the average?


texts: Sales Forecasting - Random Forest - Which features should I use for out of sample forecasting?

texts:
Sorry for the bad title, I can't find a good one. So I will try to explain what I'm looking for.

I'm doing sales forecasting with a Regression Forest. (Spark - Scala for the technology)
I've worked on some test data and I did my forecast using training data.
But some of the features which I have used can't be employed to forecast the future as they would not be known to me at any given time. For example the numbers of customers of a day, their categories, what kind of advantage they have etc.

Do I have to find others features that will be as useful as these ones or Do I need to perform prediction on these features before my sales forecasting and use the predictions?
Are there any another solutions? Also, what kind of algorithms should I use for the "features forecasting"?


texts: Package/function to convert the word to one hot vector in Python NLP

texts:
Is there a package or function in NLP which can be used to convert the word into an one hot vector.

Thank you.


texts: Which deep learning framework have support for gtx580 GPU?

texts:
I would like to train convolutional neural networks using a gtx580 gpu. I tried setting up TensorFlow but it did not work (wrong cuda compute compability).

Which deep learning framework can best support a gtx580 gpu in terms of functionality and speed?


texts: Binary classification for variable outputs

texts:
I am creating a binary classifier in Keras and here's the code

model = Sequential()
model.add(Dense(30, input_dim=60, activation='relu'))
model.add(Dense(1, activation='tanh'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])

The problem though, is that I have variable targets, i.e. they lie between -1 and 1. And here is a histogram of the distibution of my targets

I know that this still, could work very well for me, and the model would probably be able to predict the variable outputs, the input data is very complex, and the model might not be able to train to predict the exact outputs. I am perfectly okay of the model cannot predict the exact values, as a long as the sign of the output [-1,1] is correct.

So I thought what if the loss function gave zero loss for a prediction that has the correct sign, but gave whatever the required binary loss is, if the output has the wrong sign, this way, during training it would give importance to the highly incorrect predictions, but as soon as the sign of the predictions are right (just what I need), it doesn't consider them anymore, hopefully giving a better result. The catch here is, that if I just trained it to predict the signs of the targets, it would give all of the inputs equal importance

So my question is, does any such loss function exist in Keras or anywhere else, or do I have to create my own loss function? If I have to create my own loss function, how do I do it?


texts: Estimate the normal distribution of the mean of a normal distribution given a set of samples?

texts: Let's say there is a distribution, call it D, for which I don't know details (i.e. mean and variance) but can assume that it's a normal distribution. I now have N samples from D. I cannot take more samples because it's too heavy. Given the N samples, and the assumption that D is a normal distribution, how can I compute the mean and variance of the mean of D? Naturally it's less likely for the mean of D to be far away from the samples (thought still possible), while more likely to be near the samples.


texts: I have understood that data science is part of  AI. However, do we have some area in data science which we can say does not fall under AI? If yes, please name them.


texts: Bayesian network calculation to find probability for independent events?

texts:
I have the following Bayesian network:

I'm having trouble understanding how to calculate some of the conditional probabilities between nodes, in particular when they are independent. For instance, how would you calculate P(C=true, G=true|H=false)? I'm aware that I have to use the Bayes rule and the conditional probability formula. How do you go about setting up the equations for each of these and do you use variable elimination for any of these?


texts: Terrible Tensorflow performance on powerful production server

texts:
Background

I am building a simple face detection API with Python, Flask and the MTCNN face detector.

My problem is that the model and API are running really quickly (batch of 100 images takes 0.5-0.7 seconds to process) on my workstation (6 core Intel i5 9400F without GPU - macOS 10.15.1), but it has terrible performance (same batch takes 2-3 seconds) on a production server with dual 8 core Intel Xeon E5-2630 v3 (Ubuntu 18.04).

Question

How is this even possible? Is tensorflow somehow accelerated on macOS (Metal API etc)? Should I use a different server?

What have I tried:

installed Intel optimized tensorflow
recompiled tensorflow from source to include support for all CPU instructions
different servers from different vendors

Update

After more tests, it seems that even my MacBook Pro (2.8 GHz Quad-Core Intel Core i7) is outperforming Tensorflow on a GTX 2080 Ti and Tesla T4 (on Ubuntu 18.04), I really don't understand what is going on here.


texts: How to generate a sentence with exactly N words?

texts:
Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how to generate a sentence (with proper ending) of length N?

One possible approach is post-processing, that is process many sequences and choose the ones the serve the purpose! However, it could be a really daunting task to use in any pipeline.

Is there any suggestion, perhaps a secondary algorithm, to tune the hyper-parameter such that it produces sentence of desired length with higher probabilities.


texts: I want to know if there is any way that I can crawl customer reviews for particular products from amazon without being blocked. At the moment, my crawler is blocked after a few times. Any idea will be appreciated.


texts: How to update edge features in a graph using a loss function?

texts:
Given a directed, edge attributed graph G, where the edge attribute is a probability value, and a particular node N (with binary features f1 and f2) in G, the algorithm that I want to implement is as follows:

List all the outgoing edges from N, let this list be called edgelist_N.
For all the edges in edgelist_N, randomly assign to the edge attribute a probability value such that the sum of all the probabilities assigned to the edges in the edgelist_N equals to 1.
Take the top x edges (x can be a hyperparameter).
List the nodes in which the edges from step 3 are incoming.
Construct a subgraph with node N, the nodes from step 4 and the edges from step 3.
Embed the subgraph (preferably using a GNN) and obtain it's embedding and use it with a classifier to predict say f1/f2.
Propagate the loss so as to update the edge probabilities, that was assigned randomly in step 2.

I do not understand how to do step 7, i.e. update the edge attribute with the loss, so that edges which are more relevant in constructing the subgraph can be assigned a higher probability value.

Any suggestion would be highly appreciated.
Thank you very much.


texts: Neural network with fixed weights in Python lasagne

texts:
I am trying to train a neural network with the lasagne module in Python. I do not want a fully connected network as defined by lasagne.layers.DenseLayer. Instead, I would like to fix some of the weight parameters to zero. Does anyone know how to do this?

The closest solution I have found is something like:

params = lasagne.layers.get_all_params(network, trainable=True)
layer1.params[layer1.W].remove("trainable")

However, this fixes the entire set of weight parameters to their initial values. How can I fix only a subset of these weights?


texts: How to use k-means outputs (extracted features) as SVM inputs?

texts:
l have a dataset of images with their labels.
l put them into a k-means algorithm (as a feature extractor).
Now, l would like to use this new representation of images (features extracted from k-means algorithm) as SVM classifier inputs.
How can l do that ?
Number of cluster k=400 and numbers of images=1000.

However, l just have the vectors of centroids (400 centroids)

l need to get the representation for each image with respect to the centroids.

EDIT1

package update

from sklearn import mixture
gmm = mixture.GMM(n_components=6).fit(X)

Now l would like run k-means with different k=range(50,500), how can l get the distances for each k ?
Is is correct to do the following :

K=range(50,500)
KM=[KMeans(n_clusters=k).fit(X) for k in K]
distances = [np.column_stack([np.sum((X - center)**2, axis=1)**0.5 for center in C.cluster_centers_]) for C in KM]


texts: How to encode data with a feature having multidimensional features (colors)?

texts: My dataset has around 20 features, one of which is colors(in string format). There are around 50 different colors. I have converted them to RGB, but now I want to encode the data in such a way that the values are relevant, because I will cluster the data on this feature later on. To do this, one-hot encoding proved ineffective. Suggestions?


texts: VC-Dimension of Axis Aligned Right-Angle Triangles and 5-points Convex Hull

texts:
I am having trouble proving the following fact about VC dimension of triangles.
Consider right-angle triangles in the plane, with the the right-angle in the lower left corner.
The hypothesis in our case assigns label $1$ for points inside the triangle and label $0$ otherwise.
Claim is that the VC-dimension of the mentioned above hypothesis class is $VCdim(\mathcal{H})=4$.
In order to prove it need to show the following:

There is a set of cardinality 4 shattered by $\mathcal{H}$, and
Any set of cardinality greater than 4 is not shattered by $\mathcal{H}$. In other words, a set of cardinality $5$ cannot be shattered.

The  first condition is easy. It suffices to give an example of 4 points in the plane, with any labeling, and show a triangle that covers the labeled points.
I stucked proving the second condition. I split the proof into three cases, which differs by the amount of points that lay inside the convex hull created by those 5 points. The problem is with the case where none of those points lay inside the convex hull (i.e. they create a convex set).
I want to prove that for any combination of those 5 points that lay on a convex hull,
there is a labeling that cannot be shattered by a right-angled axis-aligned triangle.
Any help will be appreciated.
Thank you.


texts: Pandas DataFrame - compare the first element of each row with other elements of the row

texts:
How to correctly compare the first element of each row with other elements of the row in a Pandas DataDrame?

Below the code which gives the correct result but I think it is not an optimal solution because of the required two transformations.

Source code:

    df = pd.DataFrame([[1,2,3,1], [2,3,2,2], [3,3,5,3]])
    mask = (df.T == df.iloc[:,0]).T

    print(df)
    print(mask)

Result:

   0  1  2  3
0  1  2  3  1
1  2  3  2  2
2  3  3  5  3
      0      1      2     3
0  True  False  False  True
1  True  False   True  True
2  True   True  False  True


texts: How to adjust the hyperparameters of MLP classifier to get more perfect performance

texts:
I am just getting touch with Multi-layer Perceptron. And, I got this accuracy when classifying the DEAP data with MLP. However, I have no idea how to adjust the hyperparameters for improving the result.

Here is the detail of my code and result:

.

from sklearn.neural_network import MLPClassifier

import numpy as np
import scipy.io
x_vals = data['all_data'][:,0:320]

y_vals_new = np.array([0 if each=='Neg'  else 1 if each =='Neu' else 2 for each in data['all_data'][:,320]])
y_vals_Arousal = np.array([3 if each=='Pas'  else 4 if each =='Neu' else 5 for each in data['all_data'][:,321]])

DEAP_x_train = x_vals[:-256]        #using 80% of whole data for training
DEAP_x_test = x_vals[-256:]         #using 20% of whole data for testing
DEAP_y_train = y_vals_new[:-256]     ##Valence
DEAP_y_test = y_vals_new[-256:]
DEAP_y_train_A = y_vals_Arousal[:-256]   ### Arousal
DEAP_y_test_A = y_vals_Arousal[-256:]

mlp = MLPClassifier(solver='adam', activation='relu',alpha=1e-4,hidden_layer_sizes=(50,50,50), random_state=1,max_iter=11,verbose=10,learning_rate_init=.1)

mlp.fit(DEAP_x_train, DEAP_y_train)

print (mlp.score(DEAP_x_test,DEAP_y_test))
print (mlp.n_layers_)
print (mlp.n_iter_)
print (mlp.loss_)


texts: Best practice to select precision vs. recall threshold for a binary classifier

texts:
I have a logistic regression model in Scikit-Learn doing a binary classification.
Looking at the Roc curve for the classifier I can see that it performs really well:

The AUC score is 0.99 which is great and looking at a probably distribution for the classifier, it has no problems separating the classes:

However, when I look at precision versus recall for different thresholds I see the following:

I know that it is possible to write a function in Scikit-Learn to use a custom threshold, depending upon business needs. I am wondering if this is something I need to do here?
I assume the default threshold is at 0 (I acknowledge that may be incorrect, but I am not sure) and looking at the Precision/Recall curve, the 'sweet spot' threshold to get the most out of Precision and Recall appears to be about 1.7. Can someone please let me know if that is correct or my process has errors in it, as I don't fully have a handle on best practice for this (yet). Thanks!


texts: How do I get Kera's to use more than 1 core?

texts:
I've read that keras supports multiple cores automatically with 2.2.4+ but my job only runs as a single thread.  I'm running inside a VM else I'd try to use the GPU I have which means the solution I'm working with is CPU based.
Here's my snippet of code
import numpy as np
import tensorflow as tf
from tensorflow import keras
epochs_ = 1000
batch_size_ = 150

np.random.seed(42)
tf.random.set_seed(42)

from keras.layers import Dense, SimpleRNN, GRU, LSTM
from keras.optimizers import SGD

#simple RNN
data_ = Lagged_Set

model6 = keras.models.Sequential([
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(n_ahead))
])

model6.compile(loss="MAPE", optimizer="rmsprop",metrics=['MAPE'])
history = model6.fit(X_train, Y_train, epochs=epochs_,batch_size=batch_size_,validation_data=(X_valid, Y_valid))

I tried this which did nothing
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=8, inter_op_parallelism_threads=8)

#tf.compat.v1.ConfigProto.set_random_seed(1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)


texts: Difference between cell state and hidden state

texts:
LSTM cells consist of two types of states, the cell state and hidden state.
How do cell and hidden states differ, in terms of their functionality? What information do they carry?


texts: Current state of art approaches in generating features based on other available features and items

texts:
I am about to do a project that is about generating texts for missing entries that are related to some object (using information about other available entries related to the same object and corresponding entries in other similar objects) and my question is where I can find good litterature about state of art approaches that manage problems like these (or at least other problems that are related to this one my project will concern which I could consider while doing this project).

Thank you in advance!


texts: In Time Series forecast, should Scaling be done on both train and test features combined ( test is 1 new data point)?

texts:
Let say I have a Time series, I'm using sliding/expanding window method to split to train and test data: train would be all the data I have until day x and test is day x+1.
To avoid Data leakage I'm re-fitting the scaler for each day with the data I have until that day.
My question is: Why not fit the scaler with the data using all the training points and the Features from the next day ( Test Data)?
It wont be data leakage as I already have the test features available to me at the day of the forecast..
Thanks


texts: Training with different datasets for the same better VAE model yields poor results

texts:
The VAE model I used here https://github.com/keras-team/keras-io/blob/master/examples/generative/vae.py.
It can produce very well results for the minist and fashion minist dataset.
But when I use my datasets, the results are pretty terrible. So I sincerely hope you can help me to provide me some guidance.
The problems are as followings:

My datasets are composed of many circles in the image, as shown:

The shape of the datasets is [238, 28, 28, 1], and the range of pixel value is between 0 and 1 (floats32).

But the result is terrible.
The loss value

The genetive images:


texts: Why is convnet transfer learning taking so long?

texts:
I am using transfer learning to train a binary image classification model using keras' pretrained VGG16 model. The code can be found below :

training_dir = '/Users/rishabh/Desktop/CyberBoxer/data/train'
validation_dir = '/Users/rishabh/Desktop/CyberBoxer/data/validation'
image_files = glob(training_dir + '/*/*.jpg')
valid_image_files = glob(validation_dir + '/*/*.jpg')
# importing the libraries
from keras.models import Model
from keras.layers import Flatten, Dense
from keras.applications import VGG16
#from keras.preprocessing import image

IMAGE_SIZE = [64, 64]  # we will keep the image size as (64,64). You
can increase the size for better results.

# loading the weights of VGG16 without the top layer. These weights are
trained on Imagenet dataset.
vgg = VGG16(input_shape = IMAGE_SIZE + [3], weights = 'imagenet',
include_top = False)  # input_shape = (64,64,3) as required by VGG

# this will exclude the initial layers from training phase as there are
already been trained.
for layer in vgg.layers:
    layer.trainable = False

x = Flatten()(vgg.output)
#x = Dense(128, activation = 'relu')(x)   # we can add a new fully
connected layer but it will increase the execution time.
x = Dense(num_classes, activation = 'softmax')(x)  # adding the output
layer with softmax function as this is a multi label classification
problem.

model = Model(inputs = vgg.input, outputs = x)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
from keras.preprocessing.image import ImageDataGenerator
from keras.applications.vgg16 import preprocess_input

training_datagen = ImageDataGenerator(
                                rescale=1./255,   # all pixel values will be between 0 an 1
                                shear_range=0.2,
                                zoom_range=0.2,
                                horizontal_flip=True,
                                preprocessing_function=preprocess_input)

validation_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=preprocess_input)

training_generator = training_datagen.flow_from_directory(training_dir,
target_size = IMAGE_SIZE, batch_size = 200, class_mode = 'categorical')
validation_generator =
validation_datagen.flow_from_directory(validation_dir, target_size =
IMAGE_SIZE, batch_size = 200, class_mode = 'categorical')

training_images = 3717
validation_images = 885

history = model.fit_generator(training_generator,
               steps_per_epoch = 3717,  # this should be equal to total number of images in training set. But to speed up the execution, I am only using 10000 images. Change this for better results.
               epochs = 1,  # change this for better results
               validation_data = validation_generator,
               validation_steps = 885)  # this should be equal to total number of images in validation set.

I am training it on just 3700 images but still a single epoch is taking around 10-12 hours. Is this supposed to happen ? Am I doing anything wrong ? I had to downgrade my keras to 2.1.4 for the code to run so is it something affecteing the learning ?


texts: Scikit-learn randomforestclassifier error on fitting

texts:
So I am trying to use the random forest classifier from scikit-learn and I use tfidfvectorizer to create a feature set, then use test_train_split to create X_train and Y_train. I pass this into the fit function on a random forest classifier set up like this:

RandomForestClassifier(random_state=42, n_estimators=10, max_depth=1, verbose=True, n_jobs=-1)

However I run into a pandas error after:

[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 12 concurrent workers.
[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished

and end up getting empty dataframe failed to iterate on column errors like so:

Empty DataFrame
Columns: [(column headers listed out properly here)]
Index: []
Failed to iterate on column: (a name)
Failed to iterate on column: (b name)
Failed to iterate on column: (c name)

I have a fairly imbalanced dataset with some columns have 50k samples and some having 100ish so is that possibly and issue when using randomforestclassifier? Is there a need for there to be x many trees less than the smallest column length so that each tree can get a subset of y size?

It should also be noted that it doesn't crash, it just finishes and prints metrics which are absolutely terrible with most columns being straight up 0 and the larger categories being maybe 40% at most.


texts: Demand Prediction (Retail Sales) Datasets for Benchmarking

texts:
I've been looking for 2 days and can't seem to find what would normally appear as trivial:

a Time series (preferabl daily or intra-day resolution)
Demand or Sale Labelling
Retail Dataset
Preferably with multiple features/columns (e.g., more than 4 or 5 which seems to be all I can find)

I've crawled Kaggle and they all seem to be overly simplistic with one, two or four columns (e.g., very few features) and either seasonal (e.g., weekly or monhtly) or just too granular (multiple rows per day).
Am I looking for something that doesn't exist? What are the Demand and Sale Forecasting Datsets used to Benchmark or compare Model performance?


texts: How to find correlation between time-series of different units?

texts:
I have 3 time-series data.

NDVI(normalized difference vegetation index) mean
Precipitation
Temperature
All of these have their own unit.
Now I want to find similarity/correlation between NDVI and precipitation,
NDVI and temperature. Basically, my aim is to find "NDVI is more correlated with precipitation or temperature".
Should I normalize precipitation and temperature to NDVI values?


texts:
We have news URLS, which we want to classify into crimes or non-crimes and further identify criminals by using NERs.
For creating a model that identifies criminals, we tried SPacy which gave all the names like lawyers name , president ,criminal etc..
can Anyone help on how to get only Criminal name, Not all these irrelevant names.
I am just a beginner trying things, any help is appreciated
Thanks in Advance


texts:
I want to begin exploring OpenCV in Python but I'm stuck at importing the package cv2. I have installed the package through pip3 install opencv-python and it got installed at this location - C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages.

When I'm trying to import cv2 using this:

import sys
sys.path.append('C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages')
import cv2

It gives the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages\cv2\__init__.py", line 7, in <modul
e>
    from . import cv2
ImportError: cannot import name 'cv2'

I have searched a lot but cannot find anything relevant. Please suggest what needs to be done.


texts: in TensorFlow 2.0, what is the different between LSTM and LSTMCell objects?

texts:
I am trying to implement an RNN in TensorFlow 2.0 (beta1). Looking at the layer functions (inherited from Keras) I found:

tf.keras.layers.LSTM

and

tf.keras.layers.LSTMCell

What is the different between the two? If you take a look at their arguments, they seem the same.


texts: label encoding or one-hot encoding or none when using decision tree?

texts:
I've been learning about decision tree from multiple resources but still not fully understanding data preprocessing step.
from https://www.youtube.com/watch?v=PHxYNGo8NcI&t=535s&ab_channel=codebasics it uses decision tree with label encoder and in another resource it says we don't need to convert categories to strings, I'm confused.
Given I have data that looks like
   gender       level        score
    male          1           34
    female        2           77
    female        1           44

If we are using label encoder we would only need to convert gender however if that maps male = 0, female = 1 wouldn't the machine treat female > male? and if it ignores ordinality it will ignore level1 < level2 and treat as if level 1 and level 2 are same level which is not true.
What is the right preprocessing step and why?


texts: In the Coursera machine learning course by Andrew Ng, fmincg is used for optimization frequently. I'm not getting the mechanism of it. Can anyone give overview of how it finds the minimum of any function?


texts: Does a Convolutional Layer in a Neural Network learn the correlation between its input signals via its kernel?

texts: I am interested in the theory behing what a convolutional neural network learns with its convolutional operations. I think it learns (useful) kernels which measure the correlation between its input signals. Moreover, on a technical perspective: the convolution operation is implemented as "cross-correlation". So, is my assumption right?


texts: Cross validation and evaluation: neural network loss function continuously decreases in cross-validation

texts:
I am evaluating a neural network model using cross validation in 2 different ways ( A & B ) that I thought were equivalent.

Evaluation type A :
For each cross validation loop, the model is instantiated and fitted.
Evaluation type B :
I instantiate the model once and then that instantiated model is fitted for each loop of the cross validation procedure.

I am using the metric mean absolute error (MAE).
Question: Why do I get a continuously decreasing MAE over cross-validation loops when using type B evaluation and not when using type A evaluation?
Code and details
First I generate synthetic data :
from sklearn.datasets import make_regression

X , y = make_regression( n_samples = 1000 , n_features = 10 , n_informative = 5 , n_targets = 1 , random_state = 2 )

I then define a function to get a model ( neural network ) :
from keras.models import Sequential
from keras.layers import Dense

def get_model( n_nodes_hidden_layer , n_inputs , n_outputs ) :
   model = Sequential()
   model.add( Dense( n_nodes_hidden_layer , input_dim = n_inputs , kernel_initializer = 'he_uniform' , activation = 'relu' ) )
   model.add( Dense( n_outputs ) )
   model.compile( loss = 'mae' , optimizer = 'adam' )
   return model

After that I define 2 evaluation functions using :
from sklearn.model_selection import RepeatedKFold
from sklearn.metrics import mean_absolute_error

Type A evaluation function :
def evaluate_model_A( X , y ) :
   results = list()
   cv = RepeatedKFold( n_splits = 10 , n_repeats = 1 , random_state = 999 )

   for train_ix, test_ix in cv.split( X ) :
     X_train, X_test = X[ train_ix ] , X[ test_ix ]
     y_train, y_test = y[ train_ix ] , y[ test_ix ]

     model = get_model( 20 , 10 , 1 )
     model.fit( X_train , y_train , epochs = 100 , verbose = 0 )

     y_test_pred = model.predict( X_test )
     mae = mean_absolute_error( y_test , y_test_pred )
     results.append( mae )

     print( f'mae : {mae}' )

   return results

Type B evaluation function :
 def evaluate_model_B( model , X , y ) :
   results = list()
   cv = RepeatedKFold( n_splits = 10 , n_repeats = 1 , random_state = 999 )

   for train_ix, test_ix in cv.split( X ) :
     X_train, X_test = X[ train_ix ] , X[ test_ix ]
     y_train, y_test = y[ train_ix ] , y[ test_ix ]

     model.fit( X_train , y_train , epochs = 100 , verbose = 0 )

     y_test_pred = model.predict( X_test )
     mae = mean_absolute_error( y_test , y_test_pred )
     results.append( mae )

     print( f'mae : {mae}' )

   return results

Before using type B evaluation function I need to instantiate the model because it is an argument of the function :
model = get_model( 20 , 10 , 1 )

What I do not understand is the fact that while using type B evaluation function the MAE is decreasing for each cross validation loop which is not the case with type A evaluation function.
Is this specific to neural networks?
Note : when I am using a RandomForestRegressor() the phenomenon does not show up.


texts: The model performance vary between different train-test split?

texts: I fit my dataset to the random forest classifier and found that the model performance would vary among different sets of train and test data split. As what I have observed, it would jump from 0.67 to 0.75 in AUC under ROC curve (fitted by the same model under same setting of parameters) and the underlying range may be wider than that. So what is the issue behind this phenomena and how to deal with this problem? As my understanding, cross validation is used for a specific split of train and test data.


texts: How to estimate the accuracy on a large dataset?

texts:
Given that I have a deep learning model(handover from former colleague).
For some reason, the train/dev set was missing.

In my situation, I want to classify my dataset into 100 categories.
The dataset is extremely imbalanced.
The dataset size is about tens of millions

First of all, I run the model and got the prediction on the whole dataset.

Then, I sample 100 records per category(according to the prediction) and got a 10,000 test set.

Next, I labeled the ground truth of each record for the test set and calculate the precision, recall, f1 for each category and got F1-micro and F1-macro.

How to estimate the accuracy or other metrics on the whole dataset? Is it correct that I use the weighted sum of each category's precision(the weight is the proportion of prediction on the whole) to estimate?


texts: In practice, is relation extraction or relationship extraction the correct term?

texts:
I ask from the practitioner point-of-view, and I hope the answer does not come down to nit-picking, but I would like to settle the matter once and for all in the work I do.

One of the components of an Information Extraction (IE) pipeline is a relation(ship) extractor which picks out associations between entity mentions.  Stanford CoreNLP talks about this extractor strictly using the term relation(s) extractor.  Not that it is authoritative, but Wikipedia talks about relationship extraction.  The former seems more prevalent based upon both code and papers I have come across.  So is there an authoritative answer as to which is more proper, or are there legitimate cases to use both when talking about what we are trying to do?


texts: Predict Ethereum price up or down by given Bitcoin price

texts: How would you proceed, if you have 2 parameters, for example, the price of the Bitcoin and the price of Ethereum for each day and you want to predict based on the Bitcoin price if Etherium will go “up” or “down”?  Some ideas or some tutorials in this area?


texts: Bad classification performance of logistic regression on imbalanced data in testing as compared to training

texts:
I am trying to fit a logistic regression model to an imbalanced dataset (0.5/99.5) with high dimensionality(about 15k). I used random forest to select top 200 important features. Observations are around 120K.
When I fit a logistic regression model on based dataset (using Smote for over sampling) , on training f1, recall and precision are good. But on testing, precision score and f1 are bad. I assume it makes sense because in training there were a lot more of the minority case while in reality/testing there is only very small percentage. So the algorithm is still looking for more minority cases, which caused the high false positive.
I was wondering what kind of methods could I try to improve the performance?
I am currently trying different sampling method for imbalanced dataset, also plan to try PCA.


texts:
I have been reading around about Random Forests but I cannot really find a definitive answer about the problem of overfitting. According to the original paper of Breiman, they should not overfit when increasing the number of trees in the forest, but it seems that there is not consensus about this. This is creating me quite some confusion about the issue.

Maybe someone more expert than me can give me a more concrete answer or point me in the right direction to better understand the problem.


texts:
I'm making a multiclassifier model with 5 classes.
(it is not important in my question whether it has 2 classes or 5 classes, though).

class distribution is very imbalanced.

So, I did resampling for oversampling to match each number of classes.

MY question is.
Is there any statistical error or problem
if I resample classes for oversampling(bootstrap resampling or SMOTE)?

can anyone explain for this approach?

Thanks in advance.


texts:
I want to determine the similarity between images based on different features. The images show the same type of object (e.g. cars). I want to order images based on their similarity (e.g. through a feature vector). There are ways to solve this, for example a convolutional neural network. However, the images may be taken from slightly different perspectives: Sometimes directly from the front, sometimes with a slight ankle. What I DONT want is images ordered by their perspective rather than by the actual object. My questions:

What can I do to avoid primary order by perspective
Do I have to expect the ankle of the picture to be a primary factor in sorting order or will it more likely be negligible?
Is there a way to extract/visualise the features the network uses (unsupervised) in order to manipulate the sorting order (e.g. by saying I want to increase the weight of THIS visual feature when ordering)?


texts: Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

texts:
I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps:

obtain prediction on test data using model 1
obtain prediction on test data using model 2
construct a confusion matrix between predictions from model 1 and predictions from model 2
use the Stuart Maxwell Test to test marginal homogeneity and in this way, compare whether the difference between the 2 classifiers are significant

Will this be a correct way to tackle this task?
I have chosen this approach since the dataset I am using is large(~1 mil records) and my target variable has 10 classes. The dataset has been split into train/test/validation. In his 1998 paper, Thomas Dietterich recommended the McNemar’s test in those cases where it is expensive or impractical to use cross-validation. Since Stuart Maxwell Test is an alternative to McNemar’s test when it comes to more than 2 outcomes, I have chosen it to test my models.
I would really appreciate any opinion/advice on this!
Thank you!


texts: Best way to determine the number of features for RFE (recursive feature elimination)

texts:
I am applying the feature selection method, RFE (recursive feature elimination), from scikit-learn to a dataset. I do not have any pre-determined number of features for RFE and would rather get the number from data itself.

So far, I applied range of number of features, 1 to 10, for training data. For evaluation, I use the F1 from prediction outcome using the features from RFE. For serialization, I plan to use the number of features that provided the best F1.

What other methods may be used to determine the number of features for RFE? Thanks!


texts: Batch Normalization vs Other Normalization Techniques

texts:
In the context of neural networks, I understand that batch normalization ensures that activation at each layer of the neural net does not 'blow-up' and cause a bias in the network. However, I don't understand why it would be used as opposed to other normalization techniques such as Cosine or Weight normalization, these achieve the same goal and don't seem to be any more computationally complex.

Could someone please explain to me the advantages and disadvantages of using batch normalization vs other normalization techniques, and which contexts batch norm would be most beneficial?


texts: How does Performance function classify predictions as positive or negative? Package:ROCR

texts:
I'm performing a logistic regression on my training data. I used the glm function to get the model m. Now using the below codes from this link, I calculated AUC

  $test\$score<-predict(m,type = 'response',test)$
  $pred <- prediction(test\$score,test\$good_bad)$
  $perf <- performance(pred,"tpr","fpr")$

where score is the dependent variable (0 or 1).
To score the tpr (True positive rate) and fpr (False positive rate), you have to classify the predicted probabilities into 1 or 0.
What is the cutoff used for that? how can we change it?

Could not find anything useful in this main documentation as well.


texts: Terminology - regression with one output and multiple output variables

texts:
I am trying to predict the response when the input is represented by Fourier transform. These form the features and are typically represented as a vector, $x_1,x_2,...,x_d$ where $d$ is the length of the fourier transform. Based on my understanding each such $d$ dimensional vector can be an input to a regression model. The output is $y_1$ which is a scalar real-valued number and there is another output $y_2$ denoting another scalar real-valued number. These are the dependent variables. I have $N$ number of $d$ dimensional examples of fourier transform each labelled by $y_1$ and $y_2$.

Question 1) When the task is of predicting only one output response using the input fourier transform then is the problem termed univariate regression? Is univariate associated by the input's dimension(which is d>1) or the output's dimension (which is 1)

Question 2) When the task is of jointly predicting the two response variables - $y_1$ and $y_2$ then is the problem termed multivariate regression?


texts:
In R I have data where head(data) gives

day   count   promotion
1        33        20.8
2        23        17.1
3        19         1.6
4        37        20.8

Now day is simply the day (and is in order). promotion is the promotion-value for the day. It is simply the number of times an advertisement has been on television. count is the number of new users we got that day.

I want to investigate the impact the promotion-value has on new users (count). Since we have a count process I thought it would be best to make a poisson regression model.

model=glm(formula= data$count ~ data$promotion, data=data)

When we type summary(model) we get

Coefficients:
           (Intercept)              good_users$promotion
              13.40216                   0.24342

Degrees of Freedom: 793 Total (i.e. Null);  792 Residual
Null Deviance:      9484
Residual Deviance: 9325     AIC: 12680

Here is a plot of the data.

But when I plot the fitted values for the model

points(model$promotion, model$fitted, col="blue")

we get this

Here is another plot that shows the same but where days with 0 promotion are removed.

How should I chose my regression model (should I use lm instead of glm) or is the another better approach to solve this? Because the data is not pretty but more random like this what should one do ?

Updated

Finding the sweet spot

I have done the following for finding a sweet spot.
I divide data into 10 groups. group1 is simply a subset where the promotion-value is within 1:10. group2 is data where the promotion-value is between 11:20, and so on for the other groups. So in R we have

group1 <- subset(data, data$promotion %in% 1:10)
group2 <- subset(data, data$promotion %in% 11:20)
group3 <- subset(data, data$promotion %in% 21:30)
...
group10 <- subset(data, data$promotion %in% 91:100)

Now I can use wilcox.test to test if there is a significantly difference between the groups by typing

wilcox.test(group2, group1, alternative="greater")

which gives a low p-value, ie group2 has significant higher new_good_users than group1. The same goes for

wilcox.test(group3, group2, alternative="greater")

but for     wilcox.test(group4, group3, alternative="greater")
I get a p-value at 0.20, ie there is no significant difference in new_good_users between group4 and group3. And the same goes for the rest of the group-pairs up to 10.

So this must mean that if we increase promotion in the first groups we have an increase in new_good_users but in the last groups we do not have that increase. This means that we have a sweet spot at group3 where the promotion-value is 21:30. Is this not correct ?


texts:
There is a subsample parameter for the XGBoost (xgb.train() function in R). What is the corresponding subsample parameter for lightgbm in R?

In Python, the parameter is bagging_fraction.


texts: Time series forecasting for vibration prediction on Industrial machine production?

texts:
I'm working on a machine learning project related to an industrial machine.
The goal of the project is to build a model that would be able to predict the vibration of the machine while it's in production on a coil. In particular: the machine starts and the model after some warm up time (collecting data) could be able to predict the vibrational energy in the future (the end of the production).
The dataset is composed by many sensors variables (speed, diameter, accelerometers etc...), some derived variables (thickness, lenght of the coil, etc..) and also it's present the values of the vibrational energy (the one that the model has to predict in the future). So this let me though about a supervised model.
Many working registrations are available (around 4000) of different length.
The first work that I've done (after data understanding) is to clean the data by replacing missing values on every file, discard not consistent registrations (with some physical criteria given by the technicians) and finally i came to a reduced dataset. I've put together the registrations, normalized it and give it to the PCA in order to reduce the dimensionality of the dataset.
Maybe now i should eliminate some redundant registration by looking at the correlation between them, and repeat the PCA. Is there an algorithm to do it?
Now I'm stuck to understand which model could fit well with my problem.
I thought that Time Series Forecasting could be a good choice, but i don't know if the dataset is consistent with the forecasting. Given that I have many registration (15-20 minutes each) in a 2 years span, but not continuous given that the production is often interrupted.
Is the time series forecasting a good choice?


texts:
I'm trying to improve accuracy. I created a few new features based on old features. So I need to fill new feature's empty cell with same values in order to equaling shapes.Then, I tried it with median and also with mean but nothing changed. Is there any method that I can apply these cells in order to improve accuracy?

for example, there is a feature named age
I created new 2 feature named age_1 and age_2 ,
age1 consisting of age < 55 ,
age2 consisting of age > 55

therefore there is some empty cells in the new features and I have to fill these


texts: Noisification of categorical data proportions for privacy-preservation

texts:
Imagine I'm conducting an ongoing poll asking people's favourite animal out of a list of animals, [cat, dog, penguin, chimpanzee, ...] etc.

I want to provide an interface that lets people query this poll data to see the relative popularity of each animal by different demographics. For example, querying the general population might reveal the plurality of respondents (36%) prefer penguins, but querying the 18-25 age-bracket might the plurality of respondents in that cohort (41%) prefer cats.

It's desirable to preserve the privacy of my respondents' animal preferences as much as possible. However, an attacker may be able to use prior knowledge of a given respondent to deduce their response by asking a specific enough series of queries.

I wish to limit an attacker's ability to do this by noisifying the data presented to those querying the data. As such, I want a procedure that pseudorandomly adds or removes a fraction of a percentage point from each category, but preserves their relative ordering. I also wish this procedure to be deterministic over the same set of data (though this can easily be achieved by using a fixed seed in the pseudorandom procedure).

Formally, I want

$$f : \mathbb R_{>0}^n \rightarrow \mathbb R_{>0}^n;\;\; f(\mathbf{x}) = \mathbf{y}; \;\; |\mathbf{x}| = |\mathbf{y}| = 1$$

where $\mathbf{x}$ is the vector of proportions of each category.

One naive way of going about this would be to simply add a pseudorandom Gaussian noise vector to the original vector and then renormalise. This poses at least two problems:

1) the "zero problem": if a cohort has zero people who like cats, how should the noisification procedure treat this? I'm inclined to say it should maintain the value at zero, but I can't think of a principled way of achieving this

2) the variance of the noise should ideally be the same for all elements in the vector, but any obvious procedure for forcing positivity would all typically result in smaller variance of noise for smaller values, so the noisification would end up making large values larger and small values smaller after renormalisation.

I feel like this should be a problem people have encountered before, but I can't find it in the literature.


texts:
I am new to keras and machine learning. How can i tell if i am on the right track of setting up keras layer for binary classification, LSTM or convulsion network?

For example (from keras documentation)

    # For a single-input model with 2 classes (binary classification):
    model = Sequential()
    model.add(Dense(32, activation='relu', input_dim=100))
    model.add(Dense(1, activation='sigmoid'))

    model.compile(optimizer='rmsprop', loss='binary_crossentropy',
    metrics=['accuracy'])

Multi layer perceptron multimax softmax classification

model = Sequential()

model.add(LSTM(32, return_sequences=True, stateful=True,
batch_input_shape=(batch_size, timesteps, data_dim)))

model.add(LSTM(32, return_sequences=True, stateful=True))
model.add(LSTM(32, stateful=True))
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

Sequence classification for 1d convulsion

model = Sequential()
model.add(Conv1D(64, 3, activation='relu', input_shape=(seq_length, 100)))
model.add(Conv1D(64, 3, activation='relu'))

model.add(MaxPooling1D(3))
model.add(Conv1D(128, 3, activation='relu'))
model.add(Conv1D(128, 3, activation='relu'))
model.add(GlobalAveragePooling1D())

model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))


texts: How to use "tree boosting" with a data-driven loss function

texts:
We have a problem which has a data-driven (non-analytical) loss function. Our target contains whole numbers between 0 and 20 (the target is inherently discrete), although larger values are possible, just not present in our dataset. The fact that we have a very precise loss function leaves us with some serious issues when using algorithms like XGBoost:
The loss function is generally non-convex. It's not easily fitted by a convex function since its shape is data-driven and can vary drastically. For example, this means that a large punishment is inevitably given for predictions further from the part of the function that is well-fitted, where no large punishment is required. If we interpolate instead of fit, the hessian can be negative (see attached picture), which is a problem for determining leaf weights (right?).

We think we can adapt something like the XGBoost algorithm (I use this algorithm as an example because I'm both familiar with the paper and the API) by swapping out its dependance on the gradient en hessian with a brute-force method for finding the optimal leaf weights and best gain. However, this will slow down the algorithm massively, perhaps cripplingly so.
My questions are: is the some default way of dealing with complex loss-functions within existing algorithms? Is the an algorithm that is suited for dealing with these problems? Is there anything else you could suggest to solve the above issues?
Thanks in advance.


texts: Random-Forest-based Similarity Matrix for clustering: how does it behave?

texts:
I am in the following context:

Data: static, baseline health data at the patient level, 40 features, sparse (~ 25 binary features with many 0 or many 1 + other categorical features)

Objective : to cluster instances into clinically meaningful sub-population or clinical context, to get a sense of at risk sub-populations (on a Follow-Up outcome)

Considered approach  (see this short blog article (2 minute's read) for its rationale):

Fit a random forest on the Follow-Up outcome, using all features (no denoising or removing correlation)
Use co-occurence in trees leaves to get a similarity matrix of the patients
Turn in into a distance matrix
Cluster patients with this distance matrix

My questions are as follow:

I have only found literature using this methodology for unsupervised clustering (i.e. RF is learned on random target variable) : Shi and Horvath 2016 and Dalleau 2018 (unpublished). Does anyone have insights on those methods in general (other references, personnal experience...)?
Do you know of any article written about the supervised use of RF to create similarity matrices (with or without mention of clustering)?


texts:
I have a XML file has this structure (not exactly a tree though)

<posthistory>
<row Id="1" PostHistoryTypeId="2" PostId="1"
RevisionGUID="689cb04a-8d2a-4fcb-b125-bce8b7012b88"
CreationDate="2015-01-27T20:09:32.720" UserId="4" Text="I just got a
pound of microroasted, local coffee and am curious what the optimal
way to store it is (what temperature, humidity, etc)" />

I am using apache pig to extract just the "Text" part using this code

grunt> A = load 'hdfs:///parsingdemo/PostHistory.xml' using
org.apache.pig.piggybank.storage.XMLLoader('posthistory') as(x:chararray);

grunt> result = foreach A generate XPath(x, 'posthistory/Text');

this returns "()" (null)

Upon examining the XML file, I learnt that my XML file should be in this format:

<root>
  <child>
    <subchild>.....</subchild>
  </child>
</root>

But my XML data file (stackoverflow data dump actually) is not in this format. Is there a way the tree structure can be imposed? what is wrong with my pig query?


texts: Back-propagation and stochastic gradient descent

texts:
Is backpropagation a learning method or an optimisation method?

How are backpropagation and stochastic gradient descent related to each other?


texts:
I have certain tree structures. I am not an expert in machine learning.

As I would with take KNN, I would calculate distances via metric function and a new data point and the points from the training set. Okay, understood.

Now let's consider the data points aren't tuples but trees. How would one calculate the distances between two graphs.

For instance, I want the following

I have several stereotypical trees like:

|
 - <form class="xyz">
   |
    - <input type="text">
    - <button value="Click here">

between all these node, there could be arbitrary other nodes, the value of the button-node could be different, the type of the input node could be different, even the button-node could be a different node type, like input, the form-node will be surrounded by arbitrary nodes.

How would one calculate such a distance between trees?


texts: Why TREC set two task: document ranking and passage ranking

texts:
TREC is https://microsoft.github.io/TREC-2019-Deep-Learning/

I am new to text retrieval. Still can not understand why set the two similar task.

Thank you very much.


texts: Do neural networks have explainability like decision trees do?

texts:
In Decision Trees, we can understand the output of the tree structure and we can also visualize how the Decision Tree makes decisions. So decision trees have explainability (their output can be explained easily.)

Do we have explainability in Neural Networks like with Decision Trees?


texts:
I am using matlab, and I need to convert 16-bit images to 8-bit images (tif extention) ,
and I used

image = im2uint8(16bit_img);
image = uint8(16bit_img/256);

and both of these methods are giving me matrix with same intensity numbers as shown in the pictures (the first one is the original 16-bit image, the second one is after conversion to 8-bit)

However, converting the image to 8-bit using other software such as ImageJ, it gives different (reasonable) values as shown below

Can anyone explain the reason why it has different output?


texts: What is the proper way to bin variables for calculating WoE during credit scoring?

texts:
I have read this article about developing a credit scorecard in python, where it is stated that when binning the continuous variables, it needs to be ensured that:
1. Each bin should have at least 5% of the observations
2. Each bin should be non-zero for both good and bad loans
3. The WOE should be distinct for each category. Similar groups should be aggregated or binned together. It is because the bins with similar WoE have almost the same proportion of good or bad loans, implying the same predictive power
4. The WOE should be monotonic, i.e., either growing or decreasing with the bins
5. Missing values are binned separately

This seems like a lot of work to accomplish manually (each column needs to be divided in bins, each of these five conditions need to be checked manually, bins should then be adjusted and the conditions need to be checked again). Is there a faster way to do it? Or is there any algorithm/function that bins continuous variables in most practical way.


texts:
As I understand, ExtraTreesRegressor from sklearn works by doing random splits instead of minimizing a metric like gini for classification or mae for regression.
I don't understand why there's a criterion parameter, as the criterion for the splits should be random.
Is it just for code compatibility, or am I missing something?


texts: How to create a custom loss function from sklearn metrics in Keras?

texts:
I'd like to use the mutual information metric from sklearn as a loss function for a neural network in Keras, but I'm not sure how to do it. I'd like to try this because relationships in my dataset are non-linear and I believe it would help to use this metric instead.

Posts about creating custom loss function are more than a year old so I was hoping there was maybe some easier way to do this as some of it goes above my head.

Any idea about how to go about this?

Thanks a lot


texts: Difference between various linear regression implementations

texts:
Aim: To find the coefficients for the regression line (hyperplane in case of multiple variables?) that models the data best. Let's call this w

What is the difference between:

1) Estimating using MAP: $w=(XX^T+\lambda I )^{-1}Xy^T$ where $X$ is the input training data and $y$ is the training outputs

and

2) Using a neural network to perform regression (I don't know how this is implemented)

(and any other method used for linear regression)


texts: Why my loss is negative while training SAE?

texts:
I am using loss='binary_crossentropy' here is my code:

I tried to increase number of training image and Epoch ,but that did not help me.

input_img = Input(shape=(28, 28, 1))

x = Convolution2D(16, 3, 3, activation='relu', border_mode='same')(input_img)
x = MaxPooling2D((2, 2), border_mode='same')(x)
x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)
x = MaxPooling2D((2, 2), border_mode='same')(x)
x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)

encoded = MaxPooling2D((2, 2), border_mode='same')(x)

x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)
x = UpSampling2D((2, 2))(x)
x = Convolution2D(16, 3, 3, activation='relu', border_mode='valid')(x)
x = UpSampling2D((2, 2))(x)

decoded = Convolution2D(1, 3, 3, activation='sigmoid', border_mode='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train, nb_epoch=10, batch_size=500,
               shuffle=True, validation_data=(x_test, x_test), verbose=1)


texts:
I'm currently working on a project where I'm supposed to compare the efficiency of SVM vs RVM, there seems to be a lot of information to be gathered about RVM whereas I find rather old documents about RVM. It's the first time I use this forum so I hope my question isn't off-topic but my questions are:

1) Which datasets are more suited for the two models and what are the upsides/downsides for each model?
2) As mentioned above, the articles I find about SVM are rather old, I'm currently curious about recent applications of SVM and to what and where they have been implemented.

Sources or just standard responses are highly appreciated. Thanks in advance


texts: Detecting directions using Convolutional neural networks

texts: I am working on a task where I have to detect damages on the vehicles and exactly where the damage has occurred. So I have to not only detect the damage on the door but also mentioned which door(front left,rear right,etc). Is it possible to not only detect the objects but the direction of the objects as well using Convolutional Neural Networks. Are there any implementation of tasks similar to this?


texts: how do i fix this "too many indices " issue?

texts:
#creation of data
x,y=make_classification(n_samples=10000,n_features=2,n_informative=2,n_redundant=0,n_clusters_per_class=1,random_state=60)
#splitting of data
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3,stratify=y,random_state=42)
train_mean_scores=[]
cv_mean_scores=[]
#function to get optimum k or hyper_parameter
def RandomCVSearch(x_train,y_train,classifier,params,folds):
    for k in params:
        train_fold_score=[]
        cv_fold_score=[]
        #code to assign groups
        numbers=[]
        for i in range(0,len(x_train)):
            numbers.append(i)
        indices=tuple(numbers)
        split_indices=np.array_split(indices,folds)
        indices_parts=list(map(tuple,split_indices))
        for cv_idx in indices_parts:
            cv_x=x_train[cv_idx]
            cv_y=y_train[cv_idx]
            train_idx= list(set(list(range(1, len(x_train)))) - set(cv_idx))
            x_tr=x_train[train_idx]
            y_tr=y_train[train_idx]
            classifier=KNeighborsClassifier()
            classifier.fit(x_tr,y_tr)

            #accuracy score of train data
            y_tr_pred=classifier.predict(x_tr)
            train_fold_score.append(accuracy_score(y_train,y_tr_pred))

            #accuracy score of cv data
            y_cv_pred=classifier.predict(cv_x)
            cv_fold_score.append(accuracy_score(cv_y,y_cv_pred))

        train_mean_scores.append(np.mean(np.array(train_fold_score)))
        cv_mean_scores.append(np.mean(np.array(cv_fold_score)))
    return train_mean_scores,cv_mean_scores

from sklearn.metrics import accuracy_score
from sklearn.neighbors import KNeighborsClassifier
import matplotlib.pyplot as plt
import random
import warnings
warnings.filterwarnings("ignore")

classifier = KNeighborsClassifier()

params =[i for i in range(1,40)]
folds = 3

train_scores,cv_scores = RandomCVSearch(x_train, y_train, classifier, params, folds)

plt.plot(params,train_scores, label='train cruve')
plt.plot(params,cv_scores, label='cv cruve')
plt.title('Hyper-parameter VS accuracy plot')
plt.legend()
plt.show()

OUTPUT:
IndexError                                Traceback (most recent call last)
<ipython-input-21-1fafbc17cd57> in <module>
     13 folds = 3
     14
---> 15 train_scores,cv_scores = RandomCVSearch(x_train, y_train, classifier, params, folds)
     16
     17

<ipython-input-20-17e5d5e8d7bb> in RandomCVSearch(x_train, y_train, classifier, params, folds)
     18         indices_parts=list(map(tuple,split_indices))
     19         for cv_idx in indices_parts:
---> 20             cv_x=x_train[cv_idx]
     21             cv_y=y_train[cv_idx]
     22             train_idx= list(set(list(range(1, len(x_train)))) - set(cv_idx))

IndexError: too many indices for array


texts:
Im a noob in ml / statistical algorithm, but I do have worked with simple classifiers and regression

I like some opinions if I am going the right way, given my limited knowledge

My problem is following: I wanna best predict price of a input item, given the known price of the most similar items in database.

This at first sounded like regression but my features are mixed discrete and contiguous, (like item model type (think car model, not ml model) is discrete, year of manufactury is contiguous?)

Now I am thinking finding x closest matches via euclidean distance, and then weighted average + somehow make sure the certain features have to match exactly (like item model type)

Thanks


texts:
I'm new to machine learning, being this the first time I'm involved in a project in the area. I have a dataset of news articles and have extracted the keywords present on the news title such as ['china factory activity shrinks', 'first time', '2 years']. Note, the list size varies.

I would like to use this data in my features, but I don't know how and what is the best way to extract features based on a list of keywords.

Since the keywords cannot be converted to a categorical value, I think I can't use OneHotEnconder, but I'm not 100% certain. Is bag-of-words or TF-IDF viable possiblity? How can I encode this keywods to numerical values maintaining some meaningful information to the model?


texts: I am working on a problem that current depends on word level embeddings created using Word2Vec. I am researching new methods to apply to this model and one was a character level embedding. I have not found much information on it, and I don't imagine Word2Vec but at a character level would be effective. Is there any insight on giving vector representations to characters for an overall classification model?


texts: Why Extra-trees should only be used within ensemble methods?

texts:
I was reading scikit-learn documentation for Extremely Randomized Trees and I found this warning:

  Warning: Extra-trees should only be used within ensemble methods.

Why is that?


texts: Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

texts:
I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67.
I had the same algorithms and same set of hyperparamters during yesterday and today's run.
Please note that I also have the parameter random_state in each estimator function as shown below

np.random.seed(42)
svm=SVC()  # i replace the estimator here for diff algos
svm_cv=GridSearchCV(svm,op_param_grid,cv=10,scoring='f1')
svm_cv.fit(X_train_std,y_train)

q1) Why does this change happens even when I have random_state configured?
q2) Is there anything else that I should do to reproduce the same results every time I run?
Please find below the results that are different? Here auc-Y denotes yesterday's run


texts: K-fold cross validation of scikit-learn with confusion matrix of Keras

texts:
I intend to display confusion matrix using Keras while K-fold of scikit-learn. My code using Keras is:

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense

seed = 7
numpy.random.seed(seed)

# load dataset
dataframe = pandas.read_csv("BolMov.csv", header=None)
dataset = dataframe.values
X = dataset[:,0:24].astype(float)
Y = dataset[:,24]

model = Sequential()
model.add(Dense(16, activation='relu'))
model.add(Dense(7, activation='softmax')
model.compile('adam', 'categorical_crossentropy', metrics=['accuracy'])

y_cat = to_categorical(Y)
result = model.fit(X, y_cat, verbose=0, epochs=50)

plot_loss_accuracy(result)

y_pred = model.predict_classes(X, verbose=0)

print(classification_report(y, y_pred))
plot_confusion_matrix(model, X, y)

How should I use kfold in this code? Here the author is calling a function. What I believe is that if I do so in my code, the model.fit() will be executed twice - once for my Keras code and another time (internally) for the KerasClassifier(). I want that the model.fit() executes once only. Help from anyone is appreciated.


texts: In python/pandas what would you suggest as a couple of good basic methodologies to use to forecast weekly average usage of user activity on a given platform?


texts: Why do we use +1 and -1 for marginal decision boundaries in SVM

texts:
While using support vector machines (SVM), we encounter 3 types of lines (for a 2D case). One is the decision boundary and the other 2 are margins:

Why do we use $+1$ and $-1$ as the values after the $=$ sign while writing the equations for the SVM margins? What's so special about $1$ in this case?

For example, if $x$ and $y$ are two features then the decision boundary is: $ax+by+c=0$.
Why are the two marginal boundaries represented as $ax+by+c=+1$ and $ax+by+c=-1$?


texts:
I read a recent paper by Pedro Domingos, claiming that Every Model Learned by Gradient Descent Is Approximately a Kernel Machine.
I wanted to understand the key idea a little better. Why are neural networks equivalent to kernel methods? And in particular, is there any method to actually derive the equivalent kernel from an existing neural network?


texts: Combine K-nearest neighbor with categorical embedding

texts:
I've tried a few ways to do my multi-class classification. For categorical data, I used the embedding technique with Tensorflow, which moves the entity closer with its similarity. This technique provides me with approximately 25%-30%, which is low.
For the numerical data, I used the KNN algorithm that gave me roughly 40% accuracy. I am wondering is there any way to "combine" these two techniques together to achieve a better result. For example, perhaps using the probability given by the KNN algorithm to form a layer concatenated with the embedding layer. Then, use the Dense layer to further train these data.
I've searched on the Internet. It's not the technique of ensembling, which averages the accuracy of each model. It's more like concatenating the layer together.
Any help is highly appreciated.


texts: Two different cost functions for neural networks, how they can give the same result?

texts:
One is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K}\Big[y_{k}^{i}\log\big((h_{\theta}(x^{i}))_k\big)+(1-y_{k}^{i})\log\big(1-(h_{\theta}(x^{i}))_k\big)\Big]$$

The other one is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\Big[y^{i}\log(a^{i})+(1-y^{i})\log(1-a^{i})\Big]$$

As I can see those two equations are not equal. How both can be used to calculate cost function?

Also, one of them using $h$ function which is $a$ of output layer, whereas others are using $a$ ($a$ is $f(w*x)$ where $f$ is activation function). When I looked from the book "Pattern Recognition and Machine Learning" from Bishop, he used $a$ for both of the equations. But from another source which I took equations from $h$ is used. But using different $a$ values and using just one of them (namely $h$ which is $a$ of output) are totally different things.

Both sources are reliable, what am I missing?


texts: How to create a model to suggest similar words in realtime?

texts:
I have a huge database of job titles, I want to build a system where if you enter something like "jav " then it should suggest next some similar job titles like (java developer, java engineer) etc..
How should one approach this problem? How can build something like this, the latency is biggest concern because it has to be real time. We have to integrate this in UI at the end.
Any suggestions how to proceed further?


texts: NLP : What are some common verbs surrounding organization names in text

texts:
I am trying to come up with some rules to detect named entities, specifically company or organization names in text. I think it makes sense to focus on verbs. There are a lot of POS Taggers that can easily detect proper nouns. I personally like StanfordPOSTagger. Now, once i have the proper noun, i know that it is a named entity. However, to be certain that it is the name of a company, i need to come up with rules and possibly Gazetteers

I was thinking of focusing on verbs. Is there a set of common verbs that occur frequently around company names?

I could create an annotated corpus and explicitly train a Machine Learning classifier to predict such verbs, but that is a LOT of work. It would be great if someone has already done some research on this.

Additionally, can some other POS tags give clues? Not just verbs.


texts:
I am new to DNN still learning, have a need to build item-to-item content based recommendation using DNN. For example, say I have a column of strings where each row represents a document I need to compute the cosine similarity of this column and recommend similar documents.
id   document

1     "hi this document is about science"

2     "hi this document is about wars"
3     "This document is about peace"

now need to train based on document and column and recommend all the documents similar to a given document. I have solved this problem by using cosine similarity in ML now wanted to achieve the same in DNN (Deep Neural Nets). I have spent a lot of time on the net they all talk about movie rating where userId, rating, title is used. The problem I only have one column document how can I achieve this in DNN?


texts:
Can someone explain the following python code.
value_geojson["features"][0]["properties"]["title"]

value_geojson is a geojson variable.
I like to think that I'm not a total python newbie but these are too many [] for me. I am a GeoJSON newbie though.
Would appreciate help on that one, eventhough I know it's probably to simple for SE


texts: Most Efficient Machine Learning Algorithm for Text Analysis

texts:
Looking at Understanding Convolutional Neural Networks for NLP, Convolutional Neural Networks (CNNs) seem to be suitable not only for image recognition, but also for NLP.

Are CNNs in general the best choice for NLP text analysis, e.g. for sentiment analysis?

If not, is there an overview or comparison of different algorithms with regards to recognition performance (F1 score or something similar)?


texts: Regression: What defines Linear and non-linear models or functions

texts:
Linear regression is used when there is a linear relationship between the input and output variables. Does this linear relationship mean that there is no power over the variables or the parameters? In m understanding, by linear we mean linear with respect to the parameters (no power). Please correct me if wrong. Basically, I am a bit confused looking at this tutorial: https://www.gaussianwaves.com/2013/03/how-to-estimate-unknown-parameters-using-ordinary-least-squares-ols/

where linear regression is used to fit the straight line but if the function is a cubic or higher order polynomial (power over the variables), can we still use linear regression? I have seen that for fitting higher order polynomials, we use Regularized Linear Regression in order to penalize the higher order termed variables. Based on these confusions, can somebody please help clarify the following points?

MAIN QUESTION FOR WHICH BOUNTY IS OPENED:

1) Linearity is with respect to the variables or the parameters? I saw a similar question asked here Is a "curve" considered "linear"? but the second answer says that linearity is in terms of the parameters such as weights or other hyperparameters and polynomial regression is a special type of linear regression. So, if the parameters of the model have a power then that function is a nonlinear function and hence we get a nonlinear model. Is that correct?

Other questions that have been answered and understood by me

2) Do higher order termed polynomials fall under linear or non-linear models?

3)The sigmoid function does not have any power and is commonly used in the hidden layers of multi layer perceptron thereby making MLP non-linear machine learning model? How come sigmoid curve is a non-linear function?

4) Is there a check whether to go for linear or non-linear functions ie., whether to fit linear regression or non-linear models such as SVM with kernel?


texts:
I have a problem of multi class classification and I'm using a simple 2-Layer Bi-directional LSTM with keras.

The model in a simple form:

Bidirectional LSTM (64)
Bidirectional LSTM (64)
Dense (128)
Activation Sigmoid
Dense (14)
Activation Softmax

I have a raw and skewed dataset so I'm doing all the pre-processing myself to balance it.

Firstly, I replicated the original data and produced 3*OriginalData
to produce more examples of certain classes that weren't represented.
Then I performed classification and the loss and accuracy history are
depicted below for both training and validation sets

Accuracy built up after 25 epochs, but loss had major variations, so I decided to replicate more of my data so I produced 10*OriginalData, and performed classification again.

Now my loss and accuracy behaved so much worse.

My question is whether the problem lies in my model, or at the replication of data. Is it possible for a model to work well with less data but not so good when it has so many? Or maybe I'm causing it to overfit by providing over-replicated data?

Note: By replicating data, I don't mean duplicating, but because I work with audio data, I play with different pitch shifts.


texts: Reinforcement Learning in 2018, best tips and tricks?

texts:
Putting aside things applicable to neural networks such as dropout, l2 regularization, new opitmizers - what are the cool things I should be adding to my Reinforcement Learning algorithm (Q-Learning and SARSA) to make it stronger, in 2018?

So far, I know these:

Experience Replay (gather experiences, then train at once on some of them)
Slowly making our network approach the Q-values of a Target Network. The target Network is only cloned sometimes.
If our function approximator is RNN and our Memory Bank stores sequences of several timesteps, after selecting few random sequences from Memory bank, only backpropagate a half of the sequence, without going back to its start


texts: Why does my cost either increase or go down very slowly

texts:
def hypothesis(W, B, X):
  Z = np.dot(W.T, X) + B
  return sigmoid(Z)

def sigmoid(Z):
  return(1/(1+np.exp(-Z)))

def cost(A, Y):

  return (-Y * np.log(A) - (1 - Y) * np.log(1 - A)).mean()

def derivatives(W, B, X, Y):

  A = hypothesis(W, B, X)

  return np.dot(X.T, (A - Y)) / Y.shape[0] , (A - Y) / Y.shape[0]

def updateParameters(W, B, X, Y, learning_rate):
  # print(W.shape, X.shape, Y.shape )

  dW, dB = derivatives(W, B, X, Y)
  W -= learning_rate * dW
  B -= learning_rate * dB

  return W, B

def LogisticRegression(X, Y):
  W = np.zeros((X.shape[0], 1))
  B = np.zeros((X.shape[0], 1))

  for i in range(0, 1000000):
    print(cost(hypothesis(W, B, X), Y))
    W, B = updateParameters(W, B, X, Y, 0.0000003)

LogisticRegression(X, Y)

I'm a beginner trying to implement logistic regression from scratch.

However, my cost always goes up, and when I reduced my learning rate in case that was too high, it started going down but only extremely slowly.

Could someone point out if there is any issue in the implementation of the algorithm? Any help would be greatly appreciated


texts: Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

texts:
1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event occurring? (assuming we have an accurate model)

2) The same question as in 1), but for other models that can output raw probabilities.


texts:
I am trying to predict an output value based on several continuously-valued inputs using a regression model.

I am not sure what approach is appropriate to scale/transform the input data for the regression. Let's just pretend that it is unlabeled data.

My most naive approach would be to add each input multiple times:

input
log(input)
sqrt(input)

and then let the regression model worry about finding which flavor of each input (if any) is significant.

What are the risks with using this approach?


texts: VC dimension of half spaces over the real line

texts:
I'm studying VC dimension and I'm having a little difficulty understanding it. I read lots of explanations, but when I come across this simple exercise I did not get a good intuition. The problem is this:

Find the VC dimension for the following case,

  $h (x) = \mathbb{1}_{{a < x}}$, with parameter $a, X \in R$.

The resolution is as follows:

  VC-dimension = 1.

  (a) It can shatter point $0$, by choosing a to be $2$ and $-2$.

  (b) It can not shatter any two points ${x1, x2}, x1 < x2$, because the
  labeling $x1 = 1$ and $x2=0$ can not be realized.

My doubts are:

1) In item (a) what is parameter $a$? I looked for similar exercises, but there was no concrete explanation about this parameter, what is its function? In this item it takes as equals $2$ and $-2$, with $-2 < 0$ ok, but $2 < 0$, I did not understand, from this conclusion, why it is correct.

2) In item (b) I did not understand the inequality, as he came to the conclusion that $x1 < x2$ can not be realized, why is the inequality $x1 > x2$ not valid?


texts: Does using user-specific accumulative variables causes data leakage?

texts:
Let's say I have a scenario in which my observational unit is a bill that was issued after a certain service was given and my goal is to predict if this bill is going to be paid or not. I have users in the system so I include user-variables like number of unpaid bills the user has, user's time in the service system (seniority) etc. I train on month 1 and test on month 2 (bills that were created in those months, respectively).

In the testing month I will have user-variable's count increased, so e.g. if during training user_1 had 100 days time in my system, of course that if there is a bill associated with him in the test month - his count of days will be higher.

Is this accumulative nature of such variables is considered a data leakage between train and test sets (because part of the information that was used in training is being used, in a sense, in testing)?


texts: Which courses should I take in order to learn ML and AI?

texts: I want to learn ML and AI so I want to know which courses(It would be nice if the courses were free) should I take in order to do that and in what order should I learn them. I want the courses to provide a deep understanding of the concepts used.


texts: From Labels to Graph: what machine learning approaches to use?

texts:
Imagine a world where we have places (e.g., cities, restaurants, national parks, etc.) but no roads connecting them.
Our objective is to build roads connecting any two places while going through some given places. For instance, connect Seattle to San Francisco and make sure the connection goes through restaurants X and Y and goes through the Red Wood national park. Note, there is not necessarily one possible road, hence multiple options as output are also acceptable.
We have some roads as examples to train the system. For instance, we have a road from Washington DC to Boston that goes through Baltimore.
So, the input to a machine learning system will be a list of strings (e.g., city names and places to go through) and the expected output is a tree/graph that connects the given points.
(Note, this cannot be a graph traversing problem, as a path the connects the points does not exist.)
I want a machine learning model that can produce/generate a novel graph in the output given a set of labels.
Any thoughts on what ML machine learning methods I can use and where to start?

Update 1
Note that I have neither of the following:

A graph of route(s) from Seatle to San Francisco;
A larger graph that contains a route from Seatle to San Francisco.

Instead, I have graphs for routes from Los Angles to Texas (with stops for eating and seeing landscape), from Milan to Rome, and etc.
Therefore, I think this is not a graph searching problem.
Update 2
I take that the initial question was not clear, so I reworded it.


texts: partial fitting, how to ensure one hot captures all features consistently

texts:
Doing some data science on ~4 million samples, with lots of columns being categorical.

One column has ~1000 categories and my boss insists on including it in the analysis.

My output is also predicting classes (I'll use gnb.predict_proba())

So, I'm taking a random subset of my data for partial fitting, and repeating.

# train = ~3 million rows of data as a dataframe
gnb = naive_bayes.GaussianNB()
for i in range(10):
    dds = train.sample(n=10**4)
    (dfX,dfY) = makeXY(dds) #gets one-hot- encoded X and Y dataframes
    gnb.partial_fit(dfX,[getClass(x) for x in dfY.values],classes=np.unique([getClass(x) for x in dfY.values]))

How can I ensure I get all the possible classes AND that they are in the same order every time?


texts:
Is there a special header structure required for text classification problems?

I'm trying to do Tweet classification using the text mining plugin. I have the data in a CSV file, with one column labeled category and containing my two categories, and the other labeled text containing the tweets.

When I load the file with the corpus widget and follow the workflow given at http://orange3-text.readthedocs.io/en/latest/widgets/bagofwords.html the test widget gives the error "Train data input requires a class variable."

I opened the sample bookexcerpts.tab data and discovered it has a different header structure than just category and text.  If I modify my tweet CSV to match it seems to work. Is this a bug, or am I missing a step?


texts: Is normalizing term weight necessary when cosine similarity is used in retrieval?

texts:
When using cosine similarity in information retrieval, document vector length and query vector length are used for normalization. So if TF-IDF is used as a weighting function, then using raw frequency in TF is equal to normalizing it by document's length (mathematically speaking ).
My question is:
I am using log(TF*SomeValue + 1) as a local weight component instead of TF, where "SomeValue" is a calculated statistical property of the term. Using normalized TF in the formula gave me worse results than using raw frequency! So it's not the same in this case. How can I normalize term local weight to avoid bias toward lengthy documents?


texts:
I know this is an opinion-based question and will be closed but this is the only place I know that can answer it reasonably and it is a very important matter to me.

I am pursuing a machine intelligence track at my university and as I am moving along from cs fundamentals to track courses I am stuck on what math to take first: Linear Algebra or Statistics. I know both are essential pillars and are interdependent, but I am leaning towards  taking stats before linear, which is the opposite of what most ppl do here (hence the hesitation).

Thank you and sorry for making it opinionated.


texts: How to release datasets with fingerprinting

texts:
I intend on monetising some large datasets. These datasets are anonymised and released to (paying) clients via a web api. Are there any standard algorithms such that if the datasets are intentionally leaked publicly, the data can be altered such that the responsible party can be identified, while at the same time the data remains practically useful?

There are certain approaches which come to mind, such as every client's data being very slightly different with known changes. For example in spatial data, every lon/lat pair is altered by the same very small vector. My worry is that if the data is anonymised again by the client before being leaked, a naive attempt might easily be circumvented.

(I am not a data scientist so I'm not really sure what the correct jargon is for what I am looking for)


texts: How interpret keras training loss without compare with validation loss?

texts:
I have several implementation of the same neural network, but each one with different starting parameter.

This is one of my plot comparing the training loss of the base experiment with the training loss of another experiment.

I have also other exaples:

May anyone point me to some instruction on how understand these output from the keras fit()? Note that I don't have any validation set.

Thanks


texts: Is it correct to create topic models using both train and test data?

texts:
I have a dataset of text documents splitted into train and test sets. My task is a binary classification, classifying these documents to either 1 or -1. I have already computed some features using TF-IDF and n-grams and tested my model. Now, I want to add some other features using topic models (LDA and LSA) to see if it can improve the F1-score and performance of my model.

My question is fairly simply: should I create my topic models only using my train set? Or, since topic models are not created based on a label (target or dependent variable,) would it be correct if I use both train and test sets to create the topic model?


texts:
I have a DataFrame in python by using pandas which has 3 columns and 80.000.000 rows.

The Columns are: {event_id,device_id,category}.[]

each device has many events and each event can have more than one category.

I want to run Apriori algorithm to find out which categories seem together.

My idea is to create a list of lists[[]]: to save the categories which are in the same event for each device. like: [('a'),('a','b')('d'),('s','a','b')] then giving the list of lists as transactions to the algorithm. I need help to create the list of lists.

If you have better idea please tell me because I am new in Python and this was the only way I found out.


texts: How to apply transfer learning on a time-to-failure problem?

texts:
I am working on predicting mechanical failures and I have trained a model to predict when a component will fail on what type of machine.

I would like to now use this prediction model to predict when the next type of machine will fail. both type of machines were tested under some similar conditions so they have some overlapping features. However I am not entirely familiar with the algorithnms used for transfer learning and could do with a bit of guidance on how to choose the appropriate approaches and what to consider. I have read some papers and know the main categories of instance based, feature based and model parameter based but knowing which of this 3 would work best for my task is something I am not able to determine. Any advice?