texts: Has anybody created interesting ConvNets on 3D movie data?

texts:
In the last video of his course on Convolutional Neural Networks, Andrew Ng was discussing using ConvNets on 3D input data. He mostly discussed cat scans, but also mentioned you could treat a movie as 3D data and run a ConvNet on it (where the z axis is time, i.e. the # frame of the movie).

I find this pretty fascinating. I'm just curious if anybody is aware of anybody doing something like this in practice, and what they accomplished.


texts: ways to represent document by its keyword vectors

texts:
I have documents, say for example, D1, D2, D3... Dm.

Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components varies between documents.

What are the ways to find how close Di are? Or what is the best way to represent a document using its keyword vectors? Please note that I'm using a custom embedding here.


texts: How to reduce dimensionality of audio data that comes in form of matrices and vectors?

texts:
I'm working on a project involved with identifying different types of sounds (such as screams, singing, and bangs) from each other.  We've got our data a reasonable number of different transformations (e.g.: spectrograms, chromagrams, MFCCs, etc.), but since most of our features are 2-dimensional matrices (some are actually 1-dimensional vectors), we'd like to reduce this information in some way, so that the machine learning we're hoping to do takes a "feasible" amount of time.  However, I don't quite know enough about math and statistics to make an educated decision on this.

Our data consists of small sound files from ~1-10 seconds long.  There are recordings of screams, singing, bangs (and other man-made noises), and birds (and other natural noises).  We are hoping to be able to differentiate and identify each source type from the others.  See https://github.com/BenSandeen/surveillance_sound_classifier/blob/master/Project.ipynb for the different plots we have made to guide our selection of features to use.  Focus mainly on the 3x3 plots, as that's where the comparisons are being made.  These plots are primarily time vs. frequency, with amplitude represented by color.

I was thinking that maybe we could "collapse" each matrix down to a vector by somehow choosing some representative frequency/amplitude-related feature at each time slice (we're using Short Time Fourier Transforms to analyze the sounds) and then get a vector of some length, containing a bunch of scalars.  Although this could make accounting for different lengths of sounds difficult.  Would it be reasonable to just set the shorter sounds' vectors to be filled in with zeros if they have no useful data?  That would effectively make these sounds a projection onto some lower-dimensional space.  Then, maybe we could just use dot products to compare the vectors; if they're parallel, they'll have a large for product, but of they're nearly perpendicular, they'll have a dot product near zero.

Alternatively, I was thinking that something like a trace of our matrices, or finding their characteristic polynomial, might be useful direction to pursue.  I've read a bit about PCA, but I'm not quite understanding it enough to know if this may be what I'm looking for.

Can anyone think of any other ways of handling and reducing this data?  For what it's worth, we're currently planning on using Sci-Kit Learn (sorry that I can't use more than 2 links yet) to perform our machine learning.


texts: Difference between MDS and other manifold learning algorithms

texts:
From sklearn docs:

  Note that the purpose of the MDS is to find a low-dimensional representation of the data (here 2D) in which the distances respect well the distances in the original high-dimensional space, unlike other manifold-learning algorithms, it does not seeks an isotropic representation of the data in the low-dimensional space.

Can someone elaborate, in layman's terms, what the distinction is?


texts: How to build a OCR for reading text from image?

texts:
This is my first time working on a OCR application. I have lot of scanned images of English text-book pages like this and I want to build a OCR using DL to extract the text from each image.

  Can anyone outline just the approach I need to take to implement it?

There are few questions that I couldn't figure out:

1) How do I need to do it? I mean I don't have any target label (or) annotations
   (or) bounding box.
2) What will be my target label be here? Do I need target label?

3) Is it possible to do un-supervised learning. If so How?

4) What are CRNN algorithms I need to check out to implement it?

Not looking for any api's (or) libraries (like teserract, vision-api etc..).


texts: How can I prove bottleneck layer of my CNN auto encoder contain useful information?

texts:
I am using CNN autoencoder to create a state representation layer which I will later be feed into my Reinforcement Agent. So I trained my CNN autoencoder and it is giving nice state representations. But I have following questions,

Can my autoencoder layer be overfitted
If there's a overfit will it cause rubish information in my bottleneck layer?


texts: Is it ok to train the model only on the interested part of the data?

texts:
Let's say I have a dataset where one feature is 'Car type' : say 'A', 'B' and 'C'.
The test set consists of samples where 'Car type' is always equal to 'A'.
Therefore, should I train my model only on the subset where 'Car type' is 'A' or on the whole training set?
What are the pros and cons of both approaches?


texts:
When I read several statistical papers, they mention "oracle" property or "oracle" estimator.
What do they mean by "oracle"?
I understand this oracle is not a company name, but have no idea what this means.

For example,
Candes and Tao 2007 is about "oracle" inequality in a context of large dimensional estimation of linear model (p>>n).
https://projecteuclid.org/euclid.aos/1201012958)

Zou 2006 discusses "oracle" property of Lasso.
(http://www.tandfonline.com/doi/abs/10.1198/016214506000000735)

I am not familiar with this field but I have to read one paper which discusses high dimensional test in statistics.


texts: Preparing and combining datasets for predictions

texts:
I am trying to create and train a model to predict the winner of a professional DOTA match. This model is only for "fun" and shouldn't be used for professional purposes but is clearly for learning.
I know that a very important part of ML is to prepare your data. In my case, I have two sets of data that I am not quite sure how to combine.
First, i have a dataset of the teams overall performance that looks something like this:

Other than that, I have individual match performance:

I also have data or match that shows individual skill level for the players per match:

Now my question is, how do I prepare my data to be put into the model? How can I combine such very different datasets into a dataset that is usable for training a model on it?
My end goal is to input two teams and predict a match winner


texts: Find a Default Assumption for a series of Forecasted Loans

texts:
Your help would be appreciated!

I'm trying to calculate the net returns of a series loans, at different stages of completion. I've categorised the loans into open loans (repaying & late loans) and closed loans (repaid & default).

For the Closed Loans, I have put the loans into a cashflow and used IRR methodology to approximate the net returns.

For the Open Loans, I have put these loans into a similar cashflow, assuming they complete as scheduled and don't default.

Default Assumption

I'm trying to come up with a sensible default assumption to discount the open loans, based on the default performance of the previous years.

Tried Methodology:

-

Proposed Methodology enter image description here This methodology attempts to calculate the impact on net returns of defaulted loans across all years (2010-2016). We then take the average and compare against the actual default and discount the difference from the overall net returns. As the majority of loans are open on in the later years, particularly 2016, the impact of this discounting is great.

Calculate the % of closed (repaid+ defaulted) loans. In 2010 all
loans are closed so 100%. In 2016 only 7% of loans are closed. Use
this to calculate a weighting for averaging. Hint: the loans in 2010
are more influencial in the average that 2016 because all loans are
closed.
The overall net returns is calculated by a summation of:

a. % of repaid loans x repaid loans net returns
b. % of defaulted loans x  defaulted loans net returns (negative number)
c. % of late loans x late loans net returns
d. % of repaying loans x repaying loans net return
Take the weighted average, using the figures calculated in 1
of the (% of defaulted loans x defaulted loans net returns. This
single figure gives an effective default rate on the overall net
returns of the total portfolio.
Use this new estimate to discount the open loans by difference between the new calculated average and the actual default.

-

I can't get my head around this calculation and whether I'm way off. Be good to get your thoughts. I've attached some data to play around with.

Cheers, Iain


texts:
I have read that seq2seq is a network, similar to other networks types (CNN, RNN, ...). However, in my opinion it is actually an architecture for RNNs. Isn't that?

For example, when input and output lengths are not equal, LSTM is not applicable. LSTM is an architecture for RNNs. Instead another architecture is used consisting two units: encoder and decoder and they are built with RNNs.

So, is seq2seq a subset of RNN? a sibling of LSTM? Or it is a sibling of RNN and CNN?


texts: which neural network topology to learn correlations between time series?

texts:
I have two (or more in principle) 1xN time series, and I would like to train a NN to predict the next value of both. I can arrange them as a 2xN matrix and feed a window from this matrix as input to the NN, but I'm not sure how to structure the NN itself.

I have made a NN with convolutions that can do a pretty decent job with a single series, but I'd like to exploit cross-series correlations. What topology works to let the NN notice correlations between the time series?


texts: AWS machine learning prediction schema problems

texts:
I've trained an AWS Machine Learning model with the training data from here : https://www.kaggle.com/c/titanic/data

I'm now trying to run a batch prediction with the test data from the same source but I get the following error when I try to load the data : "  The schema in this data file must match the datasource used to create the ML model ml-xxxxxxxxx. Ensure that the data file you are using matches the schema structure."

The schema, as far as I can see, is identical. I have tried it with and without the 'survived' column which is the value I'm trying to predict. I even tried it with the same training set which obviously has an identical schema and got the same error.

What am I doing wrong?


texts: How to implement word to word Co-occurence matrix in python

texts: To implement co-occurence matrix in sucha a way that number of times word1 occured in context of word2 in neighbourhood of given value, lets say 5. There are 100 words and a list with 1000 sentences. So how can i calculate co-occurence matrix of size (100* 100) using python?


texts: when will the incareful features harm the model?

texts:
I am working on financial prediction problem(time-series prediction problem).
I think feature engineering is importance in this problem. So i am careful to check the feature's effectiveness. And i perfer linear regression, because i think it's easy to explain, and I am not good  at machine learning models
But i think i have another method: build a lot of features without careful check, select good model to handle it.
So, my question is:
In which model, add what kind of features, will harm this model's ability?
And Is there any models, i can just add without worrying the harm of trash features? in this model, what should i pay attention to?


texts: Which machine learning algorithm can be used for traffic prediction using images from Google Maps?

texts:
I am doing a traffic prediction project.
I am using data from Google Maps. I'm collecting maps of different location and analyzing them. I am converting the maps to images, analyzing the image, and predict traffic.

Which machine learning algorithm can be used for this prediction?


texts: How to determine whether a bad performance is caused by data quality?

texts:
I'm using a set of features, says $X_1, X_2, ..., X_m $, to predict a target value $Y$, which is a continuous value from zero to one.

At first, I try to use a linear regression model to do the prediction, but it does not perform well. The root-mean-squared error is about 0.35, which is quite high for prediction of a value from 0 to 1.

Then, I have tried different models, e.g., decision-tree-based regression, random-forest-based regression, gradient boosting tree regression and etc. However, all of these models also do not perform well. (RMSE $\approx $0.35, there is not significant difference with linear regression)

I understand there are many possible reasons for this problem, such as: feature selection or choice of model, but maybe more fundamentally, the quality of data set is not good.

My question is: how can I examine whether it is caused by bad data quality?

BTW, for the size of data set, there are more than 10K data points, each of which associated with 105 features.

I have also tried to investigate importance of each feature by using decision-tree-based regression, it turns out that, only one feature (which should not be the most outstanding feature in my knowledge to this problem) have an importance of 0.2, while the rest of them only have an importance less than 0.1.


texts:
Keras mentions that it provided models pretrained on ImageNet. However, it doesn't specify what they mean by "ImageNet" - like is it a certain subset of ImageNet of the complete set of images? I mean, I have a hard time imagining that they have used all the 14 million images for pretraining but maybe I am wrong.
I have read quite a lot of research papers about some of the proposed architectures and they are all trained on subsets of ImageNet specific to the ImageNet Large Scale Visual Recognition Challenge and this is explicitly mentioned in the papers. Of course, if these Keras weights are actually representing the whole ImageNet, there is no problem :-) But would like some sort of confirmation/affirmation.
Does anyone know more specifically what is mean by "ImageNet"? https://keras.io/api/applications/


texts: Question regarding: Vectorization Math of Backpropagation in a Neural Network

texts:
Formula:
These are the formula I use for backpropagation from Brilliant:

Question:
If we consider a Neural Network with the structure (3,2):

And we would start calculating the derivative (for 1 trainings example), we would have a (2,1) matrix for the error term.
Since $\hat{y} - y$ is a (2,1) matrix and $g_0'(a_1^m)$ is also a (2,1) matrix and we multiplicate them element-wise with each other.

If we know want to proceed and calculate the partial derivative with respect to w we would need to multiply $\delta$ which is a (2,1) matrix with $o_i$ which is (3,1) a matrix which is not possible.

So what to do? :/


texts: Training a VAE for random number generation

texts:
I have a high-dimensional multi-modal distribution of random numbers in R^n and consider training a VAE to learn the distribution. What I want to do with it succeedingly, is to sample from the latent distribution such that I can generate pseudo-random numbers with the decoder that behave similarly to the original input distribution.
My concerns are the following:

Is my approach fundamentally or logically flawed? Specifically, because unlike image data, the random numbers can take negative values and the input data can be considered noisy.
How do I find the right architecture aside from simple trial and error?


texts:
PROC means data=d mean;
var a;
class b; var a;
run;

I want to perform the "PROC means" for continuous "var a":
1) in general and
2) by classes.
But it performed by the classes only.
How to make procedure for "var a" here in general too?

P.S. SAS WARNING: Analysis variable "a" was defined in a previous statement, duplicate definition will be ignored.


texts: Datasets with 50+ attributes and 2+ tables

texts:
I'm looking for a data set to use in my Master Thesis. I've looked on Kaggle but all of them seem to be smaller data sets. Ideally it would be CSV files with 50+ attributes and 2+ tables, but not much bigger than that as it still needs to be appropriate for a Master Thesis.
Could anyone recommend websites to look for such (free to use) data sets?


texts: Training dataset decreasing in quality (Google data science blog)

texts:
I have a complex algorithm that decides when it should show customers of an only shop an ad on our website, after they log in, in hope that they will buy what is in the ad. We have no control what is in the ad, because another department of our company handles that - our model only chooses to whom the ad is shown (we would like to keep the number of people to whom we show the ad as low as possible, because we will just offend customers with the ad otherwise; this means don't show the ad to people who log in with an intent to buy something, only show it to those that would not buy anything if they weren't presented an ad). The model was training using training data where we randomly showed some people ads and some not and recorded their respective response.

An initial implementation of the model is running live, and works decently well in classifying to whom it makes sense to show the ad. But each day new data is acquired and the model is retrained using the whole dataset (which with each retraining gets larger) every few days. But there is the worry that its performance might decrease if we keep doing this, because the current model instance influences the new data that comes in, that is used to train future model instance: The percentage of the customer who will not be shown ads randomly will decrease and there will be more and more data of customer that were shown an ad only if the model believes they will buy what is on it.

Are we right to worry that our data will become biased in the future?

The [unofficial Google data sicence blog][2], section "Using randomization in training", might contain useful information about this problem, but unfortunately it is too technical for me to make much sense of all of it. Other than the message "Yes, worry about this: but keep sending out a small part of ads randomly and you will be fine", which I got, what else could I do to reduce this problem?


texts: Learning based on Vector Inputs and Scalar Output

texts:
I'm trying to fit a function of the following form to some time series data that I have...

Z = X/(a*Y+b)

a and b are constant values, and the form of the data for X, Y and Z are vectors of arbitrary length.  I'm trying to figure out how to learn the a and b values given that I have measured X, Y and Z values.  The tricky bit, is that although I have the full vectors for X and Y, I only have a summed up scalar value for Z.

I do have a fair amount of data to lean on.  My hope is that although I don't know the full Z vector, only the scalar sum of the values, I might be able to still, by knowing X and Y, learn the a and b constants somehow.

My question is, is there a method that would be well suited to this problem?  I feel like it would be very straightforward if Z was a vector, but as a scalar, it seems like a lot of information is lost.


texts:
Random Forest VS LightGBM

Can somebody explain in-detailed differences between Random Forest and LightGBM? And how the algorithms work under the hood?

As per my understanding from the documentation:

LightGBM and RF differ in the way the trees are built: the order and the way the results are combined. It has been shown that GBM performs better than RF if parameters tuned carefully.

Random Forest: RFs train each tree independently, using a random sample of the data. This randomness helps to make the model more robust than a single decision tree, and less likely to overfit on the training data

My questions are

When would one use Random Forests over Gradient Boosted Machines?
What are the advantages/disadvantages of using Gradient Boosting over Random Forests?


texts: Can i use survival analysis to predict if a person will "die" or not, and then get the survival time if the person does?

texts:
I want to determine, given a project, "How long will it take for this project to be successful ?"

Therefore, survival analysis seems like a perfect fit in this case (as I do have some projects that ended up as censored data as they took too long to be successful).

However, I also want to build my model so that, if a project seems like it will take too long (for instance it will be more than 2 months - the duration of the study) it will tell me not to accept it.

To sum it up ideal model for me would be :

Input : a project (with features like Founding, Past success, Location, etc.)
The models tells me if I should accept the project or not
If I should accept it, how long will it take before being successful ?

Is it doable with only survival analysis or should I consider adding a classification algorithm before that ?


texts: How can I infer no target in a target classification problem based on deep learning?

texts:
Let's take the MNIST dataset (My application is different) with a lot of noise, I am going to train a deep NN to classify the letters.
What's the right way to infer, there's no letter possibility? or a letter not included in the training set?
Do I have to add the class "not a letter"  and try to pick or simulate the part of the dataset that has no letters but noise,
Or, I can do it without adding the extra label?


texts: How can I annotate labels to images automatically?

texts:
I have 80 classes, for each class, there are 100 images. I want to label all images for object detection.
For this task I have downloaded the LabelIMG tool but it's taking a long time to do it. How can I automate labeling?


texts: Neural Networks getting stuck at local optima

texts:
I'm training a NN with 8 features and 8000 training examples with a single output (0, 1) using the scipy.optimise CG algorithm and the results are somewhat inconsistent. The goal is to get the NN to be as 'precise' as possible (recall doesn't really matter too much) so I've set the threshold for y value quite high (0.75). Most of the time it gets a precision of around 80%, however sometimes it fails (using exactly the same parameters, lambda etc..) to generate any outputs which are above the 0.75 threshold, meaning the precision equals 0.

I've successfully trained NNs before without these unusual results (albeit the goal was a somewhat more conventional multi-class classifier with many more features).

I'm wondering if the training NNs with fewer features increases the chances of it getting stuck at a local optima; or getting stuck at local optima has a more significant impact on NNs with fewer features?

Any thoughts on what's going on!?


texts: Anomaly detection on sparse categorical data

texts:
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ones) and being able in the future to identify new strange combinations of future clients immediately.

clientid
choice

cl1
a

cl2
b

cl2
c

cl3
d

cl4
b

cl4
c

If I transpose the table by clientID I have a row for each client and different columns based on the choices, it will became a sparse dataset with categorical variables (choices). Some clients have only one choice and some have multiple ones and I want to find outlier records (clientid)
Which type of algorithm could help me in this type of problem?
It is unsupervised, so I dont know what are the normal combinations and it is sparse data on categorical variables.


texts:
Suppose I am training a model for binary classification and I have a categorical feature which has been one-hot encoded. Suppose the feature is Gender and the one hot encoded features will be Gender_Male and Gender_Female. Now all values in these two columns will be 0/1. Now if I use f_classif function from sklearn.feature_selection on any of these features say, Gender_Male, the function call will be like this:
f_classif(df['Target'], df['Gender_Male'])

Will this value really make any sense? Or is it better to use Chi2 in this case?


texts: Feature engineering for hierarchical data

texts:
I am working on the KDD dataset given in this link.

The dataset is related to a typical recommendation systems dataset. So you find an item and information about the item. One of the information given about the Item is its category. Item-Category is a string a.b.c.d, where the character delimits the categories in the hierarchy ., ordered in a top-down fashion (i.e., category a is a parent category of b, and category b is a parent category of c, and so on.

I'm not sure how to use this information correctly in my feature engineering. For example, the simplest information I can derive is for each Item I can estimate the topmost category to which it belongs. However, to go beyond this and use the subcategory information, how can I model this feature for a linear regression?


texts: AI for Robotics, Udacity, Python, normalizing x,y error in toroidal world

texts:
def eval(r, p):
    '''
    param r: robot
    param p: resampled probabilities array of particles
    '''
    sum = 0.0;
    for i in range(len(p)): # calculate mean error
        # dividing by world size ensures that toroidal world doesn't skew the resulting estimated error enclosed in
        # the boundary
        dx = (p[i].x - r.x) + (world_size/2.0)) % world_size - (world_size/2.0)
        dy = (p[i].y - r.y) + (world_size/2.0)) % world_size - (world_size/2.0)
        err = sqrt(dx * dx + dy * dy)
        sum += err
    return sum / float(len(p))

This is a robot localization problem in the Udacity AI for Robotics course, section on particle filters.  The world is of size 100 *100 and p[i].x - r.x is the x coordinate of the particle - that of the robot) so it's an error calculation.  The world is toroidal or cyclic in Professor Thrun's words, so he adds world_size/2.0 and then modulo world_size then subtracts (world_size/2.0) so that in his words, "so that the cyclic world does not skew the resulting estimated error enclosed in the boundary."
He gives examples of how if a robot is at 99.9 or 0.0, it should be basically next to each other but the error calculation would consider them far different without this "normalization" procedure.

The video is Lesson, number 24 in the Udacity course mentioned above.
Does anyone know how this works?  I am not that strong in math.


texts: Are there any open-source text annotation for multi label classification tools?

texts:
I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation.

Each sentence takes two labels. If there are any please let me know.


texts: how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

texts: i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized image of 90 dpi  into OCR it barley recognizes any text in the image .


texts: Image features (spectral bands) other than RGB for image analysis

texts:
What features can I derive from the image, other than its RGB, to help me analyze an image?

Actually, I'm looking for some papers, etc about this. But I couldn't find.

I know I can use some features like its lightness, etc. But I don't want a single value as lightness, I want a 2d array just like R or G or B.


texts: ValueError: Input 0 of layer conv1_pad is incompatible with the layer: expected ndim=4, found ndim=3. Full shape received: [None, 224, 3]

texts:
I am trying to make a gender classifier. I am using MobileNet from Tensorflow with input shape as (224,224,3). After training the model, I tried to check if the model was working by passing an image to the model to predict, but it throws the error in the title. I tried debugging it and google the error and tried everything I could, but the error still persists. Please help me to understand why the error.
model = tf.keras.applications.mobilenet.MobileNet(input_shape = (224,224,3), include_top = False, pooling = 'avg')
#model.summary()

x = Dropout(rate=0.4)(model.output)
x = Dense(3)(x)
x = Softmax()(x)
model= Model(model.inputs, x)

for layer in model.layers[:-3]:
    layer.trainable = False

model.compile(
    optimizer=Adam(lr=0.001),
    loss='categorical_crossentropy'
)

datagen = tf.keras.preprocessing.image.ImageDataGenerator(preprocessing_function = tf.keras.applications.mobilenet.preprocess_input,
          shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True, validation_split = 0.1)
training = datagen.flow_from_directory(
    '...\Pictures\Model',
    target_size=(224, 224)
)

validation = datagen.flow_from_directory(
    '...\Pictures\Model',
    target_size=(224, 224)
)

batch_size = 32

history = model.fit_generator(generator = training, steps_per_epoch = training.samples//batch_size, epochs = 10,
                              validation_data = validation, validation_steps = validation.samples//batch_size, verbose = 2)

man = np.array(tf.keras.preprocessing.image.load_img('...\\Pictures\\Detected Faces\\0156224224_face.jpg', target_size = (224,224)))

model.predict(man) #Here the model throws an error


texts:
I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better)

I've run them 9 times each, yielding the following scores

Architecture 1 | 36.52 |
36.27 |
35.9 |
35.22 |
37.13 |
35.53 |
35.3 |
34.14 |
35

Architecture 2 | 36.85 |
35.64 |
36.37 |
36.82 |
36.74 |
36.46 |
35.77 |
37.31 |
36.68

Means are 35.67 for the first and 36.52 for the second

I want to calculate a P-value that confirms (or not) the superiority of the second model, with alpha = 0.05

Thanks in advance


texts: How does one individually extract elements from a tensor?

texts:
Let us assume that I have a tensor in tensorflow with 2 elements.

How do I extract these individual elements and perform calculations on them?

def multiplication(a):
    return a[0]*a[1]

def main():
    sess=tf.Session
    a=tf.constant([[2,3]])
    answer= multiplication(a)
    print(sess.run(answer))
    #answer should be 6


texts: How can I read input node and output node from .h5 file in keras model?

texts:
I trained my own dataset using MaskRCNN. My trained file is in .h5 file format. Now I want to know/read input node names and output node names.

How can I read?


texts: Query regarding (.output_shape) parameters used in CNN model

texts:
I am applying CNN model on my dataset for predictions. After reshaping the dimensions, the input_shape of my model1 becomes:

model1.input_shape:
(None, 1, 3, 4)

then i apply CNN ist input layer defined below:

model1.add(Convolution2D(128, (2,2), border_mode= 'valid' , input_shape=(1,  3, 4), activation= 'relu'))

here above 1 is number of channel, 3,4 represents my nodes means 12 input nodes or features, now wehn i check the output_shape of model1, it is:

model1.output_shape:
 (None, 128, 2, 3)

here 128 are number of neurons i specified in my input layer. My question is what does the elements 2,3 shows here?


texts: Is it necessary to tune the step size, when using Adam?

texts:
The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have:

keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)

The first hyperparameter is called step size or learning rate. In theory, an adaptive optimization method should automatically modify the learning rate during optimization. Thus, I would expect lr to be a not very influential hyperparameter, which I can remove from the list of hyperparameters I have to tune on the validation set, thus saving some time.

Does this hold true in practice? I.e., is it true that, at least for a class of problems (say, image recognition), lr doesn't affect the optimization a lot, and thus we can just leave it at the default value of 0.001? Or is it still as extremely influential, as it is for SGD with momentum?

EDIT: to clear a misunderstanding brought up in an answer, learning rate and step size are synonymous. See the algorithm definition in the Adam paper: $\alpha$, whose default value is 0.001, is clearly named the step size. I reckon it's probably not a great name (the actual size of the step in the parameter space depends on the accumulated first order and second order momentums, as well as on the gradient, of course) but unfortunately such misleading terminology is the norm in optimization (at least in Deep Learning papers).


texts: Issue translating large amounts of tweets using Google Translate

texts:
I am working on translating large amounts of tweets using this deep-translator which uses the Google Translate API.
Initially everything was fine and tweets were translated with no problems whatsoever but I recently encountered an issue.
The issue with the data currently is that when there are non-English words in a particular tweet as in the attached image, Google Translate marks it as English and does not translate them. In the API I have set the source language to auto-detect the words then translate them to English.

The only workaround I have come up with to solve this is to turn the tweet into chunks and perform batch translations on them.
Example string:
"NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM NEWS TRANSFER WINDOW UPDATE 손흥민 Son Award Link to Premier League Defen... TOTTENHAM NEWS TRANSFER WINDOW UPDATE Carabao Cup Win Final. 손흥민 Son Contract"

Output after chunking the string into batches with a maximum of 5 words:
['NEW YOUTUBE VIDEO OUT NOW:TOTTENHAM', 'NEWS TRANSFER WINDOW UPDATE 손흥민', 'Son Award Link to Premier', 'League Defen... TOTTENHAM NEWS TRANSFER', 'WINDOW UPDATE Carabao Cup Win', 'Final. 손흥민 Son Contract']

The issue is to translate the batch it takes about 13 seconds whereas if I translate the entire string(even though it won’t work) it takes under a second as the batched string performs 6 API request compared to the single API request for the normal string.
The time to translate will be quite high when I batch the strings and translate them.
Assuming it takes 13 seconds per tweet with 5000 tweets in the csv file I am currently working on it will take roughly 18 hours and I have csv files with significantly more tweets than this.
The chunk size of 5 is what I found to still be able to translate the words even if there is a single non-English word in the string. Anything more than that will still not translate the tweets.
Anyone have any workarounds that are faster than this?


texts:
Can anyone explain how field-aware factorization machines (FFM) compare to standard Factorization Machines (FM)?

Standard:
http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf

"Field Aware":
http://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdf


texts: How can I see whether a CNN layer is overfitting, underfitting or whether the activation function, normalisation requires some changes?


texts: Why a significant risk factor doesn't increase AUC-score/F1-metric?

texts:
I have a binary classification problem with 5K records and 60+ features/columns/variables. dataset is slighlt imbalanced with 33:67 class proportion

What I did was

1st)  Run a logistic regression (statsmodel) with all 60+ columns as input (meaning controlling confounders) and find out the significant risk factors (p<0.0.5) from result(summary output). Because I have to know that my risk factors are significant as well i.e causal modeling. Meaning build a predictive model on the basis of causal features. I say this because in a field like medical science/clinical studies, I believe it is also important to know the causal effect. I mean if you wish to publish in a journal, do you think we can just list the variables based on feature importance approach (results of which differ for each FS approach). Ofcourse, I find some common features across all feature selection algorithm. But is this enough to justify that this a meaningful predictor? Hence, I was hoping that p-value would convince and help people understand that this is significant predictor

2nd)  Use the identified 7 significant risk factors to build a classification ML model

3rd)  It yielded an AUC of around 82% in test data (30% of split from full dataset)

Now my question is

1) Out of 7 significant factors identified, we already know 5 risk factors based on domain experience and literature. So we are considering the rest 2 as new factors which we found. Might be because we had a very good data collection strategy (meaning we collected data for new variables as well which previous literature didn't have)

2) But when I build a model with already known 5 features, it produces an AUC of 82.1. When I include all the 7 significant features, it still produces an AUC of 82.1-82.3. Not much improvement. Why is this happening?

3) If it's of no use, how does statsmodel logistic regression identified them as significant feature (with p<0.05)?

4) I guess we can look at any metric. As my data is slightly imbalanced (33:67 is the class proportion), I am using only metrics like AUC and F1 score.

5) Should I balance the dataset because I am using statsmodel Logistic regression to identify the risk factors from the summary output? Because I use tree based models later to do the classification which can handle imbalance well, so I didn't balance.Basically what I am trying to know is even for `significant factor identification using statsmodel logistic regression, should I balance the dataset?

6) Can you let me know what is the problem here and how can I address this?

7) How much of an improvement in performance is considered as valid/meaningful to be considered as new findings?

POST UPDATE

You might wonder why do I have to do Xgboost when I do logistic regression in the first place. I do this because with Logistic regression yields only an AUC of 79-80 whereas Xgboost results with an AUC of 81-82. Reason why I do Logistic regression in the first place is because I need to find a way to justify that these variables really explain the outcome. Hence thought choosing p-value might be best approach which is usually found using Logistic regression. Problem with Feature selection approach using different algorithms is it returns different subsets with different ranking order of features. Though I find common features across all FEATURE SELECTION algorithms, I feel there is no way to justify that these variables really explain the outcome. They might have good predictive power but do are they also responsible for causing a change in the outcome? Hope I made my objective clear. Request you to let me know for any questions


texts: Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?

texts:
I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29:
slideshow

\begin{align}
\mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\theta (s,a)B(s)] &= \sum_{s \in S}d^{\pi_\theta} (s) \sum_a \nabla_\theta \pi_\theta(s,a)B(s)\\
&=\sum_{s \in S} d^{\pi_\theta} B(s) \nabla_\theta\sum_{a \in A} \pi_\theta(s,a)\\
&=0
\end{align}

Consider in this proof replacing $b(s)$ with the critic's quality estimate $Q_w(s,a)$ (see previous slide(s)). How does this proof not also show that the gradient of the objective function $\nabla_\theta J(\theta)$ should also be $0$? Does this have to do with the second summation term changing from being over $a$ to over $a \in \mathcal{A}$?

Thank you.


texts: Is the PR AUC invariant under label flip?

texts:
The ROC-AUC curve is invariant under a flip of the labels. I don't know if its a famous result so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not been able to prove or disprove it yet.
The reason this is important is that when data sets are highly imbalanced ROC-AUC curves perform well mechanically and it is better to look at PR-AUC curves. This obviously raises the question of a better curve for PR-AUC for one class being better for the other class as well or not.
=== Proof that ROC-AUC is invariant under label flip ====
$$
\begin{array}{|c|c|c|}
\hline
&predicted~positive & predicted~negative \\
\hline
\hline
Positives & TP(\xi) & FN(\xi) \\
\hline
Negatives & FP(\xi) & TN(\xi) \\
\hline
\end{array}
$$
where $\xi \in [0,1]$ is the output of a classifier.
For a fixed dataset we have
$$
TP(\xi) + FN(\xi) = P \\
FP(\xi)+TN(\xi) = N
$$
where $P$ and $N$ are actual number of negatives and positives. The ROC-AUC defined wrt to this definition of positives and negatives is
$$
A = \frac{1}{PN} \int_0^1 TP(\xi)~d FP(\xi)
$$
Now lets look at what happens under a label flip. We get
$$
\begin{eqnarray}
\tilde A &=& \frac{1}{PN} \int_1^0 TN(\xi)~d FN(\xi) \\
&=& - \frac{1}{PN} \int_1^0 (N-FP(\xi)~d TP(\xi) \\
&=&  \frac{1}{PN} \int_0^1 (N-FP(\xi)~d TP(\xi) \\
&=& \frac{1}{P} TP(\xi) |_0^1 - \frac{1}{PN} (TP(\xi))(FP(\xi))|_0^1 + A \\
&=& A
\end{eqnarray}
$$
where the last equality follows from $TP(0)=0$, $TP(1)=P$ and $TN(1)=N$.


texts: How to use a RBF kernel to create a "Kernel Space" using the similarity of each pair of point?

texts:
I am trying to use Semi-Unsupervised clustering using reinforcement learning following this paper.

Assume I have n data-points each of which has d dimensions. I also have c pairwise constraints of whether two elements are supposed to be in the same cluster or not.

The paper states that "the original input dimension of the dataset is
appended to a kernel space with a similarity metric to each
pairwise point in the set of constraints" creating a d + 2c dimensional space. They also say the kernel they use is an RBF.

Can anybody explain to me what do they mean? how can you use the similarity of a pair of elements to generate a whole new dimension for every data point in the dataset? what is a "kernel space" exactly in this case? if I try to google it I only get results relevant to operating systems kernels.


texts: Properly using activation functions of neural network

texts:
I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear.

Input Layer:

From my understanding the first step in the neural network is to weight
inputs by picking the "best" linear function.

Methods used for finding optimal weights (may not be relevant to problem):

   For finding optimal weights solving quadratic minimization problem is what I usually do, perhaps by finding global minimum of the convex (quadratic error $||Ax-b||^2$ in this case, $A$ being basis matrix and $b$ being ideal vector) function (since every critical point is global minimum there) or orthogonal projection equations (since error is orthogonal to the column space of basis, and their dot product gives us the equation - $A^T(Ax-b$), equating it to $x$: $x = (A^TA)^{-1}A^Tb$.

Once best linear coefficients are found (say $m$ as slope and $c$ as bias), next step is to weight the inputs by evaluating them into the function $f(x)=mx+c$.

Hidden Layer:

After all inputs are weighted, they must be summed up which gives us a constant number.

I understand that this constant number must then be inputted to some activation function (perhaps sigmoid: $f(x)=\frac{1}{(1+e^{-x})}$ or reLU: $f(x) = max(0, x)$.

Representation of simple neural network with single hidden layer having sigmoid as activation function:

Picture reference

Problem:

But constants that are evaluated in these activation functions obviously output constants again, so how can they be utilized for data prediction?

For example, say sum of all weighted inputs is $15$, then its evaluation in sigmoid function will be $\frac{1}{(1-e^{-15})} = 0.99999969409$. That constant number can't be utilized for data prediction (classification/regression) so what are next steps to take? If activation function returns a constant number, how can the data be predicted for different input variables?

Am I having incorrect perspective for activation functions?


texts: About the time differences in the Bellman equation

texts:
I am trying to grasp fundamental mathematics behind the Reinforcement Learning and so far I have unterstood how the Value Iteration and Policy algorithms do converge (contractions, etc.)

I have still some problems about understanding the Bellman Equality. The Value function for a state $s$ under a policy $\pi$ is the expected discounted cumulative reward:

$$ V^\pi(s_0=s) = \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t R(s_t,\pi(s_t)) |s_0=s\right]$$.

During the derivation of the Bellman Equations, when the expected cumulative rewards are calculated on an infinite horizon, meaning $t \to \infty$, shouldn't they be equal for different time steps; like $V^{\pi}(s_i=s) = V^{\pi}(s_{i+\Delta i}=s), \forall s$?
Since the summation is an infinite series, the discount factor $\gamma$ always starts with $0$ in the exponent and the state and reward distributions are stationary, as long as the the starting state is the same, the cumulative reward must be also the same:

$$ V^\pi(s_i=s) = \mathbb{E}\left[\sum_{t=i}^{\infty}\gamma^{t-i} R(s_t,\pi(s_t)) |s_i=s\right]= \mathbb{E}\left[R(s_i=s,\pi(s_i=s)) + \gamma R(s_{i+1},\pi(s_{i+1})) + \gamma^2 R(s_{i+2},\pi(s_{i+2})) + \dots\right]$$

$$ V^\pi(s_{i+\Delta i}=s) = \mathbb{E}\left[\sum_{t=i+\Delta i}^{\infty}\gamma^{t-i-\Delta} R(s_t,\pi(s_t)) |s_{i+\Delta i}=s\right]=
\mathbb{E}\left[R(s_{i+\Delta i}=s,\pi(s_{i+\Delta i}=s)) + \gamma R(s_{{i+\Delta i}+1},\pi(s_{{i+\Delta i}+1})) + \gamma^2 R(s_{{i+\Delta i}+2},\pi(s_{{i+\Delta i}+2})) + \dots\right]$$

Is this true for the Bellman Equation?


texts: How to add non-image features along side images as the input of CNNs

texts:
I'm training a convolutional neural network to classify images on fog conditions (3 classes). However, for each of about 150.000 images I also have four meteorological variables available that might help in predicting the classes of the images. I was wondering how I could add the meteorological variables (e.g. temperature, wind speed) to the existing CNN structure so that it can help in the classification.

One way I can already think of is creating another (small) feedforward neural net alongside the CNN and then concatenating the outputs of the CNN layers and the hidden layers of the non-image neural net to each other at the dense layer.

The second way I could think of is just contacting these features to the dense layer. However, in this case, the non-image variables will (I think) only be able to make linear predictions.

Are there any other (better) ways that the non-image features could be included in the model? And what would be the advisable method considering the amount of data I have?

Another question I have is whether or not I should unfreeze the convolutional layers while training with these non-image features? These layers of a Resnet-18 (which were initialized as pre-trained on ImageNet) have already been fine-tuned using the images. My guess is that I should keep them frozen and only unfreeze the dense layer since it is only here that the non-image features come into 'contact' with the image features (not earlier in the CNN). If I'm wrong on this, please say so!


texts:
I have a very small dataset with 567 images. I've used pretrained model resnet50 in transfer learning. But whenever i fit my model it's giving me 100% accuracy , 100% validation accuracy and in per epoch loss and validation loss goes to negative. The evaluation shows me output with 100% accuracy and -5915788.088888888  loss.This is preety unexpected result. What's the matter ? Is it possible or my model is getting bad output ? I have used the model like below :

def Transfer():
    model = Sequential()
    model.add(ResNet50(include_top = False, pooling = 'avg', weights = 'imagenet'))
    model.add(Dropout(0.4))
    model.add(Dense(3, activation = 'softmax'))

    model.layers[0].trainable = False
    sgd = optimizers.SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True)
    model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
return model


texts: Binary cross entropy loss for one hot encoded 2 class problem

texts:
My aim is to predict whether a person is alive or dead. In the case there are two classes which can either be alive (1) or dead (0). The output could be only one class i.e 1 or 0 and not multi label result.
I have one-hot encoded value for the label
label = [[0, 1], [1, 0], [0,1]]

And the model also predicts two raw logits as output.
output = [[2.0589, -2.0658], [-0.2345, 1.3540], [2.0589, -2.0658]]

I have seen many examples where binary cross entropy loss is used for only 1 output as label and output of the class.
I am using PyTorch and I can find two implementation of binary cross entropy loss:

BCELoss
BCEWithLogitsLoss (This uses sigmoid + BCELoss)

I am aware about the activation function used but my issue is whether I can use binary cross entropy loss for values having one-hot encoded values or not?


texts: using feature selection to improve model performance

texts:
I have a highly sparse dataset that I am using to predict a continuous variable via a random forest regression. I have achieved an acceptable level of performance following cross-validation, and I am now thinking of potential ways I might further improve accuracy.

Given that my dataset is highly sparse, I was thinking that recursive feature elimination (using the cross validated version in sklearn) might be a good way to go. My understanding is that this will give me the 'optimal' number of features and thus may reduce issues related to over-fitting.

My question is, is it then appropriate to re-run the analysis with these optimized features, or am I in someway biasing the model? I have a test set that has not been used at all in training/validation, so I am assuming that as long as I don't leak info between training and testing, I should be good. But I'm unclear if these assumptions are correct.

Is this a suitable use of RFE, or should i be considering a different path?

For info, my training/validation dataset is 370 rows, with approx. 900 features.


texts: Does k fold cross validation become less useful when number of observations is very large?

texts:
As seen in the accepted answer for variance of k-fold cross validation ,
the simulation shows that k-fold CV has the same test error rate for different values of k when n=200.
Does this mean that k-fold validation is likely to be as good as having a holdout set validation? (assuming I have abundant data to make up for the high bias for holdout set validation approach)

Apart from high bias, the problem with holdout set validation approach as described in ISL book is that the test error rate is sensitive to the random splitting of data between train and validate. My intuition is that, With very high n (and well spread out data), the problem due to random splitting seems less likely to occur.


texts: Checking Correlation of Categorical variables in SPSS

texts:
I am building a predictive model for a classification problem using SPSS. Of the Independent variables, I have both Continuous and Categorical variables. SPSS gives only correlation between continuous variables.

Is there a way in SPSS to find the correlation

a)between two categorical variables
b)between categorical and continuous variables?


texts: Does adding of many FC layers during re-training increase the model size ? Are there any ways to optimize the size of model?

texts:
I am re-training a pretrained model VGG16.

In the last layers, im using two FC layers of size 2048 each, with dropout=0.5.

When i saved the model, The size of the model was found to be 2 GB (which is so huge). I have trained this model for 9 classes.(900 images)

Is there any way, where i could reduce the size of the model but the performance shouldn't be affected ?

The reason is i have to use this model in AWS cloud, where the systems are having 8GB ram. I am worried if their performance would be affected by such models.


texts: Constraining a Deep Neural Network based on a priori knowledge of a real world system

texts:
I am new to this field and to StackExchage, so I guess I'll start by saying hello!
I am building a deep neural network to model a physical system which takes a set of inputs based on real-world measurements and predicts a single continuous value, another real-world parameter. It's a pretty straightforward regression problem.
While I am a bit limited in the amount of data I can collect and train the model on, I do have a lot of a priori knowledge about the system's physical constraints that could provide some useful "guardrails" which I'd like to build into the architecture of the model, and some googling has helped me with many of those.
Some are a little more confounding:
For a certain value of one of the n elements of X, say x_n, the result in an undefined value for y regardless of the values of x_0 through x_n-1.
Similarly, for a certain value of another element, let's say x_0, any combination of the remaining elements of input vector X result in a y equal to zero.
Outside of a model architecture approach, I've thought that maybe I could use data augmentation and randomize part of X while keeping the relevant elements of X and y constant, but that would end up making my data set like 90% augmentation, and 10% real data. Is that a good approach?


texts: What's the best way to store and then call 3 Million+ records for training?

texts:
Good afternoon all,

I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline).

Essentially I've built a function that can convert raw data into vectors of magnitude ~800, and then I have around 3-5 Million of these vectors with a simple binary category: Pass/Fail.

In my ML projects, I found just storing these records in a CSV or Excel file was sufficient as the most data I've ever worked with was around 20,000 records, but now I don't believe either would suffice for my use case.

What would be the best way to store this data so that I can call an encoder at my next stage to run through the records?


texts: How neural networks handle large variations in the input?

texts:
I am reading the paper with the title "Classification of indoor actions through deep neural networks". And I came across this statement:

  With this mechanism very complex functions can be learned combining
  these modules: the resulting networks are often very sensitive to
  minute details and insensitive to large irrelevant variations.

I have read a lot about deep neural network. However, I might have missed the logic behind why deep neural networks are insensitive to large irrelevant features.


texts: How to get Optimum Lag with Data2 tends to catch up more on the upside than on the downside of data1

texts:
Hope you are doing well!!
I have two time series, Data1 and Data2 from 01/01/2017 to 01/01/2020,
I did a cross correlation with all the dates (from 01/01/2017 to 01/01/2020) to get the optimum lag, however Data2 tends to catch up more on the upside than on the downside, so the lag has to be different, And from 01/01/2017 to 01/01/2020 I'm sure that the lag found with the cross correlation function will not be representative for all data,
My question is How can resolve that ? I mean how can I built a trend indicator to then get the best lag ?
Thanks


texts: Which supervised learning algorithms are available for matching?

texts:
I'm working on a non-profit where we try to help potential university applicants by matching them with alumni that want to share their experience/wisdom and, at the moment, it is happening manually. So I'll have two tables, one with students and one with alumni (they may have some features in common, but not necessarily all of them)

$\begin{array}{|l|c|c|} \text{Name} & \text{Gender} & \text{Height} \\ \hline \text{Kathy} & F & 165 \\ \hline \text{Tommy} & M & 182 \\ \hline \text{Ruth} & F & 163 \\ \hline ... & ... & ... \\ \end{array}$ $\begin{array}{|l|c|c|} \text{Name} & \text{Gender} & \text{Weight} \\ \hline \text{Miss Lucy} & F & 65 \\ \hline \text{Miss Geraldine} & F & 70 \\ \hline \text{Miss Emily} & F & 60 \\ \hline ... & ... & ... \\ \end{array}$

Currently, we are manually matching the members of table 1 with those in table 2. We will also collect information after the match ("Was it a good match? Please rate it on a scale from 1 to 10"). So it will look something like this:
$$
\begin{array}{|l|l|c|}
\text{Person #1} & \text{Person #2} & \text{Match?} \\ \hline
\text{Ruth} & \text{Miss Lucy} & N \\ \hline
\text{Tommy} & \text{Miss Emily} & Y \\ \hline
\text{Kathy} & \text{Miss Geraldine} & N \\ \hline
\text{Ruth} & \text{Miss Emily} & N \\ \hline
... & ... & ... \\
\end{array}$$

I would like to use a learning algorithm for this process. I know a little bit of machine learning, but I am still very much a novice (so it's also an opportunity for me to learn more about it), but I can't wrap my head around how you would do this kind of supervised learning when you have two sets both of which have multiple features. What sort of matching algorithms are available to do this? (Also, I prefer to work in R)

(By the way, I would be grateful if you could just point me in the right direction and I'll try to read about it and solve it myself. Also, I know how deeply frustrating it is to see questions that have already been answered -- if this is case, please don't hesitate to let me know without answering the question. I have already tried to search for various strings on Google and StackExchange, but mostly find lecture slides on graph theory that don't seem to be what I'm looking for (although it may just be because it's a bit over my head). Many thanks!)


texts: PointCloud value prediction - regression

texts:
I have point clouds for 400 objects. I have also some other features of these objects. For example their weights.

How I can now train model to predict the weight based on the shape(point cloud)?

I have seen pointnet but is seems it is only for classification?


texts: How to extract features from the encoded layer of an autoencoder?

texts:
I have done some research on autoencoders, and I have come to understand that they can also be used for feature extraction (see this question on this site as an example). Most of the examples out there seem to focus on autoencoders applied to image data, but I would like to apply them to a more general data set.

Therefore, I have implemented an autoencoder using the keras framework in Python. For simplicity, and to test my program, I have tested it against the Iris Data Set, telling it to compress my original data from 4 features down to 2, to see how it would behave.

The encoder seems to be doing its job in compressing the data (the output of the encoder layer does indeed show only two columns). However, the values of these two columns do not appear in the original dataset, which makes me think that the autoencoder is doing something in the background, selecting/combining the features in order to get to the compressed representation.

Here is the complete working example:

from pandas import read_csv
from numpy.random import seed
from sklearn.model_selection import train_test_split
from keras.layers import Input, Dense
from keras.models import Model

# Get input data and separate features from labels
df = read_csv("iris.data")
Y = df.iloc[:,4]
X = df.iloc[:, : 4]

# Split data set in train and test data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, train_size=0.5, random_state=seed(1234))

# Input information
col_num = X.shape[1]
input_dim = Input(shape=(col_num,))

# Encoding information
encoding_dim = 2
encoded = Dense(encoding_dim, activation='relu')(input_dim)
# Decoding information
decoded = Dense(col_num, activation='sigmoid')(encoded)
# Autoencoder information (encoder + decoder)
autoencoder = Model(input=input_dim, output=decoded)

# Train the autoencoder
autoencoder.compile(optimizer='adadelta', loss='mean_squared_error')
autoencoder.fit(X_train, X_train, nb_epoch=50, batch_size=100, shuffle=True, validation_data=(X_test, X_test))

# Encoder information for feature extraction
encoder = Model(input=input_dim, output=encoded)
encoded_input = Input(shape=(encoding_dim,))
encoded_output = encoder.predict(X_test)

# Show the encoded values
print(encoded_output[:5])

This is the output from this example:

[[ 0.28065908  6.151131  ]
 [ 0.8104178   5.042427  ]
 [-0.          6.4602194 ]
 [ 3.0278277   2.7351477 ]
 [ 0.06134868  5.064625  ]]

Basically, my idea was to use the autoencoder to extract the most relevant features from the original data set. However, so far I have only managed to get the autoencoder to compress the data, without really understanding what the most important features are though.

My question is therefore this: is there any way to understand which features are being considered by the autoencoder to compress the data, and how exactly they are used to get to the 2-column compressed representation?


texts: CNN - Confused on the output shape of second convolutional layer

texts:
I'm attempting to write a forward pass of a CNN but I'm stuck on the second convolutional layer.

From what I understand, given an image of size 28x28, a first filter of size 10x3x3, a second filter 20x3x3, and max pooling after each filter, the input shape should go from 28x28 > 10x28x28 > 10x14x14 > 20x14x14 > 20x7x7 (assuming a stride of 1).

My initial intuition says the shape after the second filter should be 20x10x14x14. However after doing some digging, from my understanding, you treat the sliding window as a sliding 'block'.

Currently my program spits out 20x10x14x14 using scipy.signal.correlate function.

input_ = x_train[0]
filters = np.random.randn(20, 3, 3)
temp = []
for i in range(20):
    temp.append(scipy.signal.correlate(input_, filters[i], 'same'))
input_ = np.array(temp)
print(input_.shape)

filters = np.random.randn(10, 20, 3, 3)
temp = []
for i in range(10):
    temp.append(scipy.signal.correlate(input_, filters[i], 'same'))
input_ = np.array(temp)
print(input_.shape)

I would be grateful for any help.


texts: Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

texts:
I am referring http://www.nltk.org/book/ch06.html for generating a movie review classifier. It considers all words (Nouns, adjectives, verbs..) as part of feature set.
I am trying to build a classifier that considers only the verbs and assesses if the movie review is positive or negative.

Please explain if this approach is better, if yes how can it be improved else what other parts of speech tags need to be included to improve the feature set.
Please refer to the following code
1)Import nltk and corpus
2)Understand the categories
3)Create a list of "documents" where each document consists of words that are not in stopwords.words() and filtered from list of "punctuations" as well.
4)Shuffle the documents and generate a list called "all_words" that contains all the words appearing in movie_reviews minus stopwords and punctuations.
5)Create a Freq Dist of "all_words"
6)Create a list of "verbs" by looking at the pos_tag of each word appearing in "all_words"
7)Create a feature dictionary for each document in the "documents" list created in step 3, such that the dictionary will contain keys corresponding to words in "verb" list and a boolean to show if that verb appeared in the document under consideration. This is handled by documentFeature().
8)Create a naive bayes classifier instance and train and compute test accuracy.

from nltk.corpus import movie_reviews
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
import nltk
movie_reviews.categories()
#['pos','neg']
#the regextokenier is used to tokenize the words
tokenizer=RegexpTokenizer(r'\w+')
#creating documents based on filtration of stopwords for each review and running a tokenizer on each document
documents=[(tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words(fileid))-set(stopwords.words()))),category)
           for category in movie_reviews.categories()
           for fileid in movie_reviews.fileids()
          ]
import random
random.shuffle(documents)
#each document contains words that are not in stopwords and punctuations.
for i in documents[:5]:
    temp=nltk.FreqDist([j.lower() for j in i[0]])
    print(temp.most_common(5),i[1])
#output of 5 documents
#[('vampires', 1), ('clever', 1), ('interesting', 1), ('sunlight', 1), ('partners', 1)] neg
#[('family', 1), ('nino', 1), ('friends', 1), ('acting', 1), ('higher', 1)] pos
#[('inconsistent', 1), ('eye', 1), ('yes', 1), ('interesting', 1), ('praise', 1)] neg
#[('acting', 1), ('science', 1), ('bucks', 1), ('huge', 1), ('terrific', 1)] pos
#[('acting', 1), ('shielded', 1), ('somewhere', 1), ('think', 1), ('touched', 1)] neg

#generate a list called 'all_words' that contains all the set of words that have appeared so far
all_words=tokenizer.tokenize(' '.join(set(i for i in movie_reviews.words())-set(stopwords.words())))
freqdist=nltk.FreqDist(all_words)

#create a list of all verbs for each word appearing in 'all_words'
verb=[]
pos_=nltk.pos_tag(all_words)
#print([i[1] for i in pos_])
for i in pos_:
    if i[1] in ['VB','VBG','VBN','VBZ','VBD','VBP']:
        verb.append(i[0])

#document - feature set, build a dictionary of verbs for each document
def documentFeature(document):
    feature={}
    for i in verb:
        feature['contains({0})'.format(i)]=(i in document)
    return feature
#build a naive bayes classifier
featureSet=[(documentFeature(d),c) for d,c in documents]
trainSet,testSet=featureSet[100:], featureSet[:100]
classifier=nltk.NaiveBayesClassifier.train(trainSet)

print(nltk.classify.accuracy(classifier, testSet))
#0.03 a very poor accuracy on the testset

Currently I am getting 0.03 accuracy, please help me with improving the accuracy.


texts:
How effective is it to use BM25 to rank words, to be more specific i have a dictionary of words and i want to rank only words in a document that are also in my dictionary. I want to rank all words in my dictionary for each document and then add the BM25 value of each word for the specific document.

Lets say i have a document and dictionary like this:

myDictionary=['bad', 'dangerous','hide', 'following]

corpus=[
['human', 'intelligence', 'computer','bad', 'dangerous'],
 ['survey', 'user', 'human', 'system', 'time', 'hide', 'following],
]

Now iam going to run the below BM25 formula in a loop for all words in my dictionary and then sum
the results of each word to get a bm25 value for each document.


texts: My name is Reza, a Master Student at Golestan University of Technical and engineering, Iran and my M.Sc. thesis is about detect fake account on social networks. so i need data set. how can i collect data set from social networks like instagram????


texts: What is Probability Fusion on this figure?

texts:
I'm reading the paper Few-Shot Semantic Segmentation with Prototype Learning.
I have tried to reach the authors, but they don't answer my emails, so I have decided to ask my question here because I haven't found the answer by myself.
On page 5 of that article you will find the following figure:

I think I understand everything except "Probability Fusion" (and either Image-wise NN Classifier, but this is another story). I have double check, and more, the article and I haven't found any reference to what it is, so I think it is a known network.
Could you please tell me what is "Probability Fusion"?


texts: Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

texts:
“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep learning) models explainable such that one can understand why the model made a particular decision. I had a look at various algorithms which do this (prominent ones like LIME, SHAP, Grad-Cam, but I've also skimmed over many papers that present very “special” approaches).

Since I am working with image data, I am particularly interested in explaining image classification decisions. However, what I realized is that, even though there is a manifold of algorithms, the (final) representation of an explanation of an image classification decision for the end user is always the same: It is the original image with some marks showing what part of the image had the most influence (whatever this means based on the underlying mathematical explanation algorithm) on the decision. Concretely, the explanations are always the original image and some sort of heatmap or outlines on top, marking the "most influential" pixels.

What other representation of an explanation of an image classification could be used? Are there any explanation methods that allow a representation as a textual or numeric explanation (instead of just heatmaps or outlines on the original image)?


texts: When are subword ngrams trained in fasttext? (Enriching Word Vectors with Subword Information)

texts:
when is the training for subword ngrams done? is it done simultaneously as when the word representation are trained? or is it done at the end, after word representations are created?
fasttext implements this paper where word representations are enriched with subword information. here, the word representation for each word is the sum of the representations of its character ngrams. just as how skipgram model is trained, so is the character ngrams, where the ngrams are the context and the word is the word.
since there are two tasks here that have to be trained, how is the training done? in sequence? or is it done together? The paper mentions two scoring objectives, one for the word and one for the subword ngrams. I am unclear as to whether these two objectives have been combined into one or there are optimized separately.


texts: Which machine learning model should I learn for this problem?

texts:
I'm working in python. Would like to practice some machine learning, and I've always been curious about an analog to the problem below...

A collection of 3 letters are drawn randomly from the 26 letters of the alphabet. None, any, or all of the letters can be discarded and replaced with an equal number of not-yet-drawn letters. This discarding and replacing happens 3 times (or more generally n times). After all the replacements have taken place a point value is awarded to the resulting 3-letter-word. Some words have high point values, and others less so. I'm interested in the optimum replacement strategy. For example, say only the words "FOX" and "THE" reward any value. All other 3 letter combos are worthless. I want the machine to learn the correct replacement strategy while holding "FOW" with one replacement remaining. In this simple case, the strategy is to replace the W only, and attempt to draw the X. This strategy is superior to replacing all 3 letters in an attempt to make "THE," since drawing "THE" only gets points 1/('count of remaining letters' choose 3) times, whereas replacing the W gets points 1/('count of remaining letters'); the instance where an X is drawn.

Can anyone point me in the right direction?


texts: Low scale ML/statistical techniques for data poor settings

texts:
I have two separate problems. One is logistic regression and other is time series prediction. But both suffer from paucity of data problems
a) For logistic regression, I have tiny dataset with 10 observations which has variables such as age, Marital_Status, income, gender and car_purchase_status (outcome flag with Yes/No values). Now I have a new 11th customer with variables such as Age, gender, Marital_Status and income. Now I would like to know whether this 11th customer will buy a car or not. Should I spend resources to influence him to buy a car? Am I spending my resources on the right customer? For ex: Is there anyway that I can find out that 11th customer has 70% or 80% pc chance of buying a car. So spending some marketing efforts such as calls can help us convince him to buy a car (100%). So, how can I do this? Any advice please? Should I just give up straight away that this problem impossible to solve with such low data or is there any simple statistical techniques that can help me gain some insights about 11th customer?
b) For time series, I have only 10 observations each spaced at 20 days gap. For ex: I have their revenue generated for day1,day21,day41,day61,day81.....day201. Now with the given 10 observations, I would like to predict the revenue generated for day500, day321, day621 etc. So, is it possible to run time series forecasting with such short series? Can you guys guide me on this please? Here, also should I give up because of low data points or is there any methods that I can use to predict the future timestamp points based on short input time series?
Can you guys please help me with list of steps/topics that can help me do this?


texts:
I want to train a neural network and I want to check its total loss in every 100 epoch but I get the error RuntimeError: CUDA out of memory. Tried to allocate ... in line #1, sometimes I get this error but this time because I've tried big numbers of neuron in hidden layers I think this time this error may be real, even though I tried
torch.cuda.empty_cache()
import gc
gc.collect()

which are codes recommended to fix the error I just mentioned so my question is not how to solve the error as I said this time it may be correct (not really even trying to know if the 3GB GPU capacity is full or not), and I just want to know is there any other way to calculate all train data loss? even with using the loss of batches.
I think another way of asking this question is how to calculate total loss using losses of batches?
btw any other suggestions about my hidden layer sizes are also appreciable because I can't really even overfit my 400000 samples train dataset.
input_size = 83
skipcolumns= 3
hidden_size = 640
output_size = 12
num_epochs = 10000
batch_size = 20000
learning_rate = .001
n_total_steps=int(len(trainx)/batch_size)+1
def seperator(i,divide,p):
    return int((p[1]-p[0])/divide*i+p[0]),int((p[1]-p[0])/divide*(i+1)+p[0])
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(NeuralNet, self).__init__()
        self.input_size = input_size
        self.lin = nn.Linear(input_size, hidden_size)
        self.relu = torch.nn.ReLU()
        self.leakyrelu =nn.LeakyReLU(negative_slope=0.01, inplace=False)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, hidden_size)
        self.l4 = nn.Linear(hidden_size, hidden_size)
        self.l5 = nn.Linear(hidden_size, hidden_size)
        self.l6 = nn.Linear(hidden_size, hidden_size)
        self.lout = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # out = self.relu(out)
        # out = torch.sigmoid(out)
        out = self.lin(x)
        out = self.leakyrelu(out)
        out = self.l2(out)
        out = self.leakyrelu(out)
        out = self.l3(out)
        out = self.leakyrelu(out)
        # out = self.l4(out)
        # out = self.leakyrelu(out)
        # out = self.l5(out)
        # out = self.leakyrelu(out)
        # out = self.l6(out)
        # out = self.leakyrelu(out)
        out = self.lout(out)

        # no activation and no softmax at the end
        return out

# Loss and optimizer
criterion = nn.MSELoss()

model = NeuralNet(input_size, hidden_size, output_size).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
tti=time.time()
for epoch in range(num_epochs):
    for btch in range(n_total_steps):
        stind,endind=seperator(btch,n_total_steps,[0,len(trainx)])
        thisx = trainx[stind:endind].to(device)
        thisy = trainy[stind:endind].to(device)
        outputs = model(thisx)
        loss = criterion(outputs , thisy)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    if keyboard.is_pressed('q'):
        x=1/0
    if epoch%10==0:
        print(time.time()-tti)
        if epoch%100==0:
            thisx = trainx.to(device)
            thisy = trainy.to(device)
            outputs = model(thisx) #1
            loss = criterion(outputs , thisy)
            print(loss.item())
```


texts: I am trying to make a Classifier using Machine learning to detect malwares,am I headed in the right direction?

texts: I am trying to make a malicious software detector using machine learning.I have obtained a dataset from internet which containes md5 hash and other features of pe files.I fed this databse to weka to find out the most important features,which came out to be only one(static_prio).I tried running the K-NN algorithm on the data using only one feature(static_prio) and I got 99.82% accuracy but I am a little confused about my approach here.I am naive with machine learning so please help by putting me on the right track.Thank you.


texts: Error bars for soft-DTW (Dynamic Time Warping) Barycenters

texts: I'm currently exploring time-series data and describe the average of the data using soft-DTW barycenter as described here. My next concern is if it is possible to create a confidence interval / standard deviation around the barycenter. I don't think a regular standard deviation would make a lot of sense in the same way that computing means doesn't work for time series data. I'm looking for possible references for this subject and any lead would be helpful.


texts:
In the following Linear Regression discussion I didn't understand a few things:

So my questions are:

In the third slide:
What does this probability means $P\left(y_i|x_i\right)$ and accordingly what does it mean to maximize it ?
Does it mean to maximize both $P\left(y_i=1|x_i\right)$ and $P\left(y_i=0|x_i\right)$, and as higher this probability, the more stable and rightful results we get, and accordingly the more correct weights $w^*$ we get ?

In the fourth slide I don't see the math, could anyone detail it ? How did we get that result ?


texts:
I am trying to perform a multi-class classification where the network is trained to classify objects into 3 categories: cars, pedestrians and miscellaneous. I am using the KITTI Dataset for car instances and Freiburg University's Polyterrasse Dataset  for pedestrian instances. ModelNet40 is being used for miscellaneous.

I am using equal number of samples for all three classes as the overall data and they have been divided into 7:3 ratio. Upon training for 75epochs, when I evaluate, the network is giving me 100% accurate classification in both miscellaneous and cars and 99.5% accurate classification in pedestrians. I am not sure if these numbers are valid. I checked the softmax values for the predictions and the network seems very confident in each classification. Around 0.98 softmax values for the predictions. I also checked the tensorboard plot for accuracy and loss and the curves stagnant after hitting a peak. Does this mean the network is overtrained or overfitting? Have I made a mistake somewhere? To me, the numbers seem unrealistic. How else can I cross-validate my results?
Loss Tensorboard plot

Other important links: PointNet implementation Github Repo


texts: multi-channel ML models resilience to missing data

texts:
I am building different machine learning models (in the specific case some classifiers) that rely on data coming from different sources.
One problem that I am facing is the occasional lack of data for some of the sources at some time points.
Is there a robust way of implementing multi-source ML models that can still take decisions when some of the data are missing?
A first idea of mine was to just set missing values to zero and hope for the algorithm to learn that they do not mean anything. Probably not unreasonable in my case but not really a general approach guaranteed to work.
A second idea was to build different models on the different channels and then stack a classifier on them that feeds on their predictions. But I am unsure about how much cross-source information this setup could catch.
Is there some more reliable way to setup a ML pipeline that would be resilient to this kind of problem?


texts: Unhashable type 'list' while looping through dataframe in Python

texts:
I have the following dataframe comments. I have segregated a list of users based on certain conditions. I want to get the count of words based on those users which are named as gold_users. But I am getting an error in my code TypeError: unhashable type: 'list'. Please help me fix this.
DataFrame(comments)
       Id|                    Text                             |    UserId
        6|  [2006, course, allen, knutsons, 2001, course, ...  |    3
        8|  [also, theo, johnsonfreyd, note, mark, haimans...  |    1

Code
for index,rows in comments.iterrows():
  gold_comments = rows[comments.Text.loc[comments.UserId.isin(gold_users)]]
  Counter(gold_comments)

Expected Output
#Top 10 Words that appear the most in the comments made by gold users with their count.
 [['scholar',20],['school',18],['bus',15],['class',14],['teacher',14],['bell',13],['time',12],['books',11],['bag',9],'student',7]]


texts: What can I use to post process an NLP tree generated from the python library `spaCy`?

texts:
Using spaCy as the NLP engine for a chatbot, I call nlp("Where are the apples?").print_tree() and receive:

[{'word': 'are',
  'lemma': 'be',
  'NE': '',
  'POS_fine': 'VBP',
  'POS_coarse': 'VERB',
  'arc': 'ROOT',
  'modifiers': [{'word': 'Where',
    'lemma': 'where',
    'NE': '',
    'POS_fine': 'WRB',
    'POS_coarse': 'ADV',
    'arc': 'advmod',
    'modifiers': []},
   {'word': 'apples',
    'lemma': 'apple',
    'NE': '',
    'POS_fine': 'NNS',
    'POS_coarse': 'NOUN',
    'arc': 'nsubj',
    'modifiers': [{'word': 'the',
      'lemma': 'the',
      'NE': '',
      'POS_fine': 'DT',
      'POS_coarse': 'DET',
      'arc': 'det',
      'modifiers': []}]},
   {'word': '?',
    'lemma': '?',
    'NE': '',
    'POS_fine': '.',
    'POS_coarse': 'PUNCT',
    'arc': 'punct',
    'modifiers': []}]}]

I can easily enough parse out (arc, lemma) pairs for where (advmod, where) and (apple, nsubj), and call a function where(apple).

However, this is a really naive way of handling the parsed tree.  Any suggestions for how to handle processing this tree?  I don't think something like a multilevel SVM would work.  Maybe a NN of some kind?


texts: Is Overfitting a problem in Unsupervised learning?

texts:
I come to this question as I read the use of PCA to reduce overfitting is a bad practice. That is because PCA does not consider labels/output classes and so Regularization is always preferred.

That seems purely valid in Supervised Learning.

What about the case for Unsupervised Learning?
We don't have any labels whatsoever.
So 2 questions.

Is overfitting a problem in Unsupervised learning?
If yes, Can we use PCA to prevent overfitting?Is that a good practice?


texts: What information does Hidden State and Cell State carry? Do they carry the summary of the input sequence or what?


texts: Increase Accuracy on Keras Multiclass Classifier

texts:
I have a dataset with the shape (430, 17). My output is a single column consisting of possible options "Best", "Medium", "Worst". I split my data into X and y, and then into training and testing sets after using the StandardScaler to scale X.
I then using the LabelEncoder and get_dummies to prepare my output values.
>>>X_train[:1]

   array([[-0.62115527,  0.73646208,  1.27931666, -1.13977066, -0.03124594,
        -0.06595102, -0.35812162,  2.47947661,  0.01705523, -0.75215293,
        -0.43173137, -0.15181261]])

>>>y_train[:1]

   array([[0, 1, 0]], dtype=uint8)

I then prepared a keras Model based on an example from the iris dataset that also worked with a multiclass classifier using Keras.
model = Sequential()
model.add(Dense(20, input_dim = 17 , activation = 'tanh'))
model.add(Dense(64, activation = 'tanh'))
model.add(Dense(64, activation = 'tanh'))
model.add(Dense(64, activation = 'tanh'))
model.add(Dense(3, activation = 'softmax'))
model.compile(Adam(lr=0.04),'categorical_crossentropy', ['accuracy'])

model.fit(X_train, y_train, epochs = 100, verbose=1)

y_pred = model.predict(X_test)

scores = model.evaluate(X_test, y_test)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

I have been changing the inputs and general architecture of the model for the last few hours, but I cannot seem to be able to get the accuracy above 63.64%.
Sample Output:
Epoch 95/100
11/11 [==============================] - 0s 2ms/step - loss: 0.6250 - accuracy: 0.7037
Epoch 96/100
11/11 [==============================] - 0s 2ms/step - loss: 0.7214 - accuracy: 0.6980
Epoch 97/100
11/11 [==============================] - 0s 2ms/step - loss: 0.7595 - accuracy: 0.7123
Epoch 98/100
11/11 [==============================] - 0s 2ms/step - loss: 0.7197 - accuracy: 0.6838
Epoch 99/100
11/11 [==============================] - 0s 2ms/step - loss: 0.6658 - accuracy: 0.6895
Epoch 100/100
11/11 [==============================] - 0s 1ms/step - loss: 0.7103 - accuracy: 0.6838
3/3 [==============================] - 0s 997us/step - loss: 0.8703 - accuracy: 0.6364

My question is:
Can you advise on what I can do to increase the accuracy of the validation data?
Edit 1:
Changed the hidden layer nodes to 12, and changed to activate to relu. Here is a picture of the training and validation so far:

Edit 2:
Changed the focus of the posting from two questions to one. I also added the most recent model, and results:
model = Sequential()
model.add(Dense(20, input_dim = 17 , activation = 'tanh'))
model.add(Dense(24, activation = 'tanh'))
model.add(Dense(24, activation = 'tanh'))
model.add(Dropout(0.11))
model.add(Dense(24, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))
model.compile(Adam(lr=0.002),loss = 'categorical_crossentropy', metrics = ['accuracy', "mse", "mae"])
history = model.fit(X_train, y_train, epochs = 800, validation_split = 0.2, verbose=0)


texts: Can we do multivariate time series analysis using holt-winter ( Exponential smoothing) method?

texts: Just like we have a method like ARIMAX and SARIMAX where we can provide exog and endog variable for perfroming multivariate analysis. I was hoping is there a way, we can achieve same using ETS as well. Please let me know in case any has worked on this.


texts: Apache Spark alternatives for local compute

texts:
I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 dimensions are free form text).

From what I've started with on Spark, it is heavily geared toward distributed computing and other production related concerns but it has a nice ml library. Is it worth using Spark if I only plan to run my project on a local dev machine level or is there a more appropriate library out there for me in Scala? Something like scikit-learn, but in Scala.


texts: Trouble understanding k means distance calculation with high dimension

texts:
If I have a dataset with 10 features, 100 samples, and set I k = 3, three rows will become the 'centroid'. However, if I have 10 features how is the centroid calculated i.e. what exactly is it?
Examples I've seen online normally have two features, so you get a nice 2d plot and the centroid is the (x, y) coordinate. However, if I have ten features I find it hard to visualize this and also how the distance using euclidean is calculated between each row and the next.
How does this work?


texts:
I have a dataset made of pairs of graphs and a binary label (0 or 1 depending on if the graphs are similar). I am trying to find a model that, when given two graphs, will output if these two graphs are similar or not.

I am not very familiar with ML and don't really know how to get started with this problem. For now I have found following ideas:

There are several publications on how to use graphs as an input for a neural network (GCN, GGNN). These models usually take one graph as input and will output either another graph or a value. They can for example be used for graph classification.
Siamese networks can be used to solve similarity problems, for example returning 1 if two input images match.

From what I've understood so far, I should probably find some way to combine these two approaches: basically a siamese network taking two graphs as input. I can represent each graph as an adjacency matrix + a feature vector for each node. However I don't know where to go from this.

Do you have any ideas on what a good approach would be? And ideally, do you know any existing implementation that I could use directly or quickly adapt to my problem?


texts: How to use GridSearchCV with RidgeClassifier

texts:
I'm trying to use GridSearchCV with RidgeClassifier, but I'm getting this error:

My problem is regression type.

  IndexError: too many indices for array

I'm new to Machine Learning, please help me out.
This is the code I've been trying to implement:

from sklearn.grid_search import GridSearchCV
from sklearn.metrics import classification_report

tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],
                     'C': [1, 10, 100, 1000]},
                    {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]
scores = ['precision', 'recall']
alphas = np.array([1,0.1,0.01,0.001,0.0001,0])
model = RidgeClassifier(normalize=True, random_state=100, tol=0.1)
for score in scores:
    clf = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))
    clf.fit(X, Y)
    print("Best parameters set found on development set:")
    print(clf.best_params_)
    for params, mean_score, scores in clf.grid_scores_:
        print("%0.3f (+/-%0.03f) for %r"
              % (mean_score, scores.std() * 2, params))

This is the complete error log:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-59-c97c7e0fc6f3> in <module>()
     12 for score in scores:
     13     clf = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))
---> 14     clf.fit(X, Y)
     15     print("Best parameters set found on development set:")
     16     print(clf.best_params_)

/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.pyc in fit(self, X, y)
    836
    837         """
--> 838         return self._fit(X, y, ParameterGrid(self.param_grid))
    839
    840

/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.pyc in _fit(self, X, y, parameter_iterable)
    551                                  'of samples (%i) than data (X: %i samples)'
    552                                  % (len(y), n_samples))
--> 553         cv = check_cv(cv, X, y, classifier=is_classifier(estimator))
    554
    555         if self.verbose > 0:

/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in check_cv(cv, X, y, classifier)
   1833         if classifier:
   1834             if type_of_target(y) in ['binary', 'multiclass']:
-> 1835                 cv = StratifiedKFold(y, cv)
   1836             else:
   1837                 cv = KFold(_num_samples(y), cv)

/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in __init__(self, y, n_folds, shuffle, random_state)
    568         for test_fold_idx, per_label_splits in enumerate(zip(*per_label_cvs)):
    569             for label, (_, test_split) in zip(unique_labels, per_label_splits):
--> 570                 label_test_folds = test_folds[y == label]
    571                 # the test split can be too big because we used
    572                 # KFold(max(c, self.n_folds), self.n_folds) instead of

IndexError: too many indices for array


texts: XGBClassifier: make the output of predict_proba ascending regarding a specific feature

texts:
I'm trying to build a classifier using Xgboost on some high dimensional data, the problem I'm having is that I have the prior knowledge that the output probabilities should be ascending regarding a feature(say x), but I don't know how can I make the model understand this!

For example for a data point with features Feat I want to have:

predict_proba(Feat[x=1]) <= predict_proba(Feat[X=2])

where the rest of the features are the same.


texts: replicability / reproducibility in topic modeling (LDA)

texts:
If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) and what can be done to solve this issue or gain more stability across runs?
Thanks for the help.


texts: What is a good non cryptographic Hash for string feature translation?

texts:
What would be a good non cryptographic Hash function to use for converting string features to a numerical representation for feeding into machine learning algorithms?

To explain the scenario my feature set has both categorical data (e.g.: Country) and non categorical data (e.g.: IP Address, Email address). I have used MurMur3 Hash function so far, is there some better algorithm?


texts: What is the advantage of positional encoding over one hot encoding in a transformer model?

texts:
I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices.

In the paper (Section 3.5), they mentioned

  Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks.

My question is that if there is no recurrence, why not use One Hot Encoding. What is the advantage of using a sinusoidal positional encoding?


texts: Is there any different between feature selection and pca? If there is could anyone please kindly explain for me please?

texts: First of sorry for asking a possibly beginner question, but i don't understand pca seems to be the same as feature selection, but when i search online they seems to be talked differently. What people usually say is PCA for reducing dimensionality and feature selection for selecting feature. Didn't those two do the same thing? Reducing number of attributes/features/dimension? Please kindly help me understand. Thank you


texts: How to train neural network for text-to-speech task?

texts: I have 12000 audio files which lengths is about 2 seconds and 12000 text files with phonemes and time stamps for audio files. My task is to train NN to synthesize audio for train data set only. I already tried feedforward NN and it works good only for 20 audio files. Which architecture should i choose? What should i do with data(preprocessing e.t.c)?


texts: Deep Learning with Spectrograms for sound recognition

texts:
I was looking into the possibility to classify sound (for example sounds of animals) using spectrograms. The idea is to use a deep convolutional neural networks to recognize segments in the spectrogram and output one (or many) class labels.  This is not a new idea (see for example whale sound classification or music style recognition).

The problem that I'm facing is that I have sound files of different length and therefore spectrograms of different sizes. So far, every approach I have seen uses a fixed size sound sample but I can't do that because my sound file might be 10 seconds or 2 minutes long.

With, for example, a bird sound in the beginning and a frog sound at the end (output should be "Bird, Frog"). My current solution would be to add a temporal component to the neural network (creating more of a recurrent neural network) but I would like to keep it simple for now. Any ideas, links, tutorials, ...?


texts:
I have a database of text files and I would like to classify each section(i.e references, abstract, etc...) in the text.

For example:

  Abstract Indexes: 100-300

  References Indexes: 9000-10000

Is there a package that is doing something like that?

Is there a recommended approach for doing so?


texts: Logistic Regression in R models for 1 or 0?

texts:
According to this link, it says SAS models for 0,

  To model 1s rather than 0s, we use the descending option. We do this because by default, proc logistic models 0s rather than 1s

What happens in case of R's glm function? Does it model for 1 or 0? Is there a way to change it? Does it matter?


texts: How to recognise when to stop training based on Overfitting/Underfitting?

texts:
I am trying to train a LSTM network, over a total of 200 epochs, with hidden layer size of 100 and 1 dense layer after the LSTM layer. I have used a batch size of 10 for the same. Basically, I am confused as to why the loss curve which I get (with MAE as loss criteria and Adam Optimiser) is looking very different from what a good model generally gives. I believe that the likely reason may be that the training is occurring over more number of epochs than should be ideal, and it is underfitting/overfitting, but I am not sure that how to recognise the same.

The loss curve for the model is

I would like to be sure of whether the model is overfitting or undercutting, and if I need to reduce the training epochs (say from 200 to 20?).Being new to this, is there any specific point to identify when to stop the training process (such as based on this loss curve). Any help in this regard is appreciated.


texts: Choosing a distance metric and measuring similarity

texts:
I am trying to decide which particular algorithm would be most appropriate for my use-case.
I have dataset of about 1000 physical buildings in a city with feature space such as location, distance, year built and other characteristics etc. For each new data point, a building, I'd like to find 3-5 buildings that are most similar based on feature space comparison.
I define similarity as weighted comparison of features. I'd like to iterate over entire feature space (w/ filter like location) and choose 3-5 most similar buildings matching the new building data point.
Here's what my data looks like:

I'm wondering what similarity measure would make sense? I work in python, so prefer a pythonic/sci-kit learn way of doing this.