texts: how to access weights of individual Neurons in the output layers in MLPs?

texts: im working on a neural network using Keras. Its an mlp(multi-layer perceptron). With  8 Neurons in the output layer. Is there a way I can access weights and biases of individual neurons of the output layer for every iteration?


texts: ValueError: could not convert string to float: '���'

texts:
I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string of 26 characters.
array([['143347', '1325', '28.19148936', ..., '61', '0', '0'],
   ['50905', '0', '0', ..., '110', '0', '0'],
   ['143899', '1325', '28.80434783', ..., '61', '0', '0'],
   ...,
   ['85', '0', '0', ..., '1980', '0', '0'],
   ['233', '54', '27', ..., '-1', '0', '0'],
   ['���', '�', '�����', ..., '�', '��', '���']], dtype='<U26')

When I convert it to a float datatype, using
X_f = X.astype(float)

I get the error as shown above. how to solve this string formatting error for '���'?
I realize that some characters are not read properly in the dataframe, and the unicode replacement character is just a result of it.
My questions:-

How do I handle this misreading?
Should I ignore these characters? Or should I transform them to zero maybe?

Additional Information on how the data was read:-
importing relevant packages
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.functions import col

loading the dataset in a pyspark dataframe
def loading_data(dataset):
    dataset=sql_sc.read.format('csv').options(header='true', inferSchema='true').load(dataset)
    # #changing column header name
    dataset = dataset.select(*[col(s).alias('Label') if s == ' Label' else s for s in dataset.columns])
    #to change datatype
    dataset=dataset.drop('External IP')
    dataset = dataset.filter(dataset.Label.isNotNull())
    dataset=dataset.filter(dataset.Label!=' Label')#filter Label from label
    print(dataset.groupBy('Label').count().collect())
    return dataset

# invoking
ds_path = '../final.csv'
dataset=loading_data(ds_path)

check type of dataset.
type(dataset)

pyspark.sql.dataframe.DataFrame
convert to np array
import numpy as np
np_dfr = np.array(data_preprocessing(dataset).collect())

split features and labels
X = np_dfr[:,0:22]
Y = np_dfr[:,-1]

show X
>> X
array([['143347', '1325', '28.19148936', ..., '61', '0', '0'],
       ['50905', '0', '0', ..., '110', '0', '0'],
       ['143899', '1325', '28.80434783', ..., '61', '0', '0'],
       ...,
       ['85', '0', '0', ..., '1980', '0', '0'],
       ['233', '54', '27', ..., '-1', '0', '0'],
       ['���', '�', '�����', ..., '�', '��', '���']], dtype='<U26')


texts:
I am currently studying colorization of grayscale satellite imagery as part of my Master's internship. After looking for various machine learning techniques, I quickly decided to go for deep learning, as the colorization process can be completely automated and gives good results. I trained a wide variety of models and architectures (CNNs, Pix2Pix, DCGANs and WGANs with UNet or residual architectures, etc.) and settled with convolutional cGANs (condition: grayscal image ; output : colored image), but end up facing the same problem every time. Indeed, when training my network, the output is quite correct but always gives me grayish roofs.

I think it has something to do with the distribution of pixels values, as roofs are usually orange or black in my geographic area. Many articles state the fact that GANs are prone to mode collapse. In this case, there are two modes (orange and black) which probably results in a bad local equilibrium.

I tried many different techniques in order to improve training and/or avoid mode collapse :

L1 loss regularization for the generator
Label smoothing
Noise taken from a normal distribution and concatenated to every layer in the generator/discriminator
Dropout for some layers in the generator
Gradient penalty
Learning procedure inspired by DRAGANs and BEGANs

So far, gradient penalty has given me the best results, despite still having these grayish roofs. I know there are other ways of improving the training procedure (e.g. batch discrimination), but I do not have the pre-requisites in mathematics or computer science nor the time for implementing such techniques.

I was wondering if someone had some ideas for getting better results, or if someone also tried colorizing satellite imagery and ended up having such issues.

Thanks for reading this long message.

Have a nice day !


texts: Machine Learning to Get X, Y Coordinates From Picture of Plot

texts:
I am currently on a project where we have people place sticky notes on a X-Y axis to plot their beliefs on certain topics. These sticky notes can be of different colors to reflect different levels of their beliefs. What we are then left with is a picture that we take of the various plots to base our findings on.

I am hoping to avoid plotting each point individually in a data frame based on the picture. I am also relatively new to utilizing machine learning, specifically in relation to image recognition. I have done a couple of random forest projects, but nothing image processing related.

My question is this, using R, is it possible to use a machine learning package to read these pictures and create a data frame of X-Y coordinates of these sticky notes plots?

I am open to all suggestions. Thanks for any and all help and let me know if you need any clarification.

Edit: Here is an example of what one of the images might look like. Unfortunately, I could only quickly find an X-axis only graph, but we do X and Y-axis graphs as well.


texts: Hypothetically, if your company's sales had dropped significantly in 2020, what approach would you take to describe the cause? can you build a model to predict the decrease (between 2019 and 2020 for example) to visualize what the leading indicators are?


texts: Keras exception: ValueError: Error when checking input: expected conv2d_1_input to have shape (150, 150, 3) but got array with shape (256, 256, 3)


texts: prediction() returning mistakenly false positives

texts:
I do not know how to interpret the result of:

prediction(c(1,1,0,0),
           c(1,1,0,0))

prediction() functino comes from prediction {ROCR}  it has this site: http://rocr.bioinf.mpi-sb.mpg.de/

The above is a working example. As per the documentation the first parameter is 'predictions' and the second 'labels' (they would be the true values).

The output is this, which I do not fully understand, specially why there is a '2' in "fp". :

An object of class "prediction" Slot "predictions": [[1]] [1] 1 1 0 0

Slot "labels": [[1]] [1] 1 1 0 0 Levels: 0 < 1

Slot "cutoffs": [[1]] [1] Inf   1   0

Slot "fp": [[1]] [1] 0 0 2


texts: How to prepare Audio-text data for speech recognition

texts:
I have gathered some raw audio from all the conferences, meetings, lectures & casual conversation that I was part of. The machine transcription did not offer good results (from Azure, AWS etc.) I would transcribe it so to have both data+label (audio+text) for ML training.
My question is whether to have small (3-10 sec.) audio files (split it at silence) and then transcribe each small file? or large file with timestamps with subtitle.srt format?
What if I have a long duration audio file with text? I heard long files have more chances of errors & not accurate training. What if I add timestamp like subtitle file srt?
Do I need small audio files?
I tried Azure custom Speech to train and test but it threw errors, saying it wont process large audio files. (so small chunks is recommended.)
What other ML platforms (AWS, Watson, GCP) have it their data labelling criteria? Sorry I couldn't find other than MS Azure.
Ideally I shall have its own speech recognition system with clean slate (open to hear suggestions on model selection), but need to know what format & style the data should be created.
The way I see it, audio splitting (say, cutting a 30 mins audio into 200 parts) can be automated but then, how to separate transcripts into 200 lines? (need to check manually to linebreak.), So, not a good option to go for large datasets.
Thus it's important to decide the data format before working on it (for assigning the transcribers the proper instructions).
So the question again: with the clean slate (a) to have large audio files with timestamp transcript, or (b) to have small audio files with single line text?
& how?
Please guide. I did bit research but finally dared to post a question.


texts: Is there a way to slow-down a computation but not get Session crash error?

texts: I need more memory to run a code snippet but I am getting Session crashed error. I am ready to compromise on run time. Can something be done? P.S: I am using Colab 25 gb RAM


texts: Which Loss function is correct for binary mapping?

texts: I have built a 3 layer neural network to perform a binary mapping (2016 inputs, 288 outputs.) I am getting decent results with mean square error and stochastic gradient decent. My question is: Is there a more appropriate loss function for regression when the output is binary?


texts: Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow)

texts:
In many ptb/mini shakespeare  LSTM generator tutorials on web, people make input data, (where every batch is subsequent. for example, sequences in batch_2 are subsequent to sequences in batch_1, and batch_1 is fed right before batch_2) and feed those input data to dynamic_rnn. tutorial: https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html

During training, these tutorials manually feed back the hidden state of LSTM, like codes below.

batch_size = 32
hidden_state_in = cell.zero_state(batch_size, tf.float32)
output, hidden_state_out = tf.nn.dynamic_rnn(cell, inputs,initial_state=hidden_state_in)
...
#For loop used in training: ...
    output, hidden_state = sess.run([output, hidden_state_out],
                            feed_dict={hidden_state_in:hidden_state})

What makes me confused, is that when generating or testing ptb/shakespreare, these tutorials feed data, with batchsize==1 and varying seqlength (at training, batchsize was 32 or above).

def generate_characters(g, checkpoint, num_chars, prompt='A', pick_top_chars=None):
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        g['saver'].restore(sess, checkpoint)
        state = None
        current_char = vocab_to_idx[prompt]
        chars = [current_char]

        for i in range(num_chars):
             if state is not None:
                feed_dict={g['x']: [[current_char]], g['init_state']: state}
                #shape [BATCHSIZE, SEQLEN] with BATCHSIZE=1 and SEQLEN=1
            else:
                feed_dict={g['x']: [[current_char]]}
            preds, state = sess.run([g['preds'],g['final_state']], feed_dict)

When using dynamic_rnn with LSTM, the hidden state has shape of  [num_layers, 2, batch_size, state_size]. I just wonder how can we use data with different batch_size at training and testing. If hidden state is fed manually, what kind of trained feature is contained in LSTM graph? Is it cell_state, or the coefficients which are used in building states?

Also, when looking at generating text part of r2rt's tutorial above, first parts of generation seems weird. I guess this is because hidden_state (which is fed to dynamic_rnn's initial_state) at the beginning of generation is yet poorly configured, cause it's right after the first prompt. Am I right about this reason?


texts: Model for classifying time-series data with distinct features?

texts:
I've heard about time-series classification being done with TCN's and CNN's combined with LSTM's very often, citing that CNN's would provide insight both forward and in the past since you already have all the information for that time period. For my application, there is a distinct shape and I'd like to classify whether it exists or not. For example, I want to detect whether the data looks like this  or this
Of course, there would be noise involved and the feature would be much less obvious making the problem worthy of using machine learning. Is there some way I can exploit this knowledge of there being a single important feature (this hump) to use a different architecture or do anything differently?


texts: Good formula for turning star reviews into upvotes

texts:
I want to turn reviews of up to 5 stars and the number of reviews into upvotes. What's a good algorithm for doing this?

A venue with 10 reviews total with a 5-star average rating should obviously get more upvotes than a venue with 10 reviews total with a 3-star average rating. Also, a venue with 60 ratings and a 4-star rating should probably get more upvotes than the one with 10 reviews and a 5-star rating.

I need this rating to be based off of the total number of reviews and the average star rating, but I would also like the number to stay below a variable number (for example, say upvotes stay below 100, but I can also plug in 200 and it would stay below 200).


texts: Representation of state space, action space and reward system for RL porblem

texts:
I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without crashing into obstacles
I have the following environment:

where the environment is a matrix. In this the obstacles are represented by 0's and the free spaces are represented with 1s. The position of the agent is given by a label such as 0.8 in the matrix.

The initial internal representation of the environment of the agent will look something like this with the agent position in it .

Every time it explores the environment it keeps updating its own map:

The single state representation is just the matrix containing-

0 for obstacles
1 for unexplored regions
0.8 for position of the agent
0.5 for the places it has visited once
0.2 for the places it has visited more than once

I want the agent to not hit the obstacles and to go around them.

The agent should also not be stuck in one position and try to finish the exploration as quickly as possible.

This is what I plan to do:

In order to prevent the agent from getting stuck in a single place, I want to punish the agent if it visits a single place multiple times. I want to mark the place the agent has visited once as 0.5 and if it has visited it more than once that place will be labelled 0.2

The reason I am marking a place it has visited only once as 0.5 is because if there is a scenario where in the environment there is only one way to go into a region and one way to come out of that region, I don't want to punish this harshly.

Given this problem, I am thinking of using the following reward system-

+1 for every time it takes an action that leads to an unexplored region
-1 for when it takes an action that crashes into an obstacle
0 if it visits the place twice(i.e 0.5 scenario)
-0.75  is it visits a place more than twice

The action space is just-

up
down
left
right

Am i right in approaching the problem this way? Is Reinforcement Learning the solution for this problem?

Is my representation of the state , action, reward system correct?

I am thinking that DQN is not the right way to go because the definition of a terminal state is hard in this problem, what method should I use to solve this problem?


texts: static and dynamic data in clinical trials

texts:
Hi everybody and thanks in advance for those who will help me for this problem.
I have multiple data regarding patients involved in a clinical trial and my goal is to predict their death/non death. These data are composed by two datasets , the first is static (age, gender ecc) and the second is dynamic (analysis results collected during multiple sessions like %hematocrit, %sodium, blood pressure and so on). I know that if I had only static data I would have used any ML classification algorithm and of course in case of only dynamic data the "right" way was to build a LSTM but with both of them I'm a bit confused.
Which kind of model can I build in this case? I was thinking at two models where the output of the first one (static) becomes the input of the second one (dynamic). Is it a good idea? Thank you.


texts:
I am trying to cluster an image using k-means clustering algorithm. However, the features that I use (color, position) have different ranges, for that I notice the clustering is biased for the feature with larger range. hence I would like to ask if the normalization is important, and is it similar to normalization if I kept the ranges unequal but multiply with weights?
Should I normalize based on per image color range, or dataset color range, or color space range?
Thanks in advance.


texts: How to obtain original feature names after using one-hot encoding

texts:
This question is on an implementation aspect of sklearn DecisionTreeClassifier

How do I get the feature names ranked in descending order, from the feature_importances_ returned by the sklearn DecisionTreeClassifier?

The problem is that the input features to the classifier are not the original ones - they are numerical encoded one from pandas DataFrame get_dummies.

For example I take the mushroom dataset from the UCI repository.
Features in the dataset include - cap_shape, cap_surface, cap_color, odor etc.

pandas dataframe getdummies encodes these into multiple features based on values of the original features.
say cap_shape has values b,c,f,k .. after encoding new columns are cap_shape_b, cap_shape_c, cap_shape_f. Similar transformation happens for other features.

After training, the classifier tells me that the top two features are:
cap_shape_b, cap_shape_c, cap_shape_f, odor_a,odor_c,odor_f,odor_l.
From this result thrown by the classifier, I want my function to return the original features, that is, cap_shape and odor.


texts: output of k-mean cluster as collection of tweets

texts:
I want to cluster some 1000 tweets using k-mean algorithm. But I don't want the correct output, I just want clustering of tweets.

Suppose 1 cluster contain 300 tweets than all the contain of 300 tweets (contain consist main words of tweet after preprocessing of tweets) in form of text as my output. I don't want top term of cluster but all the terms in cluster. Here in R I implement k-mean.

tweets_dtm <-DocumentTermMatrix(clean_corp,control=dtm.control)
dtm <- weightTfIdf(tweets_dtm, normalize = TRUE)
m  <- as.matrix(dtm)
rownames(m)<-1:nrow(m)
norm_eucl<-function(m)
  m/apply(m,1,function(x) sum(x^2)^0.5)
m_form<-norm_eucl(m)
result<-kmeans(m_form,5,5)

for(i in 1:5)
{

  cat("cluster",i,":",findFreqTerms(dtm[result$cluster],2),"\n\n");

}

At the end I get error in below code.

cat("cluster",i,":",findFreqTerms(dtm[result$cluster],2),"\n\n");

Error: $ operator is invalid for atomic vectors.

Please suggest any other way to just cluster the tweets and the output like 1 cluster contain all text terms of clustered tweets.


texts: Determine how each feature contribute to XGBoost Classification

texts:
so for a summary of what I have done:

My dataset has 5 classes and 10 parameters. I used XGBclassifer from sklearn to investigate if I could use those 10 parameters to predict the class of each data point. After training and fitting the XGBclassifier, I checked feature_importances_ and found out that 2/10 parameters played a key role in the classification.

So my question is:

Can I find out exactly how those 2 parameters contribute to the classification of each specific class? For example, can I find the cut-off values for parameter 1 and parameter 2 that will result in the prediction of class 1?

I am thinking of performing unsupervised clustering using those 2 parameters and k value = 5. Afterwards, I can just eyeball the approximate cutoff values. However, I worry that the 5 clusters will not correspond closely to the 5 groups.

Thanks a lot in advanced


texts: How to implement a Fourier Convolution layer in keras?

texts:
I'm currently investigating the paper FCNN: Fourier Convolutional Neural
Networks. The main contribution of the paper is that CNN training is entirely shifted to the Fourier domain without loss of effectiveness. The proposed architecture looks as follows:

The authors state that the implementation was done in keras, however, it is not publicly available. I know I can define a Fourier transformation in the following way:

model.add(layers.Lambda(lambda v: tf.real(tf.spectral.rfft(v))))

But this is not a Fourier Convolution, right? How should I go from here?


texts: What is the difference between data analysis and machine learning?

texts:
It seems to me that machine learning (especially deep learning) can work with thousands (even millions) of different inputs. After training an ML model on the inputs, the computer has hopefully "learned" something about the connections between these millions of inputs, and can hopefully make predictions regarding new unseen inputs.

What is the difference between machine learning and data analysis?


texts: Explanations about ADAM Optimizer algorithm

texts:
I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one problem, and i can't figure it out. Don't judge me please, but i would like to know ?

Are we using ADAM optimizer alone or are we obliged to combine it with the SGD ? I didn't understand if it works alone or if it's here to optimize NOT the neural network but the SGD of the neural network?

Thank you for your help!


texts: Accuracy doesn't increase in Binary Classification with 3D coordinates as data

texts:
I have 4000 catalogues of galaxies, in each there are 34700 objects, for each of it I have x,y,z coordinates. I want to do a binary classification creating a model which should be able to determine which catalogues have a particular value of a Cosmological parameter and which catalogues have another value of the same parameter.
The two types of catalogues are labelled differently (0 and 1).

The full program is available here:https://colab.research.google.com/drive/1n7jDwu8ekqDl3OmYY2R49J55CUMraY9K

The accuracy doesn't improve during the training: here is the main part of the code:

python
# So far we have a numpy array (train_data) with shape (4000, 34700, 3):
# 4000 catalogues, 34700 objects (galaxies) in every catalogue, 3 coordinates for every object
# Now I'm going to turn this array in a (4000,104100) array by concatenating the three coordinates

train_set=np.zeros((4000,104100))
for i in range(len(train_data)):
    train_set[i]=np.concatenate((train_data[i][:,0],train_data[i][:,1],
                                 train_data[i][:,2]))
    if (i % 250==0):
        print('Flattening '+str(i)+'th mock')

# Build the model

model=keras.Sequential()
model.add(keras.layers.Dense(64, activation=tf.nn.relu))
model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))

model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['acc'])

#Create a validation set

x_val0=train_set[:100]
x_val1=train_set[1000:1100]
x_val2=train_set[2000:2150]
x_val3=train_set[3000:3150]
x_val=np.concatenate((x_val0,x_val1,x_val2,x_val3))

partial_x_train0=train_set[100:999]
partial_x_train1=train_set[1100:1999]
partial_x_train2=train_set[2150:2999]
partial_x_train3=train_set[3150:3999]
partial_x_train=np.concatenate((partial_x_train0,partial_x_train1,
                                partial_x_train2,partial_x_train3))

y_val0=train_labels[:100]
y_val1=train_labels[1000:1100]
y_val2=train_labels[2000:2150]
y_val3=train_labels[3000:3150]
y_val=np.concatenate((y_val0,y_val1,y_val2,y_val3))

partial_y_train0=train_labels[100:999]
partial_y_train1=train_labels[1100:1999]
partial_y_train2=train_labels[2150:2999]
partial_y_train3=train_labels[3150:3999]

partial_y_train=np.concatenate((partial_y_train0,partial_y_train1,
                                partial_y_train2,partial_y_train3))

#Train the model

history = model.fit(partial_x_train, partial_y_train, epochs=20,
               batch_size=200,validation_data=(x_val, y_val),verbose=1)

I have tried with different combinations of Dense keras layers, but here's the result:

Train on 3496 samples, validate on 500 samples
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/20
3496/3496 [==============================] - 8s 2ms/sample - loss: 7.9712 - acc: 0.5000 - val_loss: 7.9712 - val_acc: 0.5000

Epoch 2/20
3496/3496 [==============================] - 7s 2ms/sample - loss: 7.9712 - acc: 0.5000 - val_loss: 7.9712 - val_acc: 0.5000

Epoch 3/20
3496/3496 [==============================] - 6s 2ms/sample - loss: 7.9712 - acc: 0.5000 - val_loss: 7.9712 - val_acc: 0.5000

Epoch 4/20
3496/3496 [==============================] - 6s 2ms/sample - loss: 7.9712 - acc: 0.5000 - val_loss: 7.9712 - val_acc: 0.5000

Epoch 5/20
3496/3496 [==============================] - 6s 2ms/sample - loss: 7.9712 - acc: 0.5000 - val_loss: 7.9712 - val_acc: 0.5000

And so on until the end.


texts: nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"

texts:
While trying to remove stopwords using the nltk package, the following error occurred:

from tqdm import tqdm
import nltk
from nltk.corpus import stopwords

preprocessed_reviews = []
for sentance in tqdm(final["Text"].values):
    sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)

TypeError
  Traceback (most recent call last)
<ipython-input-136-ac5c19fafd9c> in <module>()
---> 7     sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
     8     preprocessed_reviews.append(sentance.strip())
     9

TypeError: argument of type 'LazyCorpusLoader' is not iterable


texts: Why I need to generate train instances and load negative samples?

texts:
If you look at this GitHub link ( here is the paper link for the implementation ) you can see that the get_train_instances method generates trainingns instances. In addition, negative test data is loaded.
My question now is why do I create such training instances and why do I have to load negative data?
Are there any other scientific publications that describe this in more detail? I would be very happy if you could help me with a source.


texts: Which is the best type of curve for these peaks on the waveform?

texts:
I have a waveform which looks like this. The peaks are indicated by the points. I want a type of curve fit which best connects these peaks as smoothly as possible.

I have tried polynomial fit but that does not yield a smooth curve. It looks something like this.

Here the green line is polynomial fit. As you can see it isn't as uniform as I want it to be. Also, poly fit fails to reach the max peak amplitude for some waveforms. So overall it is not what I am looking for. Is there a better fit that can account for these points better?


texts: Use pretrained word vectors over custom trained word2vecs

texts:
Currently i'm working on a sentiment analysis research project using LSTM networks.

As the input I convert sentences into set of vectors using word2vec.

And there are some well  pretrained word vectors like Google word2vec.

My problem is, is there are any advantages of using custom trained word2vecs(train using a dataset which related to our domain, such as user reviews of electronic items)  over pretrained ones.

Whats the best option

use a pretrained word2vec
Train our own word2vec using a dataset related to the domain

Can any one help me on this
Thanks


texts: Using accuracy metric during training for unbalanced multiclass classification

texts:
I am training a convolutional neural network and the sensitivity and precision of the minority class is what is most important to me. I am using 10-Fold cross validation, and the test fold is evaluated on sensitivity and precision by the "best" model from the 40-epoch training loop. My problem is, I am currently determining the "best" model by its raw accuracy on the validation set, the model weights in the epoch with the highest validation accuracy is used for the test fold. I do this to avoid selecting over fit models.
I am worried that because I am choosing the model based on validation accuracy for the test fold, I am not actually choosing the best model for the sensitivity and precision of the minority class which is my focus, and thus the cross validation result is not the best it can be.
Can someone enlighten me on this topic? Am I just worrying for no reason or should I maybe try evaluating the models based on the precision and sensitivity of the minority class?


texts: Cleaning NaNs with averages pre or post split?

texts:
I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode.
Technically, the validation/ test data has never been seen before - so how could I include it in the average? That would bias the values.
Do I "fit" the average to my training data only, just like scaling? Or do I take the average using the entire dataset?


texts: Remove noise by clustering on which step of pre-processing is better?

texts:
I am working on a classification task. The dataset is a UCI data set about machine learning with 200 observations and 2 classes.

Part of my model includes the following preprocessing steps:

remove missing values
normalize between 0 and 1
remove outlier
smoothing
remove trend from data
SMOTE

I would like to use a clustering method to remove noisy data points. The question is, at which step should this happen?


texts: 2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

texts:
What I want to do

Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also want this to work cross-gender), with the face/hair region of another pre-defined Person B. Then output the resulting image-sequence of Person B seemingly mimicking Person A's face pose and expressions.

The project should preferably be available open source and on github for everyone to use and improve. It would also be nice if I could do it in Python 3.6.x.

What I have

A very large dataset of images with Person B's face from all different angles and with all kinds of different facial expressions, aligned and centered (suitable for generative adversarial network training)

Ideas on how to do this

1. Database Approach

Process

Extract a multitude of landmarks of every image in the dataset of Person B and save them in a table with the reference to the image.
Extract a multitude of landmarks for each image in the sequence of Person A and per-image select the closest matches in the table of Person B, then use these matching images as replacements.

Concerns

I can see this approach resulting in a very long and resource intensive calculation with frames that just can not find a good match and may seem off or just plain fake.

2. ML Approach

Process

Use a BEGAN or WGAN process that will be trained similar to the famous male->female converters and then interpolate

Concerns

This technique will result in a very low resolution output. All in all from the results I have seen in different github projects, it's only photorealistic up to 125x125 pixels

Why I'm writing this question

To ask all of you what direction you would recommend me to take =)


texts: approach to classify text with natural language processing methods

texts:
I have a problem with regards to text classification/categorization. The task is bugging me for days already and as I am pretty new to AI and the field of natural language processing (NLP) I am just overwhelmed by the content online and available tools/libraries (e.g. NLTK, Keras, spaCy, etc.). It would be awesome if you could give me some guidance or clues on how you would approach the problem.

Issue: basicially I try to set up a tool for classifying text. I already have an extensive labeled dataset to work with. The input will always be a list of some sort (think of an Excel file with 500 rows). Each row contains a single word or a combination of words, i.e. no sentences.

A simplified example of my labeled dataset - input on the left, classification on the right:

"dog" -> "animal"
"dog owner" -> "person"
"dog owner house" -> "building"
"owner" -> "person"
"dog food" -> "food"
"food court" -> "building"

My existing labeled dataset has around 2,000 of these classifications with in total 50 unique categories. How can I set up an algorithm that scans the input for example for the word "dog" - if it is only "dog" then it is the category "animal", if it is "dog" and "owner" it is the category "person", if it is "dog", "owner" and "house" it is the category "building" and so on.

If I set up a ton of if-else-statements as a decision tree it is just cumbersome and intransparent. Is there a way with NLP to solve such an issue?

Thank you very much in advance! Looking very much forward to your ideas and please let me know if I have to be more specific in any way.

Best regards, pythoneer


texts: How to handle missing data data in dependent variable?

texts:
I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k records in the dependent variable that have missing values.
- How do I solve this problem? Should I be excluding these records?
- Is imputing the missing values a good method of solving this problem? Wouldn't that give me inaccurate values as the dependent variable is dependent on a number of variables? I'm not sure.

Can anyone please help?


texts: Orange canvas will not properly parse/open a file with a long header (column) line

texts:
I have a tab delimited data set with 23 columns.  The header for this file names the 23 different features in the dataset.  The column names are...

  file  yval    size    err a-value b-value c-value d-value e-value f-value intensity   sat a-id    b-id    c-id    d-id    e-id    filter  training    drift   status  classification  second

This file does not open/parse properly in orange canvas.  The File widget shows this...

However if I remove any three characters from the columns row it works.  For example if I change second to sec like this...

  file  yval    size    err a-value b-value c-value d-value e-value f-value intensity   sat a-id    b-id    c-id    d-id    e-id    filter  training    drift   status  classification  sec

Or change f-value to f-va like this...

  file  yval    size    err a-value b-value c-value d-value e-value f-va    intensity   sat a-id    b-id    c-id    d-id    e-id    filter  training    drift   status  classification  second

Orange canvas properly parses the file and I get this...

This has to be a bug.  Why would orange canvas have a character limit on the header of the file?  It makes no sense.  Is there a fix for this?

EDIT:

Should have mentioned I'm on Ubuntu 16.04 using Orange version 3.5.0.dev0+feea5ef


texts: Probability of similarity of two clusters

texts:
For example, there are 4 objects. Each object has some elements. There are 3 clusters. Each element of each object belongs to some cluster. Clustering was done before, and we know only how many elements of each object belong to some cluster.
How we can calculate the porbability of similarity of two objects?

         Cluster1  Cluster2  Cluster3
Object1    500       300        200
Object2    200       200        100
Object3    250       2500       1250
Object4    190       210        300


texts:
I'm running a classifier (logistic regression). The information on my dataset are the following:
dataset size= 279 observations

(80/20 rule)
train size= 233
test size = 56

# of events in train = 31
# of events in test = 8

I think my classifier and results may be affected due to this not equal proportion. Is there any way to avoid bias issues and improve accuracy? What do you personally think of such data?


texts: Can i pursue a career in data science with a business analytics degree?

texts:
I had a question regarding about a career as a data scientist.  Im pursuing a degree in business analytics with a minor computer science.  Could i still become a data scientist with a bachelors degree in business analytics. Or would i have to pursue a masters degree in data science in order to become a data scientist.

(I have a good programming background in data structures and algorithm in java, plus web development, python)


texts:
I am trying to parse addresses from various documents using spaCy using NER but the results are not so accurate.
I know this is bit generic question but it would be a great help if I could get reference of any past work or good articles or techniques to apply to this.


texts: Dataframe looks the same but the structure is different when loop

texts:
I am generating a dataframe from a JSON file, this JSON file can come from 2 different sources, so the internal structure is slightly different, so what I am doing is first detecting the source and from there I do a set of operations that gives me a Dataframe

Everything is good until here (I thought), as when I print it in jupyter it shows me the way I wanted they look the same (structure), the problem goes when I loop through them,

I get completely different results (this df have each same number of columns, 7 columns)

When I loop:

In 1 I have only 2 columns in the other one I get all the columns.

I am looping:

for i, (index, row) in enumerate(df_trans.iterrows()):
    print(row)

Is there a way to see how is structure, I am quite confused of why the print of the df loops the same but when looping is not

EDIT

I notice that when I print the dataframe after a grouping I get the followin

df_summary_trans_cs.groupby(['Date'])['sale', 'refund','Balance'].agg('sum')

I get all the columns

but when I add the column

df_summary_trans_cs.groupby(['Date'])['sale', 'refund','Balance', 'Trans'].agg('sum')

I only get that column, the other 3 dissapears


texts: Error rate of AdaBoost weak learner always bigger than 0.5?

texts:
As far as i understand, weak learners of AdaBoost should never yield a error rate > 0.5

After training one, i only receive error rates above 0.5. How is that even possible? The AdaBoost Tree still gives quite good results, but all learners weights should be zero, so it should fail. Also the trees get worse from iteration to iteration

is it possible that my threshhold for the error rate instead is 0.9 (accuracy 0.1), as i got 10 classes and literature mostly focusses on binary cases?

from sklearn.ensemble import AdaBoostClassifier

adaboost_tree =  AdaBoostClassifier(
        DecisionTreeClassifier(max_depth=max_d),
        n_estimators=estimators,
        learning_rate=1, algorithm='SAMME')
adaboost_tree.fit(data_train, labels_train)


texts: Understanding Scaling With Multiple Datatypes in a Neural Network

texts:
I've looked around and seen a huge amount of discussion on scaling inputs and targets for neural networks, but can't seem to find universal agreement on a few issues. Suppose I had a dataset, with input features X and targets Y.

Suppose for each data-point, the features are of the format:

X_1: A categorical variable, 'red', 'blue' or 'green'

X_2: A continuous variable, bi-modal. The range between 0 and 1440.

X_3: A binary variable, specifying if the data-point has a particular component or not.

X_4: A continuous variable, with a heavy-tailed distribution across the dataset.

Also, suppose the target for each data-point is of the format:

Y: A continuous variable, with a heavy-tailed distribution across the dataset.

My question is, is there an agreed-upon 'best way' to scale each of these input types if one were trying to model the data using a neural network? This is a toy example, but I often find myself with various different properties of data and just make ad-hoc decisions without a sure idea of the optimal way.

My current understanding is:

X_1: should be one-hot encoded, giving me 3 new features, each of which are valued at 0 or 1.

X_2: typically, we standardize or use min-max scaling, but if continuous data is bi-modal, is this still appropriate?

X_3: as a binary variable, I see often people suggest leaving it as it is, but others suggest using -1, 1 instead of 0, 1 encoding.

X_4: again, I could standardize it, I could use min-max scaling. I could also log the data beforehand, then apply another form of scaling, but it is not clear which is best.

Y_4: As with X_4, multiple options exist.

Any advice on what accepted best practices to do when scaling a dataset with varying properties across features and targets would be appreciated. Additionally, should one-hot encoded variables be normalized after encoding? Similarly with binary variables?


texts:
I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same test but, due to minor changes in the titles, titles are shown as unique.

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]

labels = ['title', 'description']
import pandas as pd
labtest = pd.DataFrame.from_records(values, columns = labels) # Create data frame

This is how the data frame looks like. [The actual dataset has many such titles, this is only for the purpose of this question]

Title                       Description
Complete blood test         AB
COMPLETE Blood test\        AB
Blood glucose               AB
Complete blood picture      AB

And this is what I would the like the data frame to look like:

Title                       Description
Blood test                   AB
Blood test                   AB
Blood test                   AB
Blood test                   AB

I would like to search for the word "blood" in each title and if its true, then change the whole title with "Blood test". Is there a way to do this?


texts: Using Wasserstein loss function for image-to-image-regression

texts:
The context
I have a 3D array (representing a grayscale 3D image) and want to turn this into another 3D array of the same size. In this output array the value of each pixel is a number that measures how likely the corresponding point in the input array is to be a sort of edge. So it is kind of like an edge detector (such as the Sobel filter), only it is very specific to this situation. Several formulas have been designed for this specific situation to accomplish this, but I would like to do it with a neural net. In order to train this net, I obtained some examples of 3D images and their corresponding 3D outputs that have been obtained through such complicated filter methods.
My approach
I want to use transfer learning (since my dataset is not very big) and most pre-trained models are for 2D image classification. Therefore I plan to split the 3D images into slices of 2D images and basically solve the same problem in 2D. I know this will throw away some information, but it might still be good enough. For the input image I could also take 3 adjacent slices and put them into 3 RGB channels, so that I throw away less information. As a model I would take one of those pre-trained 2D image classification models, only keep some of the convolutional layers in the beginning of the network and then add an extra layer that outputs an image of the correct size. I'm using Keras for this.
My problems
I currently have two practical problems with this approach:

Ideally I would use something like a Wasserstein loss function, since it's not a problem if an edge lies a few pixels too much to the right, for example. Is there any way in Keras to use this loss function for training such an "ordinary" neural network? When I search for it, I only find examples of GANs, which is not what I want.
I would like to save these 2D images (both the input images and the target images) in a folder and then load the images in this folder into python. However, I don't know how to do this (efficiently) while matching each image with the correct target image. In example projects that I can find (on image classification) this usually isn't a problem since the image label is inferred from the name of the image. Do you have any idea how I can do this? (I'm new to machine learning, so sorry if this is a stupid question.)


texts: How does BERT deal with catastrophic forgetting?

texts: In the ULMFit paper authors propose a strategy of gradual unfreezing in order to deal with catastrophic forgetting. That is, when the model starts be fine-tuned according to a downstream task, there is the danger of forgetting information on lower layers. Although Google's BERT is also a pre-trained language model, which makes use of fine-tuning for downstream tasks, authors don't mention this phenomenon. Why is that the case? Is BERT immune to it? Or does it deal with this in another way?


texts: YOLO: What is better? multiple networks per label or multiple labels in single network?

texts:
I would like to use a YOLOv3 network,load in the preexisting trained weights, then retrain the ending layer to recognize say, 20 labels.

Would adding labels to a single network in training reduce its accurary?

Would it be better to train multiple models on 5,10 and 5 labels respectively.

e.g.

Model A : recognizes 5 labels (ferrari,honda,ford,toyota,tesla)
Model B : regonzies 10 labels (store front / stopsign, speed sign, super charger station , etccc. )
Model C : recongizes 5 labels (pedestrian, dog, marathon runner, bicycler, scooter rider)

OR

Model D: Recognizes 20 labels (all labels above put together)

Would Model D be as accurate as Model A,B,C put together?


texts:
I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable.  I can of course detach the tensor, but then I lose the flow of gradients to earlier weights.  I guess the question is quite simple, how do you continue the flow of gradients when necessary.  For example, using the following code ...
x = self.linear1(x)
min, max = int(x.min()), int(x.max())
bins = torch.linspace(min, max+1, 16)
x = torch.bucketize(x.detach(), bins) # forced to detach here
x = self.linear2(x)

I know it's possible, as it was done with VQVAE and the gradients still flow fine for their quantization purposes.  But I checked several versions of VQVAE code and it doesn't seem logical how the gradients pass through the argmin method.  It just seems to work regardless, which confuses me.  I'm quite perplexed by this.  I would be grateful for any help on this one.  Thanks in advance.


texts: Why do we move in the negative direction of the gradient in Gradient Descent?

texts:
It is said that backpropagation, with Gradient Descent, seeks to minimize a cost function using the formula:
$$ W_{new} = W_{old} - learningRate \cdot \frac{\partial E}{\partial W} $$
My question is, if the derivate indicates in which direction the function (the graph of the error with respect to the weights) is decreasing, then why subtract from an already negative gradient?
Why not allow the current direction of the gradient (negative lets say) to be the driving factor for updating the weights:
$$ W_{new} = W_{old} + learningRate \cdot (-gradient) $$


texts: How are weights represented in a convolution neural network?

texts:
I have been trying to develop a convolution neural network following some guides online. However, most guides I have encountered gloss over an important detail, which is how to programmatically represent the weights in a CNN.

As far as I understand, in a "regular" neural network, the weight of a connection is a numerical value, which is adjusted in order to reduce the error; then back-propagation is used to further update the weights, reducing thus the error, etc.

However, in a CNN, the input is an array of numbers (the image), and a subset of those (the filter) to calculate the mean error, by multiplying the filter pixels by the original pixels.

So, is there a weight neuron for each filter (kernel or feature map) of the image? Or is a single weight neuron represented by the sum of all the mean error's calculated from convolving the filter over the receptive field, such that you have one value, in the end, that is the total error for the entire image?


texts: How to appropriately set weights for weighted KNN

texts:
I'm currently trying to predict a continuous variable using KNN. Instead of treating each neighbor equally I would like to use the weights to create a weighted average. The weights by themselves are not ideal, as the closer a neighbor the more I would like that neighbor to influence the final results.

This lead me to consider the inverse of each of the distances, but this doesn't handle the case where an instance is the exact same -> with a distance of 0.

Any recommendations on how to properly set the weights of each neighbor relative to their distance? Similar to how the inverse would handle this, but one that allows for 0 values.


texts: How do we get the coefficients and intercept in Logistic Regression?

texts:
I'm using Codecademy to learn about logistic regression and there are some holes in my understanding of this topic.
import numpy as np

hours_studied = np.array([[ 0],[ 1],[ 2],[ 3],[ 4],[ 5],[ 6],[ 7],[ 8],[ 9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19]])
calculated_coefficients = np.array([[0.20678491]])
intercept = np.array([-1.76125712])

def log_odds(features, coefficients,intercept):
  return np.dot(features,coefficients) + intercept
# z = b0 + b1x1 + b2x2 + ....
# log odds measures how likely it is that the data sample belongs to the positive class

def sigmoid(z):
    denominator = 1 + np.exp(-z)
    return 1/denominator
## In order to map predicted values to probabilities, we use the sigmoid function.
# The function maps any real value into another value between 0 and 1.
# In machine learning, we use sigmoid to map predictions to probabilities

# Create predict_class() function here
# features = matrix
# coefficients = vector
def predict_class(features, coefficients, intercept, threshold):
  calculated_log_odds = log_odds(hours_studied, calculated_coefficients, intercept)
  probabilities = sigmoid(calculated_log_odds)
  return np.where(probabilities >= threshold, 1, 0)
# If a value in array_to_check is above threshold, the output is 1. If a value in array_to_check is below threshold, the output is 0.

# Make final classifications on Codecademy University data here
final_results = predict_class(hours_studied, calculated_coefficients, intercept, .5)
print(final_results)

hours_studied, calculated_coefficients and intercept were given by codecademy.
I'm not sure how to get these inputs myself to plug into the model and codecademy didn't explain either. Hours_studied I understand.
Maybe could someone help explain what are the possible steps before I define log_odds, sigmoid, etc to get coefficients and intercept?


texts:
I am trying to read the following list of books on statistical learning. I have a BSCS and about 4 yrs of experience working in image processing and parallel programming. I won’t be an expert in the field by any means, however my aim is:

not to be a script kiddie, using tools and algorithms without understanding the hows and whys.
be able to read and digest the latest research in statistical learning, specially w.r.t computer vision.

Prerequisites I have studied in preparation:

Matrix Algebra by James E. Gentle
Statistical and Mathematical Methods lectures by Carlos Fernandez-Granda

Books to read:

An Introduction to Statistical Learning with Applications in R by Robert Tibshirani et al.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Robert Tibshirani et al.
Understanding Machine Learning From Theory To Algorithms by Shai Shalev-Shwartz et al.
Pattern Recognition and Machine Learning by Christopher M. Bishops
Information Theory, Inference, and Learning Algorithms by David J.C. MacKay
Deep Learning by Ian Goodfellow et al.
Convex Optimization by Stephen Boyd et al.

I am looking for a reading strategy. I'd specially appreciate input from users who’ve read the majority of the books.


texts: How to determine Nonnegativity in Matrix Factorization?

texts:
We have information about what the user likes in our app and we want to recommend content to similar users even those who may not have explicitly like a particular content but are similar to those who have explicitly liked that content.

I was planning to use the implicit feedback variation of Alternating Least Squares using Apache Spark to implement a recommender system which will find me the additional users that I can target.

One of the tuning parameters is a boolean flag nonnegative for non-negative matrix factorization.

I will be counting the number of likes per user and tag so I won't have negative values. Can I say nonnegative=true or does it mean something completely different. My understanding is that the nonnegative constraint is for the values inside the two matrices which the algorithm will decompose my original matrix to,  but I don't have the knowledge to know if those values will be non-negative for my scenario.

Ref:
Spark ALS Algorithm. Not exactly since there is another API but good enough for this question.


texts: What is the difference in xgboost binary:logistic and reg:logistic

texts:
What is the difference in R in xgboost between binary:logistic and reg:logistic? Is it only in evaluation metric?

If yes, how does RMSE on binary classification compare to error rate? Is the relationship between the metrics more or less monotonic, output from tuning on one metric should not differ significantly between those two approaches?


texts: Normalize Test and Control, or as a whole?

texts:
I have the following data:
\begin{matrix}
 &Test Sample\ 1 & Test Sample\  2 \dots & Test Sample \ 6& Control Sample\ 1 & Control Sample\ 2 \dots & Test Sample \ 6\\
measurements& 100 & 90 & & 30 & 20\\
\end{matrix}

If I want to normalize and compare between the test and control, should I normalize test and control samples separately or the whole matrix ?


texts: Compute Baseline/Representative of Time-Series Data

texts:
I have time-series data of 10-days for the same time interval as shown in below figure. Here it shows one-hour power consumption for 10 days. Data is sampled at 10 minutes rate.

I need to show this 10-day usage with a single baseline/representative curve. I can calculate baseline curve simply by taking mean/median of these 10 days data, but before that I need to answer following questions:

The baseline should not represent the outlier (abnormal) usage days. Here in figure, we see that day 1 and 2 are following an unusual pattern from the rest of days. I think the usage of these days should not be used in baseline calculation. How should I exclude these days from baseline calculation automatically?
How should I find the most similar usages out of these 10-day usages for my baseline calculation? I think the most/maximum similar usage days represent the days used for baseline calculation.


texts: Tune hyperparameters for cost-sensitive classification

texts:
I have an unbalanced data set with about 8% of negative examples. The goal is to minimize false negatives given a cost matrix. It seems like SVM (with radial kernel) and random forest work best. How should I tune hyperparameters in this setting?

My suggestion: separate data into train/validation set, use probabilistic output together with cost matrix to assign predicted classes, tune hyperparameters to maximize accuracy.

How can I increase performance? Currently I use random forest with nodesize=1 and mtry=5 which gives about 97% accuracy.


texts: Find partial derivative of softmax w.r.t logits in python

texts:
I have trouble implementing back propogation for multi class classification

My neural network has 2 layers

Forward propagation

X -> L1 -> L2

weights W are initialized as random

np.random.randn(this_layer_units, previous_layer_units) * 0.01

X is input of size (no_features * number of examples)

Z1 = (w1 * x) + b1

A1 = relu(Z1)

L1 has ReLu activation

Z2 = (w2 * A1) + b2

A2 = softmax(Z1)

L2 has softmax activation

cost is caluclated using cross entropy

cost = -(1/m)*np.sum((Y * np.log(A2) ) + ((1 - Y)*np.log(1-A2)))

Back propagation

derivative of cost is calculated

dA2 = -(1/m)*(np.divide(Y, A2) - np.divide(1 - Y, 1 - A2))

dA2 = derivative of cost w.r.t softmax

Y = one hot encoded True values

softmax is

np.exp(z)/ np.sum(np.exp(z))

now, how do i find dZ2 (derivative of Z2) w.r.t dA2 (softmax activation)

Link to entire jupyter notebook code


texts: How to create interactive plot of thousands of images as output of t-SNE?

texts:
I have many images that I want to plot as a result of running t-SNE and I want to be able to interactively explore them.  matplotlib does not allow enough interactivity to explore, and plotly is too slow.

Is there an easy way to plot thousands of images in a way that different patterns can be explored?


texts: How can I include prior knowledge about monotonic features into a neural network?

texts:
Let's make an example: I want to build a neural network which should predict if a person is obese. It gets the following features:

Weight
Height
Number of hours the person makes sports
superspecial health index
foobar

The output of the model is a probability that the given person is obese. The higher, the more likely.

Now I know, if everything else stays the same, that a higher weight should always be the same or a higher output probability. But I don't know the exact relationship, only that it changes monothonically.

How can I include this knowledge into the network and force it to show the desired behaviour?


texts: Finding repeating string patterns in thousands of files

texts:
I have files like the following: https://pastebin.com/5mkXY1aU

These are created by filling predefined forms so there are thousands of them that would match a pattern.

I will try to give more generic example for the sake of simplicity.

Let's say I have thousands of documents created using different patterns like `

- "sjkghkjfs <data> skjfs <data> kjskdfjsfkjs <data> sahkj";
- "tretyer erytewr fskjdf <data> trjk";
- "sdhfjsdhj <data> <data> <data> dsjadh";
- "<data> djfhsdk";` and so on.

<data> could be just any string.

I don't know what these patterns are beforehand. I want to find them out.

I can easily come up with some bruteforce solution but it would obviously not work on such amount of data.

It sounds to me like a problem that might have already been solved.

I wondered if there is some readily available software tool I could utilize directly to solve this or a library atleast ?

In case there isn't one how should I proceed to solve it in the most painless way ?

It is a one time task to find these patterns out, so I am not into the idea of spending days on implementing a solution from the scratch.


texts: Over fitting in Transfer Learning with small dataset

texts:
I am using Transfer Learning to perform image classification.
Base model used : Resnet50 using ImageNet dataset
class_1 and class_2 are the classes each having 1000 samples each (small dataset). And the dataset is not similar to ImageNet dataset.
Number of FC layers used here are 3 with [1024, 512, 256].
I have used a drop out of 0.5 to reduce over-fitting.
When I trained the model with 100 epochs, I could clearly see the model over-fits with training accuracy of 0.9985 and testing accuracy of 0.875.
Is the number of FC layers used is too many which is causing this over-fit problem?
How can I make the model more generalised?
The code used is as given below:
from keras.applications.resnet50 import ResNet50, preprocess_input
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Dense, Activation, Flatten, Dropout
from keras.models import Sequential, Model
from keras.optimizers import SGD, Adam
from keras.callbacks import TensorBoard
import keras
import matplotlib.pyplot as plt

HEIGHT = 300
WIDTH = 300
TRAIN_DIR = "/home/ubuntu/dataset/training_set/"
TEST_DIR = "/home/ubuntu/dataset/test_set/"
BATCH_SIZE = 8
class_list = ["class_1", "class_2"]
FC_LAYERS = [1024, 512, 256]
dropout = 0.5
NUM_EPOCHS = 100
BATCH_SIZE = 8

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
    for layer in base_model.layers:
        layer.trainable = False

    x = base_model.output
    x = Flatten()(x)
    for fc in fc_layers:
        print(fc)
        x = Dense(fc, activation='relu')(x)
        x = Dropout(dropout)(x)
    preditions = Dense(num_classes, activation='softmax')(x)
    finetune_model = Model(inputs = base_model.input, outputs = preditions)
    return finetune_model

base_model = ResNet50(weights = 'imagenet',
                       include_top = False,
                       input_shape = (HEIGHT, WIDTH, 3))

train_datagen = ImageDataGenerator(preprocessing_function = preprocess_input,
                                   rotation_range = 90,
                                   horizontal_flip = True,
                                   vertical_flip = False)

test_datagen = ImageDataGenerator(preprocessing_function = preprocess_input,
                                  rotation_range = 90,
                                  horizontal_flip = True,
                                  vertical_flip = False)

train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
                                                    target_size = (HEIGHT, WIDTH),
                                                    batch_size = BATCH_SIZE)

test_generator = test_datagen.flow_from_directory(TEST_DIR,
                                                  target_size = (HEIGHT, WIDTH),
                                                  batch_size = BATCH_SIZE)

finetune_model = build_finetune_model(base_model,
                                      dropout = dropout,
                                      fc_layers = FC_LAYERS,
                                      num_classes = len(class_list))

adam = Adam(lr = 0.00001)
finetune_model.compile(adam, loss="categorical_crossentropy", metrics=["accuracy"])

filepath = "./checkpoints" + "RestNet50" + "_model_weights.h5"
checkpoint = keras.callbacks.ModelCheckpoint(filepath, monitor = ["acc"], verbose= 1, mode = "max")
cb=TensorBoard(log_dir=("/home/ubuntu/"))
callbacks_list = [checkpoint, cb]

print(train_generator.class_indices)

history = finetune_model.fit_generator(generator = train_generator, epochs = NUM_EPOCHS, steps_per_epoch = 100,
                                       shuffle = True, callbacks=callbacks_list, validation_data = test_generator)

Update :

Weight file generated from the model after training is 2.7 GB. Is it normal considering the complexity of the model?

How would I select the steps_per_epoch value? Is there any standard?


texts:
I have a data of 700,000 transactions of forex traders which use various tools for investments. For beginners i want to find the association patterns between following metrics

performance / ROI
activity..
Instruments (e.g. Eur/USd, Eur/JPY etc.)
win/loss ratios
Margin used.

Then after that i want to find out what kind of above mentioned attributes contribute to the greatest win/loss ratios. Once the association patterns are made, i can look for the patterns with win/loss ratio and find out what associations are positively affecting this ratio.

I am currently running apriori but it's taking forever to output results. I wanted to know what are some other ways i can solve this problem?


texts:
I am learning data science. I have the following dataset for train tickets:

1. order_date_meduim

order,date,medium
95062,2017-09-11,35
171081,2017-07-05,39
122867,2017-08-18,39
107186,2017-11-23,
171085,2017-09-02,

2. order_ordercityA_ordercityB [some order has only 1 ordercity, I think ordercity means here which city is something like source and destination]

order,ordercityA,ordercityB
81773,4,11
105838,4,
76153,24,18
93058,12,
11623,24,3
3070,24,3

3. order_ticketcount,ticketclass

order,ticketcount,ticketclass
246783,1,pax
1693998,2,pax
1958576,1,other
673681,1,pax
1593899,1,pax
194035,1,pax

I need to forecast the ticket sales for a week and also the ordercity with medium of booking.

As I am new, could someone give a possible answer about how to create a prediction model that could predict the sales for 1 week? Also, I doubt the data is time-series data.

I code in Python.


texts: Do non-parametric models always overfit without regularization?

texts:
Let's scope this to just classification.
It's clear that if you fully grow out a decision tree with no regularization (e.g. max depth, pruning), it will overfit the training data and get full accuracy down to Bayes error*.
Is this universally true for all non-parametric methods?
*Assuming the model has access to the "right" features.


texts: Classification engine with multiple Textual independent fields

texts:
I have made a classification engine with only one independent field (Comments) and classified them in multiple dependent variables. Now I want to have multiple independent variables in training data (comments,state,age,gender etc).

Currently i am using python 3.6.

Example of what I have done:

Comments                                Classified
(The car wash service is good)          Positive

Example of what I want to do:

 Comments                          Gender    City   Age       Classified
 (The car wash service is good)    Male      LA     40        Positive

As  you can see in the second example we have 4 independent variables affecting the outcome(Classified sentiment). I want to implement a classification engine based on this use case. Please suggest how should i move ahead...???

Edit:- I have used SGDClassifier from sklearn.linear_model.

I have first passed list of comments as vectors which i have vectorized using TfidfVectorizer. Below is the code of vectorization:-

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(min_df=5,
                                 max_df = 0.8,
                                 sublinear_tf=True,
                                 use_idf=True)

train_vectors = vectorizer.transform(Train_data_Comments_list)

then I pass these vectors to train along with their classified labels. Below is the code(I have used partial_fit as i want to train again and again):-

classifier = SGDClassifier.partial_fit(train_vectors, Train_Labels)

Then I use the the classifier to classify rest of the vectors.

test_vectors = vectorizer.transform(Test_data_comments_list)
prediction_liblinear = classifier.predict(test_vectors)

If I combine all the fields value in my second Example and then form a single vector from that and then pass those vectors for classification, would that be the ideal way of doing it?


texts:
There's this side project I'm working on where I need to structure a solution to the following problem.

I have two groups of people (clients). Group A intends to buy and group B intends to sell a determined product X. The product has a series of attributes x_i, and my objective is to facilitate the transaction between A and B by matching their preferences. The main idea is to point out to each member of A a corresponding in B whose product better suits his needs, and vice versa.

Some complicating aspects of the problem:

The list of attributes is not finite. The buyer might be interested in a very particular characteristic or some kind of design, which is rare among the population and I can't predict. Can't previously list all the attributes;
Attributes might be continuous, binary, or non-quantifiable (ex: price, functionality, design);

Any suggestion on how to approach this problem and solve it in an automated way?

I would also appreciate some references to other similar problems if possible.

Great suggestions! Many similarities in to the way I'm thinking of approaching the problem.

The main issue on mapping the attributes is that the level of detail to which the product should be described depends on each buyers. Let’s take an example of a car. The product “car” has lots and lots of attributes that range from its performance, mechanical structure, price etc.

Suppose I just want a cheap car, or an electric car. Ok, that's easy to map because they represent main features of this product. But let’s say, for instance, that I want a car with Dual-Clutch transmission or Xenon headlights. Well there might be many cars on the data base with this attributes but I wouldn't ask the seller to fill in this level of detail to their product prior to the information that there is someone looking them. Such a procedure would require every seller fill a complex, very detailed, form just try to sell his car on the platform. Just wouldn't work.

But still, my challenge is to try to be as detailed as necessary in the search to make a good match. So the way I'm thinking is mapping main aspects of the product, those that are probably relevant to everyone, to narrow down de group of potential sellers.

Next step would be a “refined search”. In order to avoid creating a too detailed form I could ask buyers and sellers to write a free text of their specification. And then use some word matching algorithm to find possible matches. Although I understand that this is not a proper solution to the problem because the seller cannot “guess” what the buyer needs. But might get me close.

The weighting criteria suggested is great. It allows me to quantify the level to which the seller matches the buyer’s needs. The scaling part might be a problem though, because the importance of each attribute varies from client to client. I'm thinking of using some kind of pattern recognition or just asking de buyer to input the level of importance of each attribute.


texts: How to transform dictionary data into a string vector?

texts:
I have key,value data where each record is in a Python string.
An example record looks like this:

record = {
    'first_name': 'john',
    'last_name': 'doe',
    'age': '50',
}

To encode this into a neural net, I would like to firs have this data as a string vector. A sample output is like this:

"first_name john last_name doe age 50"

How can I make this transformation?


texts: Is there any time series model which handles data at variable frequencies.?

texts:
Goal: Predict the yellow points.(yellow events appear at varying frequencies)

But I'm struggling to find a good model to fit this use case.
Most of the time series algorithms are handling data which are at same frequencies(like per day/every 10 secs). I tried a lot of stuff but may be completely on the wrong approach.

Thanks a lot for any hint!!!


texts: Detecting and confirming sentiments in chat messages

texts:
Say I have a spreadsheet of 1000 chat messages that are labeled as happy, sad or funny.

What is the best way of detecting if the labels are indeed correct or if they need improvement?

How would I go about performing an analysis on the messages? How can I build a predictive model?

What type of feature and model selection should be used?

Python is my preferred language and I wish to load them as a CSV.


texts: How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

texts:
I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that one will understand their purpose after reading about how they are used for attention. I believe I understand what they do, but I'm unsure about how they are created.

I'm aware that they come from the multiplication of the input vector by three corresponding weights, but I'm not sure how these weights are derived. Are they chosen at random and trained like a standard neural network, and if so how if there's no predefined attention data in the training corpus?

I'm very new to this, so I hope everything I'm saying makes sense. If I've got something completely wrong, please tell me!


texts: Modelling for similarity between two descriptions

texts:
I have a dataset of companies and research projects that they were involved in. A subset of the dataset is shown below.

I am trying to find a way to model similarity between the company and the research projects using this description.
For each pair of descriptions, I would like to output a number, similar to cosine similarity, which will indicate how similar the description in second column is, to the project title in third column.
How can I go about this?
Thanks in advance!


texts: What is preferred upsampling or zero padding?

texts:
When training a CNN one option is either to zero pad an image to make it bigger or upsample it. When should I choose each one?
What criteria is leveraged for choosing a method?


texts: In classical kernel regression, is there a task which responds almost exclusively to a single kernel choice?

texts: I'm curious if there is any well-known kernel regression/classification task which can only be "solved" using a specific choice of kernel?


texts: Why class weight is outperforming oversampling?

texts: I am applying both class_weight and oversampling (SMOTE) techniques on a multiclass classification problem and getting better results when using the class_weight technique. Could someone please explain what could be the cause of this difference?


texts: Why is Amazon P2 instance cost much more than G3?

texts:
I'm currently looking for the correct number of 2018 budget for the AWS instances in preparation of more deep neural net training, but now a bit confused because of this chart below:

p2.16xlarge costs 3 times more than g3.16xlarge with the same number of processors but slightly less memory. Is this diff just because of the memory difference? or is there any other key difference between p2 and g3 instances? If I'm missing anything here, please share your wisdom!

Thanks,


texts: How to cluster multiple time-series from one data frame

texts:
I have a data.frame which has multiple time series in it, in the following manner:

01.01.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1
01.02.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1
01.03.16 Value1-Time-Series-1 Value2-Time-Series-1 Value3-Time-Series-1 ID-1
01.01.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2
01.02.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2
01.03.16 Value1-Time-Series-2 Value2-Time-Series-2 Value3-Time-Series-2 ID-2

Meaning:

There is one timeframe which is the same for every time series
There are multiple (thousands) of time series within that time
All are made up by the same type of values (columns)

I want to cluster this dataset in order to find the time series objects similar to each other.
I want to find those time series(s) which changed the same with respect to the values mentioned. I'm obviously interested in the measured distance. But using a simple hierarchical clustering algorithm is not cutting it since I need to tell the (whatever clustering algorithm I end up using) what is a time series and what not.

Any ideas how to this?

I'm using R for this. If there might be a better option with Python, I'm totally open to use it.

Thanks for any kind of help!


texts: How to cluster/identify points away from a regression line

texts:
For many vine plots, I have NDVI and Leaf Area values for each vine.
I already know that NDVI and LA has a strong positive correlation as you can see in this picture.

But as you can see too, there are some points away from this pool of points.
How can I cluster or identify these points?
I'm using Python with sklearn, scipy and other libs like them.


texts:
I found (How do I merge two data frames in Python Pandas?), but do not get the expected result.

I have these two CSV files:

# f1.csv
num   ano
76971  1975
76969  1975
76968  1975
76966  1975
76964  1975
76963  1975
76960  1975

and

# f2.csv
num   ano   dou  url
76971  1975 p1   http://exemplo.com/page1
76968  1975 p2   http://exemplo.com/page10
76966  1975 p2   http://exemplo.com/page100

How do I merge these for to get the result given below?

# Expected result
num   ano   dou  url
76971  1975 p1   http://exemplo.com/page1
76969  1975
76968  1975 p2   http://exemplo.com/page10
76966  1975 p2   http://exemplo.com/page100
76964  1975
76963  1975
76960  1975


texts: problem of choosing right statistical method for scheduler prediction

texts:
I am struggling to choose a right data prediction method for the following problem.
Essentially I am trying to model a scheduler operation, trying to predict its scheduling without knowing the scheduling mechanism and having incomplete data.

(1) There are M available resource blocks that can carry data, N data channels that must be scheduled every time instance i

(2) Inputs into the scheduler:

Matrix $X_i$ size M by N, consisting of N column vectors from each data source.    Each of M elements is index from 1 to 32 carrying information about quality of data channel for particular resource block. 1 - really bad quality, 32 - excellent quality.
Data which contains type of data to be carried (voice/internet etc)

Scheduler prioritizes number of resource blocks occupied by each channel every time instant i.

Given that

I CAN see resource allocation map every time instant
I DO have access to matrix $X_i$
I DON'T know the algorithm of scheduler and
I dont have access to the type of data to be scheduled.

I want to have a best guess (prediction) how the data will be scheduled based on this incomplete information i.e, which resource block will be occupied by which data channel. What is the best choice of prediction/modelling algorithm?
Any help appreciated!


texts:
i have to create four group using this data :-
Own_House   Own_vehicle Monthly_Salary  Standard_Savings Bank_Debts Person_ID
1   0   1   66278   3989    1
2   0   0   64701   7076    1
3   1   1   28045   5659    1
4   0   0   21690   6314    0
5   0   0   67479   9894    0
... ... ... ... ... ...
1996    0   1   54292   1468    1
1997    1   0   63079   7574    0
1998    1   0   69959   1788    1
1999    0   1   70333   7494    0
2000    0   0   83376   8616    0

i have used KMeans PCA but not getting so accurate results.
please assist me how to solve this dataset to get the better groups/ clusters.


texts: How to handle multi-channel 2-D geo-spatial grid like data samples in machine learning with number of features associated with each grid?

texts:
I am looking into a problem wherein the whole geographic area is divided into number of bins/pixels so we get nxn matrix covering whole region. Now each bin/pixel has number of parameters/features associated with it e.g., number of buildings in that bin, number of people living in that bin, poverty level of that bin, crime level in that bin, etc. This whole information represents one sample of the training dataset i.e., we have this matrix like data for different geographic regions with corresponding labels. How to best handle this kind of dataset for machine learning task
e.g., ML trained on some nxn grids for different areas like this will classify labels for some unseen test nxn grids. I am thinking that in term of CNNs, it may be represented in terms of channels so each channel represents associated features of a bin. Whats your suggestion?


texts: Standardization After PCA for Kmean clustering

texts:
I want to apply Kmean for clustering after PCA dimensionality reduction. I have standardized data with StandardScaler before the PCA, then I want to train Kmeans for finding clusters. However, the variance among the PCA components could not be of the same order of magnitude.

It is a good practice to standardize the PCA components before the clustering?


texts: Determining the right box size for packaging

texts:
I was given a very challenging problem at a logistics company and I would appreciate your help.
My company ships products from a clothing brand from the warehouse to final e-commerce client. The products are packaged in boxes. Two products can only be shipped in the same box if they were ordered by the same e-commerce client.
We know the exact dimensions (height/width/depth) of each product. We also know the products each e-commerce order had.
The task is to recommend 4 new box sizes to the clothing brand in order to minimize the volume when transporting the products.
Any help will be highly appreciated.
Thanks in advance.


texts: correlation coefficient between weather data and power consumption

texts:
I have power consumption and the corresponding temperature data for one year, that is, for several weeks (week1, week2,..., week48) as one month has 4 weeks. I want to forecast power consumption for a month.

Forecasting power consumption means forecasting 4 weeks(a month has 4 weeks). I am using Ms excel to obtain the pearson correlation coefficient. The power consumption data and temperature data have a strong correlation of 0.8 for week1 to week48, that is, they have a good correlation for one year. One year means over a large time interval.

But over a small time interval, for eg, correlation b/w power consump and temperature for 12 weeks (3 months) is weak (0.2). I want to use a neural network with window size of 3 to represent 12 weeks(3 months) to forecast the next 4 weeks(next month).

My question is: since the power data and temperature data has good correlation(0.8) for 48 weeks(1 year) but weak correlation(0.2) for 12 weeks(3 months), is it possible to use a neural network with window size of 3 as mentioned above to predict the power consumption for the next month with input variables power data and temperature data?


texts:
I am trying to use Caffe on the usual Fisher's Iris data set (150 flowers, each having 4 features, and split into 3 classes):

if a flower belong to class 1 (setosa), the network output should be [1, 0, 0]
if a flower belong to class 2 (versicolor), the network output should be [0, 1, 0]
if a flower belong to class 3 (virginica), the network output should be [0, 0, 1]

I use the SigmoidCrossEntropyLoss as it used for predicting K independent probability values in [0,1]. Let ip3 be the layer connected to SigmoidCrossEntropyLoss (in addition to the label layer): looking at ip3's output, I interpret negative values as class absent (0), and positive values as class present (1). E.g. if ip3's output is [-0.3; 0.9; -0.4], then I interpret it as [0; 1; 0], i.e. class 2 is present but classes 1 and 3 are absent.

The network does a good job classifying classes 1 and 3 (accuracy over 90%), but consistently fails to predict class 2: the network always predicts that class 2 is absent, i.e. the network's second output is always 0, i.e. that ip3's second output is always negative, no matter what the input is.

Here is the architecture I use. Am I doing something wrong? Is the architecture not suited for the task?


texts: use hidden layer of word2vec instead of 'one-hot', to reduce number of weights in other nets?

texts:
I've been reading about word2vec and it's ability to encode words into vector representations. Coordinates (probabilities) of these words are clustered together with their usual context-neighbor words.

For example, if we have 10k unique words in our dataset, we will feed 10 000 features into the network. The network can have for example 300 hidden neurons, each with linear activation function.

The output of the net are 10 000 neurons, each with a softmax classifier. Every such a "soft-maxed output" represents probability to select the appropriate word from our 10k dictionary.

Question 1: Well, I was able to get benefit of "semantically grouped" words via that final softmaxed-output layer. However, it seems way too large - it still has dimension of 10 000 probabilities.

Can I actually use that 300 neurons result in some other networks of mine? Say, in my LSTM, etc?

Then weights of my LSTM wouldn't take so much space on disk. However, I would need to decode the 300-dimensional state back into my 10k-dimensional state, so I can look it up

Question 2: ...are the "Encoded" vectors actually already sitting in the Hidden layer or in the Output Layer?

References:
first, second, third


texts:
How to replace a part string value of a column using another column.

My DataSet here is :

ID          Product Name                            Size ID    Size Name
1   24 Mantra Ancient Grains Foxtail Millet 500 gm      1       500 gm
2   24 Mantra Ancient Grains Little Millet 500 gm       2       500 gm
3   24 Mantra Naturals Almonds 100 gm                   3       100 gm
4   24 Mantra Naturals Kismis 100 gm                    4       100 gm
5   24 Mantra Organic Ajwain 100 gm                     5       100 gm
6   24 Mantra Organic Apple Blast Drink 250 ml          6       250 ml
7   24 Mantra Organic Apple Juice 1 Ltr Tetra Pack      7       1000 ml
8   24 Mantra Organic Apple Juice 200 ml                8       200 ml
9   24 Mantra Organic Assam Tea 100 gm                  9       100 gm

Requirement here is the Product Name column value is 24 Mantra Ancient Grains Foxtail Millet 500 gm and the Size Name column has 500 Gm.  In this case my output will be 24 Mantra Ancient Grains Foxtail Millet.
If the Size Name contains in the Product Name string remove the size name word ignoring the case else no need to take any action.


texts: Built strong base for Unsupervised Learning

texts:
I’m am new into machine learning, recently I have put a task upon my shoulders to Detect Outliers in Dataset.

The anomaly detection should be done using Unsupervised learning and preferably use Scikit-Learn

Can anyone give me any good source or course where I can specific dive into this ?
It so saturated that I don’t know where to start, a guidance on how to tackle this problem would be more than appreciated.

My dataset is in a form of a dataframe in a csv file


texts:
There are 2 parts to this question. Suppose we are looking at sales $S$ of a product across $> 1000$ stores where a it sells. For each of these $1000$ stores we have 24 months recorded data.

We want to be able to predict $S_t \leftarrow f(S_{t-1})$. We could build a RNN for each of the store time series, calculate test RMSE and then take an average after taking care of normalizing values etc. But the problem is there are very few samples per time series. If we were to segment stores into groups (by say Dynamic Time Warping) then could we create a monologue of text sentiment mining where like in text two sentences are separated by a dot here we would have two time series separated by a special symbol (let's say). In that case, we would generate a RNN model on

$$Train_1 | Train_2 |\ldots|Train_t$$

data and predict on

$$Test_1 | Test_2 |\ldots|Test_t$$

After this, we would like to set it up as a panel data problem where $S_t\leftarrow f(x_{t1},x_{t2},\ldots,x_{tn})$. In that case should I build a separate neural network for each $t$ and then connect the hidden layers from $t \rightarrow t+1 \rightarrow t+2 \ldots$.

How should I implement these through packages like Keras/Theano/Mxnet etc.? Any help would be great!


texts: Does increasing the n_estimators parameter in decision trees always increase accuracy

texts:
I'm using some ML algorithms (from sklearn lib) and on most of them there is a parameter n_estimators which is (if I understood well) the number of used trees. Intuitevely I would say that the more trees you use, the more accurate results you get. It turned out to be not exactly true, sometimes, a very few number of trees give much better results, but I can't figure out why ?

Edit

Some precisions: this is a regression problem, with a dataset containing
about 4000 samples and 500 features. I'm using GradientBoostingRegressor, ExtraTreeRegressor, AdaBoost, RandomForest.

Edit 2

Additional info: I use a cross-validation with KFold=10 to evaluate the accuracy of the algorithms. The best n_estimators value seems to be 50, which give a R2 score of ~56/57% +- 8% for all above cited algo. When I try to increase it, the score quickly decreases. I tried several values, from 100 to 500, it keeps decreasing even reaching 52%.


texts: What is the difference of R-squared and adjusted R-squared?

texts: I have in mind that R-squared is the explained variance of the response by the predictors. But i'd like to know how the adjusted value is computed ? and if the concept has any change from the original.


texts: Caret + RStudio: Error "Please make sure `y` is a factor or numeric value" when training

texts:
I'm new to Caret and I've been trying a couple things to get the hang of things. But this error happened to me and I'm not sure why.

I've been trying to train a model with some data I got from "PimaIndiansDiabetes". These are the "x" and "y" I'm using:

  >str(pima.Datos.Train[pima.Vars.Entrada.Usadas])
  'data.frame':   615 obs. of  8 variables:
   \$ pregnant: num  6 1 8 1 3 10 2 8 4 10 ...
   \$ glucose : num  148 85 183 89 78 115 197 125 110 139 ...
   \$ pressure: num  72 66 64 66 50 0 70 96 92 80 ...
   \$ triceps : num  35 29 0 23 32 0 45 0 0 0 ...
   \$ insulin : num  0 0 0 94 88 0 543 0 0 0 ...
   \$ mass    : num  33.6 26.6 23.3 28.1 31 35.3 30.5 0 37.6 27.1 ...
   \$ pedigree: num  0.627 0.351 0.672 0.167 0.248 ...
   \$ age     : num  50 31 32 21 26 29 53 54 30 57 ...

  >str(pima.Datos.Train[pima.Var.Salida.Usada])
  'data.frame':   615 obs. of  1 variable:
   \$ diabetes: Factor w/ 2 levels "neg","pos": 2 1 2 1 2 1 2 2 1 1 ...

  >pima.modelo <- train(pima.Datos.Train[pima.Vars.Entrada.Usadas],
                        pima.Datos.Train[pima.Var.Salida.Usada],
                        method='mlp')

train returns an Error: Please make sure y is a factor or numeric value.
But as far as I know, "y" is a factor, so I'm not really sure where the Error comes from. ¿Any help with this?
Thanks in advance


texts: Why is the variance going down so much in this weight initialization problem(using pytorch)?

texts:
first look at this example

>>> x = t.randn(512)
>>> w = t.randn(512, 500000)
>>> (x @ w).var()
tensor(513.9548)

it makes sense that the variance is close to 512 because each one of 500000, is a dot product of a 512 vector and a 512 vector, that is sampled from a distribution with a standard deviation of 1 and mean of 0

However, I wanted the variance to go down to 1, and consequently the std to be 1 since standard deviation is square root of variance, where 1 is the variance.

To do this I tried the below

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1/512)
>>> (x @ w).var()
tensor(0.0021)

However the variance is actually now 512 / 512 / 512 instead of 512/ 512

In order to do this correctly, I needed to try

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1 / (512 ** .5))
>>> (x @ w).var()
tensor(1.0216)

Why is that the case?


texts: Find similarity between 2 sample data points using Random Forest Classifier

texts:
Let’s assume that you trained a Random Forest model with 10 estimators on a dataset and passed 2 sample data points (S1 and S2) through each of the trees in the forest. You get to see the leaf node indices that correspond to each data point.

S1 = [5,4,1,7,8,2,5,6,1,2]

S2 = [4,5,1,7,9,1,7,6,3,2]

Note: The leaf node indices shown above are not one-hot encoded and correspond to the leaf node number in each tree.

I calculated the similarity between two sample data points by finding the common leaf indices and dividing them by total number of estimators so am getting 0.4.

Is this the right correct approach to solve the underlying problem?


texts: Dimensionality reduction for geometric curves using an autoencoder - what is wrong?

texts:
I am trying to play with toy models in order to study autoencoders. In particular, I want to do dimensionality reduction for simple geometric curves in 3D.
First, I take a toroidal helix.
n = 10

u = np.linspace(0, 2*np.pi, 1000)
v = np.linspace(0, 2*np.pi, 1000)

R, r = 1.0, 0.2

x = np.cos(v) * (R + r * np.cos(n * u))
y = np.sin(v) * (R + r * np.cos(n * u))
z = r * np.sin(n * u)

X = np.c_[x,y,z]
X_train, X_test = train_test_split(X, test_size=0.5)

Building an autoencoder that learns a 2D representation of this curve.
# The size of encoded representations
encoding_dim = 2

input = keras.Input(shape=(3,))
# "encoded" is the encoded representation of the input
encoded = layers.Dense(32, activation='tanh')(input)
encoded = layers.Dense(encoding_dim, activation='tanh')(encoded)

# "decoded" is the reconstruction of the input
decoded = layers.Dense(32, activation='tanh')(encoded)
decoded = layers.Dense(3, activation='tanh')(decoded)

# This model maps an input to its reconstruction
autoencoder = keras.Model(input, decoded)
autoencoder.summary()

I also create a separate encoder and decoder models similarly to this tutorial: link
# This model maps an input to its encoded representation
encoder = keras.Model(input, encoded)

# Encoded (2-dimensional) input
encoded_input = keras.Input(shape=(encoding_dim,))
decoded_output = autoencoder.layers[-2](encoded_input)
decoded_output = autoencoder.layers[-1](decoded_output)
decoder = keras.Model(encoded_input, decoded_output)

Train the autoencoder
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.005)

autoencoder.compile(loss='mean_squared_error',
                optimizer=optimizer,
                metrics=['mean_squared_error'])

autoencoder.fit(X_train, X_train,
                epochs=1024,
                batch_size=128,
                shuffle=True,
                validation_data=(X_test, X_test))

X_encoded = encoder.predict(X_test)
X_decoded = decoder.predict(X_encoded)

Plot the encoded features:
fig = plt.figure()
ax = fig.add_subplot()
plt.scatter(X_encoded[:,0], X_encoded[:,1])
ax.set_aspect('equal', adjustable='box')

I think, it looks fine but this is what I get when plotting the reconstructed curve:
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.scatter3D(X_decoded[:,0], X_decoded[:,1], X_decoded[:,2]);

What am I doing wrong? I tried to choose different hyperparameters but failed to reconstruct the helix.
I also tried it on a trefoil knot,
x = np.sin(u) + 2*np.sin(2*u)
y = np.cos(u) - 2*np.cos(2*u)
z = -np.sin(3*u)

The encoded features:

Reconstructed:

What am I missing? Did anyone play with autoencoders on similar toy data?


texts: Reward(t) vs. Reward(t+1) ? Reinforcement Learning, Q-learning

texts:
In "Reinforcement Learning, An Introduction; Richard S. Sutton and Andrew G. Barto" is written on page 70:
3.1  The Agent–Environment Interface

At each time step t, the agent receives some representation of the
environment’s state State(t), and on that basis selects an action
A(t). One time step later, in  part as a consequence of its action,
the agent receives a numerical reward R(t+1),  and finds itself in a
new state S(t+1).
[4] We use R(t+1) instead of R(t) to denote the reward due to A(t)
because it emphasizes that the next reward and next state, R(t+1) and
S(t+1), are jointly determined.  Unfortunately, both conventions are
widely used in the literature.

In case of trading a reward for opening a SELL(-1)/NONE(0)/BUY(1) could look like this:
reward = self.direction * (self.price[t+1] - self.price[t])

We step t+1 into the future...
However, i often see people doing it different: Q-Trader
for t in xrange(l):
        action = agent.act(state)

        # sit
        next_state = getState(data, t + 1, window_size + 1)
        reward = 0

        if action == 1: # buy
            agent.inventory.append(data[t])
            print "Buy: " + formatPrice(data[t])

        elif action == 2 and len(agent.inventory) > 0: # sell
            bought_price = agent.inventory.pop(0)
            reward = max(data[t] - bought_price, 0)
            total_profit += data[t] - bought_price
            print "Sell: " + formatPrice(data[t]) + " | Profit: " + formatPrice(data[t] - bought_price)

He's clearly using the current Price(t) for OPEN/CLOSE. And for the CLOSE that makes sense to me, as he's closing the trade NOW at Price(t) and NOT at Price(t+1), so the reward is Price(current)-Price(open)...
I somehow get confused because in one case the next value is used, in the other case the current. For opening a position the first way (t+1) makes more sense to me "self.direction * (self.price[t+1] - self.price[t])" as we are interested in the next value. But for closing it, we need the current value.
Question: Can i use both versions, OPEN rewards are R(t+1) and CLOSE rewards are R(t)? Is this a valid approach?
Thanks!


texts: Autocorrelation and Confidence Intervals

texts:
I was reading the Regression chapter of ISLR and could not understand the following line.

"If in fact there
is correlation among the error terms, then the estimated standard errors
will tend to underestimate the true standard errors."

How is correlation among the error terms going to underestimate the standard errors and there by confidence intervals? Please note that I am new to statistics and statistical learning.

Could you please help me with an explanation.

Thanks,
G Ravi Kiran.


texts: Does Arima support the usage of categorical variable? Some ways to get it working can be using one-hot encoding to represent categorical variables, but I am not sure how good it is.