texts: Is anything missing from my model evaluation procedure?

texts:
I have been building a model, can someone please review my methods and let me know if I am making a mistake?

I trained a model with a support vector machine as follows :

Split data into training and test sets as 10 partitions for K10 fold cross validation.

Split training set into training and validation set with K5 fold.

Train a parameter $C$ using validation set by choosing $C$ that gave best results from K5 fold tests.

Train a model with parameter $C$ and training data from K10 fold, 10 times for each partition from the K10 fold.

Take 1000 random samples of 80% of the test set partition data, and classify these random samples with the SVM. Calculate a mean and standard deviation. Repeat 10 times for each of the K10 fold partitions. Calculate a mean of all K10 partition means, and their combined standard deviation.

I am in the process of repeating this entire process 10 times, and then I will compute a mean and standard deviation of all 10 experiments.

For real world testing, I plan to repeat the above procedure but instead of splitting the data into train, test, and validate, I will use all data to find $C$ with K5 fold cross validation, and then test on real world data. Meaning, there will be no test set, the test set will become part of the training set, and the training set will be bigger because of this.

Is this the correct way to go about it?

Edit : Here is a diagram, hope its helpful. (Hyperparam = $C$)


texts:
AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability.

Is Roc the same as AUC?


texts: Probabilistic Machine Learning model to match spatial data

texts:
I have spatial data from multiple sources. This data consists of ID, lat, long, and time.

My goal is that given a new lat-long, the model needs to return (preferably with a probability) the data points that match the new lat-long. This matching should be based on the features (such as lat, long, timestamp).

I could only think of clustering. ie. Cluster the dataset and try to predict which cluster the new data belongs to. The drawback is that if the cluster has a lot of points then its hard to accurately pin point to which point in the cluster matches the closest to the new point.

Is there any other ways to do this? Any probabilistic model (HMM?).


texts: Regarding "modification" of feature columns in supervised learning

texts:
I have a training set with columns as follows:
$x_1,x_2,x_3,x_4, x_3/x_4$.
I want to know if I should consider $x_3$,$x_4$ and $x_3/x_4$ as separate features (as $x_3/x_4$ is derived from other features)? Also I found that dropping $x_3$ as a feature increases my k-fold neg-loss score (marginally!).

The underlying question is how would I know if entertaining $1/x_1$ or $x_1/x_2$ or even $\sqrt{x_4}x_3^4/x_1$... (you get the idea) as another feature column will/won't increase my score?

Is there any method/algorithm to know which "modified" features will increase my score?


texts: Machine learning model with simultaneous function optimization

texts:
Consider the following scenario. I am a sculpturer and customers ask me for what price I am willing to provide them with some statues. Their request for sculptures can vary in difficulty, quantity, material, size. In response, I reply with the price and subsequently they will either accept or reject (1 or 0) my proposal. I know of traditional machine learning methods that I can train to predict the probability of acceptance given all inputs.

In this case the price can be influenced by myself. Not let's assume that my expected profit is this function: (quantity * price * (probability_of_acceptance)). I would like to respond to the customers with the price where this expected profit is the highest. Of course you can input different prices in the machine learning model, which will give you some kind of probability acceptance curve that you can subsequently optimize against. But I am looking to do this in one go.

I am wondering:

What is the theoretical name for this type of problem?
What machine learning method is suitable here?
Are there any python packages that implement these models?


texts:
I'm trying to detect voice in a noisy environment. This is a New Zealand suburban/bush soundscape. We're using low quality microphones (but I don't think this has an effect on my problem) to periodically record the soundscape in an effort to record birdsong. One of the main challenges is we need to remove all potential human voice for privacy reasons. I'm stuck on the problem of how to detect voice in a noisy environment.

Here's an example of a spectogram with a person speaking 8 steps away from the microphone. There is crackle in the recording (don't believe this has an effect) and a bird chirping (activity around 2000hz) in the background. You can see the voice at 256-1024hz from 8sec to 13sec. The activity around 128hz from 15 to 22sec is a vehicle.

I'm trying to find a way of recognising voice activity (with high degrees of precision) just like in this file.

I believe that using a Machine Learning algorithm will have the best success due to the variability of voice and these sound files. (I have also been working on a mathmatical method based on energy proportion in the voice freq ranges per sample but this just doesn't seem to work with the variability of the sound files).

We only have around 10minutes of voice recorded like in the above spectogram. We then probably have about 24hours of non-voice recordings.

Q1: Will this be enough data, if not, would using data from freesound.org be advisable?

Q2: What types of algorithms would work best? - processing time is not a worry, the data contains noise from a multitude of sources (vehicles, animals, people, weather), I don't have much voice to work with(but I expect I can just use voice files from freesound.org) and I want to achieve 100% precision with not so much of a worry about recall.

Q3: Does anyone know of any projects in a similar space to mine where I could see what they've done? (there has been work done on voice detection [VAD] but from a few experiments it all isn't suited to as noisy environments as mine ie. this)

Q4: Last question, does anyone know any efficient techniques to label discrete events in continuous audio data to use in ML algorithms in Python? (At the moment I expect I'll be listening to files on Audacity and recording where voice starts and stops via a spreadsheet)

Otherwise, any other advice would be awesome!

(FYI I have done courses looking at ML-regression, Neural Nets and SVM/tree's but wouldn't call my self 'ML proficient')

Also, are there any other places I can ask these questions? I realise DSP.stack is more suited to solving small problems in the dsp area but I don't know where else to ask this!


texts: How to use Machine Learning to discover important biomarkers in an unbalanced small data set

texts:
I have a project which I am just starting out, I am only just learning machine learning and statistics so I am somewhat unsure as to what approaches will be good to start off with, and I am sorry if this does not belong here.

The data set is of  different patients carrying a certain disease and each patient has different biomarkers and  physical measurements such as heart rate at different time points, until death, if they do die. I was told that the goal was to identify the key features, which would be associated with a a patient dying.

I only have 33 patients, and only 16 of them have died. But disregarding patient the biomarkers came from I have 300 odd time slots, I first tried to approach it as a binary classification problem,  classifying the 'death' point from other points. The problems were:

The data imbalance and,
How to you interpret the models to discover most important features.

For imbalance, I tried SMOTE oversampling with didn't work as I thought, then I randomly under-sampled, which gave decent results but the data set was even smaller, so I wasn't sure if its a good idea.

Simple binary classification models like Gaussian Naive Bayes and Logistic Regression did okay even with the imbalanced data, but they don't (at least as far as I know) give a way to discern feature importance..

So my main questions are:

What's the best way to approach this problem, or in general what kind of approaches work when you want to identify most influential features (data measurements).
If I do want to approach it as a binary classification problem what approaches can I take to combat class imbalance?


texts: Do adaptive learning optimizers follow the steepest decent?

texts: For adaptive learning optimizers such as Adam and RMSProb, the effective learning rate is not the same for all weight parameters. This means that we are not really following the direction of the steepest decent vector in the weights space, correct? If so, why then should we bother with computing the gradient (via backprop) in the first place?


texts: Would averaging two vectors in word embeddings make sense?

texts:
I'm currently using the GloVe embedding matrix which is pre-trained on a large corpus. For my purpose it works fine, however, there are a few words which it does not know (for example, the word 'eSignature'). This spoils my results a bit. I do not have the time or data to retrain on a different (more domain-specific) corpus, so I wondered if I could add vectors based on existing vectors. By E(word) I denote the embedding of a word. Would the following work?

E(eSignature) = 1/2 * ( E(electronic) + E(signature) )

If not, what are other ideas that I could use to add just a few words in a word embedding?


texts: What's the best / easiest way to create a graph from address data?  For example, if I have 100 houses from all across a city is there any easy way to determine the shortest distance between two houses and all that good stuff?  Would this require changing the data into coordinates and using GIS software or can I get away with using Python or R?


texts: train_test_split() error: Found input variables with inconsistent numbers of samples


texts: How to split training/test/validation set according to a particular ID?

texts:
I have written the following code to split my dataset into a training, validation and test dataset. The dataset consists of multiple audio files from 39 participants. To prevent data leakage, I need to ensure that the audio files of each participant are either in the test/train/validation dataset (i.e. are not divided when the dataset is split). I also want to ensure that the ratio of the participants' diagnosis is evenly split between the three groups). I know it's possible to use GroupSplitShuffle however I can't find any examples that don't use dataframes. How do I modify my code to do this?
# create storage for train, validation, test sets and their indices
train_set,valid_set,test_set = [],[],[]
X_train,X_valid,X_test = [],[],[]
y_train,y_valid,y_test = [],[],[]

# convert waveforms to array for processing
waveforms = np.array(waveforms)

# process each diagnosis separately to make sure we builf balanced train/valid/test sets
for diagnosis_num in range(len(diagnosis_dict)):

    # find all indices of a single unique diagnosis
    diagnosis_indices = [index for index, diagnosis in enumerate(diagnoses) if diagnosis==diagnosis_num]
    print(diagnosis_indices)

    # seed for reproducibility
    np.random.seed(69)
    # shuffle indicies
    diagnosis_indices = np.random.permutation(diagnosis_indices)

    # store dim (length) of the diagnosis list to make indices
    dim = len(diagnosis_indices)

    # store indices of training, validation and test sets in 80/10/10 proportion
    # train set is first 80%
    train_indices = diagnosis_indices[:int(0.8*dim)]
    # validation set is next 10% (between 80% and 90%)
    valid_indices = diagnosis_indices[int(0.8*dim):int(0.9*dim)]
    # test set is last 10% (between 90% - end/100%)
    test_indices = diagnosis_indices[int(0.9*dim):]

    # create train waveforms/labels sets
    X_train.append(waveforms[train_indices,:])
    y_train.append(np.array([diagnosis_num]*len(train_indices),dtype=np.int32))
    # create validation waveforms/labels sets
    X_valid.append(waveforms[valid_indices,:])
    y_valid.append(np.array([diagnosis_num]*len(valid_indices),dtype=np.int32))
    # create test waveforms/labels sets
    X_test.append(waveforms[test_indices,:])
    y_test.append(np.array([diagnosis_num]*len(test_indices),dtype=np.int32))

    # store indices for each emotion set to verify uniqueness between sets
    train_set.append(train_indices)
    valid_set.append(valid_indices)
    test_set.append(test_indices)

# concatenate, in order, all waveforms back into one array
X_train = np.concatenate(X_train,axis=0)
X_valid = np.concatenate(X_valid,axis=0)
X_test = np.concatenate(X_test,axis=0)

# concatenate, in order, all diagnoses back into one array
y_train = np.concatenate(y_train,axis=0)
y_valid = np.concatenate(y_valid,axis=0)
y_test = np.concatenate(y_test,axis=0)

# combine and store indices for all diagnoses train, validation, test sets to verify uniqueness of sets
train_set = np.concatenate(train_set,axis=0)
valid_set = np.concatenate(valid_set,axis=0)
test_set = np.concatenate(test_set,axis=0)

# check shape of each set
print(f'Training waveforms:{X_train.shape}, y_train:{y_train.shape}')
print(f'Validation waveforms:{X_valid.shape}, y_valid:{y_valid.shape}')
print(f'Test waveforms:{X_test.shape}, y_test:{y_test.shape}')

# make sure train, validation, test sets have no overlap/are unique
# get all unique indices across all sets and how many times each index appears (count)
uniques, count = np.unique(np.concatenate([train_set,test_set,valid_set],axis=0), return_counts=True)

# if each index appears just once, and we have 1440 such unique indices, then all sets are unique
if sum(count==1) == len(diagnoses):
    print(f'\nSets are unique: {sum(count==1)} samples out of {len(diagnoses)} are unique')
else:
    print(f'\nSets are NOT unique: {sum(count==1)} samples out of {len(diagnoses)} are unique')


texts:
I am a first year PhD student in statistics. During this year I have analyzed the scopes of interest of my scientific advisor and found them unpromising. He is majored in mixtures with varying concentrations models for which I have not found any references to authoritative sources.

Now I want to change my PhD theme, however, there are no other scientific advisors in my university which majored in statistics. Therefore, I have 2 questions:

Is it possible to write at least 5 articles together with a PhD thesis without scientific adviser? If yes, what is a proper way to do this? Here I mean how to choose a theme, where ask for a help and so on.
Is it possible to find a remote adviser to consult with? If yes, how and where can I find him?

Also I have no much time for the search. I am interested in statistics, especially in machine learning. I would like my PhD thesis to be of practical value, not pure research one which is popular in my department. Also I have commercial experience in programming (C/C++, R, Python) if that can help.

Thanks in advance for any help!


texts: Converting a string to dummy encoded variables

texts:
Here's the data

PlayerID, Characters, Win or Lose

I can make it look like this

8PYPY0LLQ,valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1, 0

Or like this

2GRG822L9,"barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,",1

The second column is an 8 character combination from 70+ n characters.

I need to encode the variables to be dummy variables, so each character gets its own column. Is there a way to do this in python/R? I'm assuming you have to leave the second column as a string rather than outputting a csv file that looks like this.

2GRG822L9,barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,1
8PYPY0LLQ,valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1,0

It should probably look like this before dummy encoding (I can get rid of commas within the string)

2GRG822L9,"barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,",1
8PYPY0LLQ,"valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1,",0


texts: How to get the number of steps until a certain accuracy in keras?

texts:
I want to see how many steps does it take for my model to reach a certain accuracy.Say 90 percent on cifar10.How can I get this info from the keras model ?
EDIT:
accuracy in each epoch is accessible in history object fit() returns,but im looking for accuracy in each step
Solution:
I made a callback object that keeps loss in each step
import pickle
from tensorflow.keras.callbacks import Callback

class LossHistory(Callback):
    def __init__(self,path='',name=''):
        super(Callback, self).__init__()
        self.path = path
        self.name=name
        self.accuracy = []
        self.losses=[]

    def on_batch_end(self, batch, logs={}):
        self.accuracy.append(logs.get('accuracy'))
        self.losses.append(logs.get('loss'))

        history_={}
        history_['accuracy']=self.accuracy
        history_['loss']=self.losses

        with open(self.path+self.name+'_history.pkl', 'wb') as f:
            pickle.dump(history_,f)


texts: selecting only a certain number of top features using tsfresh

texts: How can I select top n features of time series using tsfresh? Can I decide the number of top features I want to extract?


texts: What is the best practice for data folder structuring?

texts:
I work for a small data science consultancy firm and we are trying to standardize our project folder structure.
We started from the cookiecutter structure which is a great base.

However one of the discussion point lies in the subfolders of the data folder, which is structured as:

Raw
Interim
Processed

Let's think about the following situations:

The client gives you a manually extracted csv file -> This obviously goes into Raw
You have acces to SQL databases and make a no-modification extract
    -> Still into Raw I guess?
Because of very large databases, you create a semi-complex SQL query
    as base for a feature -> Is this Raw or Interim?

What are the best practices you apply? What would you recommend?

Ps: links to Github projects constructed following this kind of structure are very welcome


texts: how do deep Q network deal with varying input size?

texts:
I am conducting research with multiply agents in an environment. The main concept of my methodology is a centralized control system, which means we take the positions, as well as other information, of all agents as a collection. Say we have 5 agents in the environment, the state is

S = [X_0, X_1, X_2, X_3, X_4]

And the output we want is also a vector(matrix):

A = [a_0, a_1, a_2, a_3, a_4]

I have managed to apply this as inputs to a fully connected Deep Q Network. The input layer is 5 nodes, which contain the positions in the grid world. However, the number of agents is unknown in the environment, and if there are n = 4 or 6 agents, the fully connected neural network can't be applied to yield the action of desire. We have to build a new neural network.

What are some nice ways to deal with this, so that we can have a DQN network structure dealing with varying size of input layers. So we only need to train the network once and apply it for offline training.

I did some research on it, it suggests using recurrent neural network (RNN), while deep recurrent neural network (DRQN) seems to aim for a different target. I also think about "padding" an input state metric containing, e.g, 100 agents, but only first n agents observed by the environment are fulfilled with meaningful information. The rest is nullified so that they wouldn't interact with the dynamics of the environment.

S = [X_0, X_1, X_2, ..., X_99] where X_n, X_n+1,...,X_99 = 0(or sth invalid, as long as it doesn't influence next state)

I am wondering if such padding strategy would work, and how efficient it would be because it seems like we built an extremely large and inefficient neural network structure.

Thanks in advance!


texts:
I ran auto_arima on my model from the library pmdarima and I'm trying to interpret the results printed to the console. I see two examples of similar parameters that yield different results:
ARIMA(0,1,0)(0,0,0)[108] intercept : OOB=.......
and
ARIMA(0,1,0)(0,0,0)[108]           : OOB=.......
The first example has the term "intercept" in the output but the second doesnt. The second instance has much better performance than the first, which indicates to me I want an ARIMA model with no intercept? I can't however find any indication in the ARIMA package that there's an "intercept" field I can either specify or not specify.
Any help is appreciated in understanding this output.
Thanks!


texts: Synchronizing timestamps between multiple sources of time-series data that are both asynchronous and imprecise

texts:
I have time-series data coming from multiple sources. The data from each source arrives at a "roughly known" interval (i.e. once per hour), but the timestamps across the sources are not synchronized with each other in any way, and to make matters worse, are not very precise (i.e. a data source that generates a new sample "once per hour" may actually send a sample every 60 +/- 5 minutes or so; and the error "drifts" for a given source between reports).
In other words, I know that all my sources of asynchronous/imprecise timestamps will each produce one sample in a known interval. I need to be able to accommodate the individual "drift" of each source, and still end up with one "normalized" timestamp for each sample per source per interval.
What are some ways, without regard to any particular software (I'm trying to understand the underlying approaches to this problem), to synchronize these timestamps over a long period of time?


texts: Dealing with Skewed data using Boxcox transformation by adding 1

texts:
I was trying to reduce the skewness of data using boxcox transformation. But was facing an error:

  ValueError: Data must be positive.

I figured out why it was throwing an error, my data ranged from .0 to 1 so I added one and it worked.

Although is it the right way to deal with it?


texts: Best practice on count of manual annotations for building criminal detection from news articles?

texts: We have 7 million news articles corpus, which we want to classify into crimes or non-crimes and further identify criminals by using NERs/annotating criminals, crime manually. For creating a model that identifies criminals, what is the number of annotated articles that we must train/build our model on? Is there any industry best practice on this count? Is there any better way to come to this number of training(annotated) dataset, than random guessing? Are there any best practices resources that anyone can point to? Thanks in advance!


texts:
I came across this sort of flowchart:

Below the flowchart, this is what appears:

“Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance. Using the best configuration of tuning parameters, we then train the models M2 and M3 on the original training set and use the original test set to compute the corresponding test RMSEs.”

They are only mentioning the cross validation error (validation) and never mention the train cross validation error.
Is  the phrase “The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance” correct? I mean, assuming that by “lowest overall cross-validation error leads to the best model performance”, they are referring themselves to the “validation” error of the cross validation technique, I wonder why are they making such assumption? Should we care about the averaged train cross validation error or just the averaged validation error?

I am using a library to play with recommender systems which has a parameter called return_train_measures = True. Then it throws both, train and test errors:


texts: How the squared Euclidean distance is an example of non-metric function?

texts:
I am reading a book on Pattern Recognition (by Prof V Susheela Devi and Prof Murty) where in the chapter of data representation 2.3.3 the non metric similarity function is defined as those which do not obey either the triangular inequality or symmetry.

In that context, it further adds The squared Euclidean distance is itself an example of a non-metric, but it gives the same ranking as the Euclidean distance which is a metric.

Is the squared Euclidean distance different from the Euclidean distance?

How is the squared Euclidean distance non-metric?


texts: Which NLP library has the most mature Chinese language models?

texts:
I am trying to do some NLP on Simplified Chinese texts (needing to extract sentence structure and to do named entity recognition).  I've used spaCy previously for English texts, but I see the notes on the Chinese models suggest they are a work in progress, and the NER extraction accuracy has been poor for the examples I've tried.
Which NLP library has the most mature pre-built Chinese language models? Ideally Python based.


texts: What are some nice projects to do on Knowledge graphs?

texts:
Okay, so coming straight to the point, I have to do a major project to demonstrate the use cases and advantages of using knowledge graphs. I have seen some ideas in internet like fraud detection and movie recommendations system but quite not convinced with them. Please suggest any good idea.
Thanks.


texts: Predicting a final exam score from arbitrary set of practice exams

texts:
I posed this question to math.stackexchange without response, and feel it is better suited here. I am writing an application to predict a final exam score given at least one tuple, where the tuple represents the results of a practice exam:

(score, exam_name, date_taken[optional])

Users input practice exam scores and, later, their official exam score (date required); not all users take all available practice exams. The dataset can be represented like this:

$$
\begin{bmatrix}
a & b & c & d & e & f\\
g\\
h & i & & j\\
k & l & m &&n
\end{bmatrix}
\begin{bmatrix}
(actual,date)\\
(actual,date)\\
(actual,date)\\
(null,date)
\end{bmatrix}
$$

Each row is a user. Each column in the first matrix is a particular practice exam; the second matrix is the score of the final exam (actual) and date taken. Each element in the first matrix is a tuple of the form described above.

Users complete an arbitrary combination of practice exams. The user represented by the 4th row has inputted four events (four practice exams), the date he plans to take the official exam, but not his actual score; as such, we want to provide him with a prediction.

What model(s) might be suited for this problem?

I have a hunch that naïve Bayes might be best because of the "incompleteness" of the matrix, and because I want to provide a prediction to the user after any input of data (i.e., user inputting a single tuple with no date, (512, mcat_24), should still receive a prediction). The feature of the problem most difficult to me is the fact that not all users take all practice exams; I'm not sure how to formalize that mathematically. I am working my way through this paper1, which may help.

[1] https://arxiv.org/pdf/1508.03865.pdf


texts: One hot encoding at character level with Keras

texts:
I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says:

Note that Keras has built-in utilities for doing one-hot encoding of text at the word level or character

I have looked into Keras metods and I cannot find which function he is referring to. keras.utils.to_categorical does not seem to be applicable directly here, as it requires int inputs.

I would like to efficiently encode some text at character level for a small RNN project: what can I use there?


texts:
I'm studying Data Scaling, and in particular the Standardization method.
I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit variance.

Can you explain me ?


texts: Algorithm for multiple input single output ML

texts:
As an ML newbie, I have a question. I have a set of data with 2 inputs and 1 output. I'm trying to predict the output.

input1 is an integer number, input2 is like a category between 1-5. Output is also a number.

input1=25 input2=2 output=25
input1=34 input2=2 output=35
input1=12 input2=5 output=29
input1=3 input2=4 output=48
input1=45 input2=1 output=36

With this data, I want to predict the output for input1=27 and input2=2

I have a small set of data (10-20 items). I wonder which ML algorithm should I learn for this kind of multiple inputs and single output small sets of data?

Edit

With a high probability, while calculating the output, there is a mathematical relation between input1 and input2 like:

output = (input1)*x + (input2)*y (x and y is unknown of course and the equation can be linear or logarithmic or something else. No idea.)


texts: Sequence-to-Sequence Autoencoder with asymmetric length of encoding sequence

texts:
I am trying to design a sequence-to-sequence autoencoder where the encoding sequence has a length shorter than the sequence itself given that the encoding's dimensionality could be more to compensate for that.
Each unit of the resulting encoding sequence should have encoded many timesteps of the input sequence based on similarity or relatedness with each other and then decoder should be able to decode the exact sequence from the above encoding. Basically, if we expect an input sequence to consist of shorter sub-sequences of variable length, I want an encoder than identifies the similarity between the timesteps of a sub-sequence and encode that pattern in a single unit so that the decoder can use that information to decode that information by "calculating" how to the output all the timesteps over a short distance of sub-sequence. All this is to be done in an unsupervised fashion as these shorter sub-sequences are not labelled, hence these short term patterns have to be learned.
As an example, consider the task of phenome extraction where a sound utterance has to encoded into (shorter in length but dense) phenome sequence and then decoded back to original utterance.
The below image tries to depict what I want.
Mainly, I would want to know the bottleneck that could bring out this type of encoding in an autoencoder. I'm also open other models/architectures that might help.
NOTE:
The input sequence would be a time-series sequence and please expect individual timesteps to be related to their adjacent few timesteps in similarity.
For instance, if input is a speech utterance, I would want to capture the relation of timesteps at phenomic level which would be localised together in the input in continuity. Like the red circles can be timesteps of /k/, the green ones /æ/ and the blue ones can be /t/ if input utterance is "cat". Hence, it is easy to see that red circles would be clumped together as the blue and green ones would be with themselves.


texts:
I have a short time series of daily counts ( 6 days = 6 counts : 12, 15, 69, 35, 97, 107). This looks like a rapid increase from its initial count of 12.

What are some of the statistical techniques to detect such large increases in short time period?

"rapid increases" and "short time period" are rather loose terms, but, I am looking for techniques where these can be 'parameters' that I can provide.  For example, "short period" could be 5 days. 'Rapid increase' could be 20% increase day-to-day or 90% increase in 5 days...something like that.

Suggestions are appreciated.


texts:
It sounds like dump question but as a beginner, I'm really getting confuse.
For my academic thesis, I choose a conference paper on Big Data in Healthcare field. Now, problem is to get the Data sets.
I can't find any resources to download the data sets to work on it. When I google it, people suggested some resources which are really good but those file is not Big to call Big Data. I'm really confuse in this point and can't get right answer on this from anyone.
I was thinking, may be we work on small batch of data sets and further apply the models on Big Data in production sphere. However, it's just an assumption. I want to know, for thesis or analytical research in academic level, can we get real life Big Data sets to download for free?
Please correct me if I'm missing something. Thanks in advance.


texts: Does finetuning BERT involving updating all of the parameters or just the final classification layer?

texts:
Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP.  But during finetuning, for example trying to classify sentiment based on another text, are all of the BERT parameters (110M+ parameters + final classification layer) updated or just only final classification layers?  Couldn't find a concrete answer to this in the resources I've been looking at.
Thank you in advance.


texts:
I have a data set that looks like the following:

 Time        V1     V2    V3     ...    V40
13:00        0.44   0     0.33          0.55
13:01        0.55   0     0.34          0.52
13:02        0.58   1     0.20          0.58
.
.
.
15:01        0.57   0     0.24          0.70

Where V2 is the binary equivalent of on/off switches. Currently, I am still pre-processing my data and normalized the data-set from (0,1) using sklearn.preprocessing. I am wondering if applying dimensionality reduction/PCA to my dataset will affect the outcome of my model and whether if it is advisable to use it to process my data.


texts:
Say I want to predict the final size of a flower depending on the raining and temperature during 200 days. The flowers are in differents towns, so each flower has its own conditions of rain and temperature.

I have :

500 flowers

For each flower :
- 1 time serie with 200 points corresponding to raining volume each day
- 1 time seris with 200 points corresponding to temperature on each day

known target : final length for each flower (500 data)

what should be the input dimensions of my LTSM network ?


texts: Is the number of iterations in gradient tree boosting just the number of trees?

texts:
I have been searching for a while and I just can't find any indication. When people talk about iterations in algorithms like XGBoost or LightGBM, or Catboost, do they mean how many decision trees i.e. base learners will be built? I.e. XGboost m=100 means the algorithm will build a total of 100 base learners, each calculating and optimizing towards the residual value of the previous prediction?
Or is it more like 1 epoch in deep learning?


texts: How to interpret classification output - Predective model

texts:
What is the significance of macro avg ? I'm not sure if this report signify a good predictions by the model.
Thanks in advance.


texts: How to handle the CEO expectations from a company that's new to data science?

texts: I'm new here. I'm about to have a final interview for a data scientist position for a company (it's in the e-commerce field) that is new for data science. It's a pretty new position for the company, and from the interviews I had so far, I noticed that they don't fully understand what they want from a data scientist. They barely know what data science is. I explained them the "standard" data science workflow (Ask A Question, Get the Data, Explore the Data, Model the data and Communicate the data). But I don't think they got it. I want to set the proper expectations, so the company and I can agree on my job description, and what they can expect from me in one month, tree months, six and twelve months. So, how do you handle the expectations(specially the CEO) from a company that's new to data science?


texts: solving multi-class imbalance classification using smote and OSS

texts:
I am trying to solve multi-class imbalance classification problem for that i am using SMOTE for oversampling and OSS for under-sampling. But I have a doubt as I am working on multi-class so i have to convert it into binary classification. So we can convert it using OVA/OAA. So how can I use OVA/OAA with both under-sampling and oversampling on the same data-set.

Please solve my problem, It will be a great help.


texts:
Have some issue with understanding how to use TSFERSH-library (version 0.4.0) to forecast next N-values of particular series. Below my code:

    # load data train/test datasets
    train, Y, test, YY = prepare_train_test()
    # add series ID
    train['TS_ID'] = pd.Categorical(train['QTR_HR_START']).codes
    test['TS_ID'] = pd.Categorical(test['QTR_HR_START']).codes
    # add ordered id for concrete event of series
    for id in sorted(train['TS_ID'].unique()):
        train.ix[train.TS_ID == id, 'TIME_ORDER_ID'] =  pd.Categorical(train[train.TS_ID == id]['DATETIME']).codes
    for id in sorted(test['TS_ID'].unique()):
        test.ix[test.TS_ID == id, 'TIME_ORDER_ID'] = pd.Categorical(test[test.TS_ID == id]['DATETIME']).codes
    # perform feature extraction for my signal
    extraction_settings = FeatureExtractionSettings()
    extraction_settings.IMPUTE = impute  # Fill in Infs and NaNs
    X = extract_features(train, column_id='TS_ID', feature_extraction_settings=extraction_settings).values
    XT = extract_features(test, column_id='TS_ID', feature_extraction_settings=extraction_settings).values

    # there should be as example
    # model = xgb.DMatrix(X, label=Y, missing=np.nan)
    # model.fit()
    # model.predict(XT)

However, after line X = extract_features(...) I see at debugger following results

It's mean that initial X-dataset/features (shape=(722,10) were transformed to shape (80, 1899).

Where does '80' come from? I guess from train.TS_ID comes. But my XT-dataset still contains 722-rows (9 days * 80 different series per day).

So, how can I predict for 9 days in advance? or is there only forecast for next period?


texts: Preferred approaches for imbalanced data

texts:
I am building a binary classification model with imbalanced target variable (13% Class 1 vs 87% class 0). I am considering the following three options to handle the data imbalance

Option1: Create a balanced training dataset where with 50% / 50% split of the target variable.
Option 2: Samples the dataset as-is (i.e., 87% / 13% split) and use upsampling methods (e.g., SMOTE) to balance the target variable to 50% / 50% split.
Option 3: Use learning methods with appropriate hyperparameters to account for data imbalance for example: scale_pos_weight in XGBoost, class_weight in LGBMRegressor, class_weight in RandomForestClassifier

Assuming I have enough available data, is the first option is always the best approach?
What are the Cons and Pros of each of the three methods? especially the 2nd and 3rd options (I assume that it is always preferred to avoiding creating new synthetic samples)


texts: How to extract embeddings of categorical variables

texts:
I am little bit confused about encoding categorical variables. There are other posts/blogposts on this issue but none is talking about the problem I am facing.
I have a dataset with mixed variables (i.e, numerical as well as categorical). Some of the categorical variables has a lot of categories (close to 100). So instead of using One Hot encoders, I am looking into using embeddings.
My goal is to: Use the embeddings of the categorical variables and extract them and put them together with my numerical variables to form the design/training matrix and then run an autoencoder on it (the reason to run autoencoder is to detect anomalous data ).
I have put together a code snippet (see below) but what confuses me is to how to extract the embeddings to put it together with my numerical variables. I need to know because when new data shows up, I need the embeddings to run it through prediction to isolate any "odd" datapoint from the reconstruction error.
from keras.models import Model
from keras.layers import Input, Dense, Concatenate, Reshape, Dropout
from keras.layers.embeddings import Embedding

inputs = []
embeddings = []

# for categorical variables
for cat in cat_model_vars:
    inp = Input(shape=(1,))
    inputs.append(inp)
    emb = Embedding(cat_sizes[cat]+1,cat_embedding_size[cat], input_length=1,name=f"{cat}_embedding")(inp)
    emb = Reshape(target_shape=(cat_embedding_size[cat],))(emb)
    embeddings.append(emb)

# for continuous/numerical variables

inp_num = Input(shape=(len(num_model_vars),),name='cont_vars')
emb_num = Dense(10)(inp_num)
inputs.append(inp_num)
embeddings.append(emb_num)

# concatenate the embeddings, this is going to be input to our Autoencoder model
output = Concatenate()(embeddings)

#%%

# build a stacked Autoencoder below and feed the output as it's input and output

en_x = Dense(50, activation = 'relu')(output)
en_x = Dense(32, activation = 'relu')(en_x)
en_x = Dense(16, activation = 'relu')(en_x)
en_x = Dense(4, activation = 'relu')(en_x)

de_x = Dense(16, activation = 'relu')(en_x)
de_x = Dense(32, activation = 'relu')(de_x)
de_x = Dense(50, activation = 'relu')(de_x)

output = Dense(25, activation = 'relu')(de_x)

stacked_ae_model = Model(inputs, output)
stacked_ae_model.compile(loss='mean_squared_error',optimizer='Adam',
                         metrics=['mse','mape'])

"""NEED TO EXTRACT THE EMBEDDINGS THAT ARE USED AS AN INPUT TO THE STACKED AUTOENCODER (NOT DONE YET)"""

Now I will have to run the training and prediction. But I am not sure how to extract the embeddings so that I can use those for training.
Any help would be greatly appreciated.
Thanks and regards.


texts: How to handle One Hot Encoded columns with changing categories in supervised ML Problem?

texts:
Scenario:
I have the following game data about participants, game and the winner in the following format:
id  game        player  winner
-----------------------------------
1   Monopoly    Paul    Paul
1   Monopoly    Frank   Paul
1   Monopoly    Sandra  Paul
2   Monopoly    Peter   Katie
2   Monopoly    Paul    Katie
2   Monopoly    Erwin   Katie
2   Monopoly    Katie   Katie

So the table above shows information about two sessions of monopoly that have been played, the first one with Frank, Paul (Winner) and Sandra; The second one with Peter, Paul, Erwin and Katie (Winner).
My question is: If I want to predict the Winner of a game, how would I transform the data so that each row corresponds to one session?
Idea 1
My initial idea is of course to one hot encode the "player" feature:
id  game        Paul    Frank   Sandra  Peter   Erwin   Katie   winner
-------------------------------------------------------------------------------
1   Monopoly    1       1       1       0       0       0       Paul
2   Monopoly    1       0       0       1       1       1       Katie

That way, it would be easy to predict the target with a simple model.
However, if I want to predict a winner of a new monopoly session with players Sandra, Erwin, Katie and Tim (New Player), the transformed (i.e. one hot encoded) data
would have one more column. This column is unknown to the model and thus cannot be processed.
Idea 2
Take Idea 1, add a "new player" binary dummy column (the new player name does not contain any useful information anyway), store the winner, retrain the model with the new (additional) information. However, if I play with multiple new players, this would not work (unless I add multiple dummy columns, which would be quite inflexible).
Idea 3
Monopoly can be played with 2 to 8 players, so I could change the table structure to the following:
id  game        Pl_0    Pl_1    Pl_2    Pl_3    Pl_4    Pl_5    Pl_6    Pl_7    winner
---------------------------------------------------------------------------------------
1   Monopoly    Paul    Frank   Sandra  None    None    None    None    None    Paul
2   Monopoly    Peter   Paul    Erwin   Katie   None    None    None    None    Katie

However, the problem with this approach is of course the structure assumes that the is a semantic meaning behind which player takes which "seat":
In game 1, Paul was Player 0 (i.e. the value of Pl_0 is "Paul") but in game 2, Paul was Player 1 (i.e. the value of Pl_1 is "Paul"). In this structure, the
model cannot know that it makes no (semantic) difference, whether Paul is Player 1 or 2. Furthermore, a game that can be played with 10 players cannot be mapped like that. So rejected that idea pretty quickly. Or is there any way to overcome this problem?
Final Question
How would I be able to predict the winner of a game, such that

the model can handle new (unseen) players
the model can handle multiple games with different numbers of players


texts: is there any way to plot ROC curves from weka

texts: i am using some algorithms from weka . i was willing to plot some algorithms' roc curve for comparison . Is it possible and how ?


texts: How to convert google trend relative values into more “absolute” values?

texts:
Searching key words or terms on google trends provides a time series of relative search numbers. I want to be able to analyse these trends more accurately compared to general trends in the same category across the last 90 days, as relative searches provide little value into the true increase of a certain key word. For example, an increase in relative searches from 50 to 100 could only mean an absolute increase in 1 additional search.

My initial approach was to compare each term I was interested in with a benchmark term. For example I would get the relative searches for the benchmark term "fruits", and then I could divide the relative daily searches of "oranges", "bananas" and "apples" by "fruits" (trend data example shown in the image below).

Google Trend Example - Comparing relative searches using "fruits" as a benchmark

The issue I have with this is that if the benchmark relative search is 0, then the standardised values will be infinite. I was thinking a simple subtraction of each fruit's relative searches from the could be used as an alternative. I also came across this prior question of a similar goal: https://stackoverflow.com/questions/15135334/google-trends-obtaining-absolute-values. It suggests using a pair wise approach (e.g. apple relative to orange, orange relative to banana and so on). Could anyone explain the logic behind this in a bit more detail?

I've cross-posted this from stack overflow, but I think it is actually more appropriately posted here.


texts: Choosing an algorithm with normalized data(Classification)

texts:
I am having a dataset which all the features are from 0 to 1(real numbers) and the output is 0 or 1(integer numbers). Example:

var1   var2   var3  output
0.01    0.1    0.7       1
0.01    0.1    0.7       1
 0.1    0.2    0.3       0
 0.2    0.4    0.4       0
 0.4    0.1    0.9       1

Which classification algorithm is recommended when the variables are 'normalized' from 0 to 1? Does SVM or logistic regression 'react' good in these types of data?

I have noticed that most of the people that are using SVM,NN or logistic regression when making feature scaling they use stardadization( (value-mean)/std ) ). Is there a reason not to rescale the values between from 0 to 1?


texts:
I'm confused,

Is it the job of the modelling approach to consider or not consider independence/dependence of features? Or should it be known before-hand?

If I know nothing about independence/dependence, but my data could have either, then how should this affect the modelling approach that I take?


texts: Classifications as long-term memory and short-term memory in LSTM

texts:
How is the data classified as long-term memory and short-term memory?
Is there some standards programmers set?


texts: Does it make sense to train multiple classifiers for multiple partitions of data?

texts: Let's say I have a dataset consisting of 100 features and a binary target variable. On exploring the data, I see that Feature 10 which is binary, seems to split the data in an 80:20 ratio with respect to the target variable. Does it then make sense to partition the data into "Feature 10 is 0" and "Feature 10 is 1" and try to build separate classifiers for both cases? Or is this what models like Random Forests do under the hood?


texts: Extracting Products Name from Unstructured text

texts:
I have unstructured text like this
Sample1
From: Gujarat, IN
To: Kerala, IN
Milk and egg, 100kg
Please handle with care.

Sample2
Hello,
(Tooth Brush and Paste)120KG in total
P/U: London, GB
Dest: QC, Canada
Keep the product away from heat
F:111-222 333
demo@gmail.com
http://www.sample123.com

Like these, I have lakhs of samples with different products
From the above 2 samples, I want to extract the product name
From sample 1 Milk,egg from sample 2 Tooth Brush,Paste.
Things I have done so far
Based on my research it seems to be a NER problem
I worked on Custom NER using Spacy to extract the product from the samples but it is not efficient, I watched some videos related to NER and also explored similar content in stack overflow but it didn't work well for my problem.
How can I approach this problem further ?


texts: Stability of value function approximation in policy gradients

texts:
In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated updates?

For example, in the REINFORCE with baseline algorithm the updates are applied to each time step in a temporally sequential order. I understand that in policy gradients the goal is to estimate the value of the policy and not necessarily the entire state space; however, in a stochastic environment and/or under a stochastic policy, not all states will be sampled with the same probability leading to overfitting to a specific trajectory meaning the value function will not be useful as a baseline for the other trajectories of the policy. Are there algorithms that shuffle the trajectory before fitting the data and/or collect batches of trajectories and then randomly sample from the batch as in DQNs?


texts: How is calculated the error with multiple output neurons in neural network?

texts:
Machine Learning books generally explains that the error calculated for a given sample $i$ is:
$e_i = y_i - \hat{y_i}$
Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:
$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$
The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).
What does change in the equations above? Since we now have multiple outputs, both $e_i$ and $y_i$ should be a vector?


texts: Are there any publicly available scan document database containing text together with ticked/unticked checkboxes?

texts:
I would like to train an optical mark recognition model (OMR) to detect and classify the ticked/unticked state of checkboxes in documents.
Does anyone know where I can have access to a publicly available scan document database containing text and ticked/unticked checkboxes?
Thanks.


texts:
The following is what I'm trying to accomplish:

I have a charity contact data set. Each contact has features such as sex, age and so on, which we define as X.

Now we are doing a solicitation campaign: we send mails to certain contacts, with a cost of 5 dollars per person, and they can respond by making a donation of however much money they wish. For the training set, we have the persons that have received the mails, and Y_amount is the amount they donated (0 if they didn't do so). People who didn't receive the mails couldn't make unsolicited donations.

Our objective is to maximize the amount raised by the campaign after deducting the cost. However, here comes the tricky part: how could I write a loss function out of this problem and implement it within python?

My original thought was to train a neural network that gives the confidence of generating net revenues from each person Y_confidence on X using the following loss function (the money we could raise if I follow the rule):

Y_decision = 1 if Y_confidence > 0.5
Y_decision = 0 if Y_confidence < 0.5
Loss = -SUM(Y_decision * (Y_amount - 5))

Do you know how to implement it or do you have a better idea of solving this problem?


texts: Python make_scorer giving incorrect outputs for Root Mean Square Logarithmic Error

texts:
I want to generate a grid search for which I need the scoring parameter based on which the search will take place. I have defined the following function to provide me a Root Mean Squared Logarithmic Error. But I feel that the scorer is considering the greater value to be a better score, whereas it should consider the lower value as better score. Please let me know if I have defined a correct scorer.
Function for RMSLE
def score_func(y_true, y_pred, **kwargs):
  y_true = np.abs(y_true)
  y_pred = np.abs(y_pred)

  return np.sqrt(mean_squared_log_error(y_true, y_pred))

scorer = make_scorer(score_func)

I had to use np.abs in the above code, otherwise it was giving an error that RMSLE cannot be used when Target has negative values.


texts: What is the difference between classification and regression?

texts:
I understand classification....a discrete response or category, like animal is dog or cat.

The author says..."Regression techniques predict continuous changes such as the change in temperature, power demand, or stock market prices."

I can't wrap my head around what he means.

Thanks.


texts:
I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions of rows and all rows are distinct. I can query this table by hive.

Now, I want to read this table in memory and transform this table to rdd so that I can use spark to process it.

Particularly, I want to transform the table in a form similar to a 3 dimensional ndarray in python's numpy: $X_{ijk}$ = 1 if person $i$ bought item $k$ on date $j$, otherwise $X_{ijk}=0$. In other words, I want an rdd such that each record is a matrix with 0s and 1s.

How can I achieve this in python?


texts: My accuracy changes throughout every epoc but the val_acc at the end of each epoc stays the same

texts:
I am training a transfer learning CNN on 161 pictures that have been augmented into 966 photos, 4 classes. I am training on a balanced data set, so there are 52 images of each class and also in the validation and test data, I am keeping the data balanced.

I am using MobileNet as a transfer learning network, and actually only using the first 8 layers as non-trainable layers, the rest are trainable and then I added another 3 dense layers with activation functions of relu and the last one softmax so as not to encounter the vanishing gradient descent.

Currently using the adam compiler, with a learning rate of 1e-6, beta_1=0.9, beta_2=0.999 decay=0.0000001 and amsgrad=True. Loss is categorical_crossentropy and metrics are accuracy.

Now, I am running 35 epochs, and with each epoch the accuracy changes from 0 to 40% (it changes every epoch, so far acc=0.4 was the highest that I saw and it doesn't always start from 0, it changes) but at the end of every single epoch the validation accuracy is 0.2 and the following epoch starts, it seems like, randomly. Why is that and how do I fix that? It's as if each epoch starts training it from the beginning. In addition, the actual loss hasn't changed much at all, it started, first batch first epoch with a loss of 1.3866 and it fluctuated max +-0.01 (not even).

Thanks in advance!


texts: Linear kernel in SVM performing much worse than RBF or Poly

texts:
When trying to train a SVM on some Kaggle data, I have encountered a situation where the linear kernel fails to give any results.

This doesn't make sense to me because the RBF kernel works just fine, and my understanding is that a linear kernel is a strictly simpler version that doesn't map the data into higher dimensions.  In fact, all my research suggests that not only is a linear kernel possible, but that in most cases, it should be faster to converge (with the trade-off that the results might not be as accurate).

However, this has not proved to be the case.  While the RBF kernel was able to produce a result during cross-validation after ~6 minutes, the linear kernel just sat there with no output after ~6 hours.  After this point, I just force quit the training and moved onto testing other SVM parameters.

A breakdown of some facts:

Roughly 60,000 examples, around 120 features (Sometimes cutting down to 30 features helps, and linear kernel will produce a result after many hours)
I'm using SKLearn GridSearchCV to perform my training
I have tested this used SVC(kernel='linear') as well as LinearSVC(), both with the same outcome
This issue has come up with multiple data sets across different competitions

My current hypothesis is that the training stops possibly because the data is not linearly separable.  I might be missing something else obvious though.  Any guidance is appreciated, thanks!


texts: Feature Scaling both training and test data

texts:
It is stated that for:
Feature Normalization -

  The test set must use identical scaling to the training set.

And the point is given that:

  Do not scale the training and test sets using different scalars: this
  could lead to random skew in the data.

Could someone explain what that means?


texts: Relative merits of different open source natural language generators

texts:
Does anyone know what (from your experience) is the best open source natural language generators (NLG) out there? What are the relative merits of each?

I'm looking to do sophisticated text summarization and would like to use theme extraction/semantic modeling in conjunction with NLG tools to create accurate, context-aware, and natural-sounding text summaries.


texts: Finding similar data columns with machine learning

texts: I'm trying to find similar data columns (text and numerical) in other tables by using classification in machine learning. What are some good features that can represent a column for my feature space ?


texts: Android: NLP library for date recognition in string

texts:
I am currently working on an android app which should make appointments automatically by reading the incoming messages from your mobile phone. I've managed to create a service which monitors the incoming messages, but now I need an Natural Language Processing algorithm in order to find the date for the appointment.

I've tried DialogFlow, but I found out it cannot be used offline and that is not the purpose of the app. It should work offline too!

Does anyone have a suggestion for a library I can use?


texts: How to apply Normalisation using the MinMaxScaler() to all Columns, but Exclude the Categorical?

texts:
Below, I have the following datatset:
sample_df.head(2)
ID     S_LENGTH     S_WIDTH     P_LENGTH     P_WIDTH     SPECIES
-------------------------------------------------------------------
1      3.5          2.5          5.6         1.7        VIRGINICA
2      4.5          5.6          3.4         8.7         SETOSA

Therefore, how to I apply normalisation to this dataset using the following code below to all my columns, excluding the ID and SPECIES columns?
I basically want to use the preprocessing.MinMaxScaler() to apply normalisation, so that all the features are in a range of 0 and 1.
This is the code I am using...
min_max = preprocessing.MinMaxScaler()
min_max.fit_transform(sample_df)

...but when I execute it, I get this error:
ValueError: could not convert string to float: 'SETOSA'

Alternatively, if I do this...
min_max = preprocessing.MinMaxScaler()
min_max.fit_transform(sample_df[['S_LENGTH', 'S_WIDTH']])

sample_df.head(2)

...I get this error:
AttributeError: 'numpy.ndarray' object has no attribute 'sample'

Any help on how to accomplish what I want to do is much appreciated!


texts: How can i decide on which features to use for clustering?

texts:
I am clustering on a dataset where each row is a customer and each column is a feature. I have 200 features, this seems like alot for clustering. I plan to experiment with a variety of clustering models e.g. k-means since all my data is numerical. H
How can i reduce/select the features ?
I am only familiar with SelectKbest, etc as these are used for predictive modelling. But here i have NO target variable. Note i plan to use python.


texts: Cannot create new percent columns from CSV file

texts:
I am fairly new to coding and cannot figure out how to create the AB_perc and CD_perc columns in the Desired Output table below.

The original csv file I am using has all sorts ot DNA sequencing data, so below is a simplified example of what I am trying to do.

CSV file:

Desired Output:

I was able to generate the AB_sum and CD_sum columns by using:

df['AB_sum'] = df['A'] + df['B']
df['CD_sum'] = df['C'] + df['D']
df.groupby('RunNumber', as_index=False).agg({'AB_sum': 'sum', 'Confusion C<->G': 'sum'})

But I cannot figure out how to create the AB_perc and CD_perc columns.

For example:

To calculate the percentage of AB for Run 1 (AB_perc: 79.94), the AB_sum value, 570, is divided by 713, the sum of all orange numbers in the csv file, and then multiplied by 100

I tried several things all day and nothing worked for me. I'm sure there's a simple solution, but I feel like if I try any harder my nose will start to bleed. Haha.

Any help would be much appreciated!


texts: How do SOMs make visualization of high-dimension data possible?

texts:
Self Organizing Map is credited to be a very effective tool for exploratory data analysis as it aids in visualization of high-dimensional data.

However, the part where the dimensionality reduction is happening is not clear to me. From this article, I understand that the information is essentially embedded in the weight vectors of neurons which are of the same dimensions as of the original patterns. The same weight vectors are supposed to be used for visualization (if I follow from the highly popular colors example).

If this is true, I do not understand how SOM is performing dimensionality reduction and how it is making visualization of high-dimensional data simpler.

Could someone explain (if possible with an example of >3 dimensional data) how SOM performs dimension reduction and makes visualization possible?


texts:
I am a newbie to python.

I have uploaded about 2000 images into my google drive, assigned a variable to all content in the folder and tried to split the folder's content into training and testing, but it kept bringing error. Can anyone direct me on how to get to splitting the data stage?


texts: Correct way of computing dice score for image segmentation?

texts: In binary image segmentation, for given a set of images, it's true mask and predicted mask. How do you compute dice score? Should I compute the dice score for each image separately and then find mean across all images? Or compute the dice score for all images at once by flattening tensor? Which is the correct way?


texts:
Let's say I want to classify if the employee will churn or not. In my random forest, I have 6 estimators where 3 of them predict the employee to churn and the other estimators predict the employee to retain. In short, 3 estimators = predicted to churn and 3 estimators = predicted to retain.

I know that the random forest works by getting the most votes for each class or by getting the mode.

The question is how does the random forest predict in this case?


texts:
I'm trying to implement the One Hidden Layer Model presented in this article using Keras.

This is my code:

from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Activation
from keras import optimizers

model = Sequential()
model.add(Dense(100, input_dim=9216))
model.add(Activation('relu'))
model.add(Dense(30))
model.compile(loss='mean_squared_error', optimizer=sgd, metrics=['accuracy'])

sgd = optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)
hist = model.fit(X_train, y_train, epochs=10, verbose=0, validation_split=0.2)

y_pred = model.predict_classes(X_valid)

  X_train shape is (2140, 9216)

  y_train shape is (2140, 30)

  X_valid shape is (1783, 9216)

But y_valid shape is (1783,). I'm trying to understand why I'm not getting a (1783, 30) output. Am I missing something?


texts: XGBRegressor hyperparameter optimization using xgb cv function

texts:
I am trying to optimize hyper parameters of XGBRegressor using xgb's cv function and bayesian optimization (using hyperopt package). Here is the piece of code I am using for the cv part.

dtrain = xgb.DMatrix(X_train, label=y_train)
cv_results = xgb.cv(params,dtrain,num_boost_round = 1000, folds= cv_folds,
                        stratified = False, early_stopping_rounds = 100, metrics="rmse", seed = 44)

However, I am getting the following error within the xgb.cv function (part of the Trace):

414     cvfolds = mknfold(dtrain, nfold, params, seed, metrics, fpreproc,
--> 415                       stratified, folds, shuffle)
    416
    417     # setup callbacks

/anaconda3/envs/py36/lib/python3.6/site-packages/xgboost/training.py in mknfold(dall, nfold, param, seed, evals, fpreproc, stratified, folds, shuffle)
    261         except TypeError:
    262             # Custom stratification using Sklearn KFoldSplit object
--> 263             splits = list(folds.split(X=dall.get_label(), y=dall.get_label()))
    264             in_idset = [x[0] for x in splits]
    265             out_idset = [x[1] for x in splits]

AttributeError: 'int' object has no attribute 'split'

I can't figure out why am i getting this error. The documentation for xgboost is also not very clear and sparse.
So any help would be greatly appreciated.

Thanks


texts: How to prevent vanishing gradient or exploding gradient?

texts: Whats causing the vanishing gradient or exploding gradient to occur, and what are the measures to be taken to prevent it?


texts: Class Imbalence Problem even after Balancing Data

texts:
So I am training a neural network on a binary classification problem and my Case (1) and Controls (0) were imbalanced so I oversampled my cases so that that the training set was 0.5053 made up of controls. I did not balance my test set which was 0.562 controls. In the beginning my train and test accuracy rises (it is not very accurate still but I expect this to be the case) but then the training accuracy steeply drops while the test accuracy plateaus.

They end up with accuracies of 0.5053 and 0.562 respectively so the network is just classifying everything the same. I do not understand how this behavior arises as I thought that balancing my training set would avoid the problem of classifying everything the same? Also, the training set begins to learn initially upwards from 50/50 but I cannot understand its reversion. Is there anything I can do to prevent this? Or should I just employ early stopping when the training accuracy begins to decrease?

Any insight would be appreciated!

opt = tf.keras.optimizers.SGD(lr=0.000001, momentum=0.9, decay=0, nesterov=True)

model = keras.Sequential([keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu,kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(1, activation=tf.nn.sigmoid)])

model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])


texts: Interactive Filter based on logic to catrgorically color in lines on a bokeh graph?

texts: I'm looking to make some bokeh line charts for a personal finance analysis, where the lines show the growth of money over time, with each line representing different combinations of life choices. I want to make an interactive filter where a person looking at the graph can select buttons that will show the split between different scenarios, and will highlight the lines in different colors to differentiate those scenarios (i.e check number of kids, lines are highlighted different colors which correspond to number of kids the person had in that scenario). How can this be done using Bokeh?


texts: How to generate bootstrapping samples in R?

texts:
Here is my original dataset from a voting activity. Each participant voted for one option (A,B,C,D,E,F) listed below. The numerical value is the total number of participants who voted for each option.

I want to bootstrap the voting for 1000 times (sample with replacement) and make a comparison between the pre-event and post-event voting for each category using independent sample t-test.

However, I don't know how to start with generating the 1000 bootstrapping samples...I have the "Boot" packages installed on R. Could anyone give me a hint what kind of steps I should take in order to generate 1000 bootstrap samples based on the original data I provide above?


texts: Neural network example not working with sigmoid activation function

texts:
I'm running the Neural Network example written in in BogoToBogo

The program worked fine:

(array([0, 0]), array([  2.55772644e-08]))
(array([0, 1]), array([ 0.99649732]))
(array([1, 0]), array([ 0.99677086]))
(array([1, 1]), array([-0.00028738]))

The neural network learned XOR, using tanh as activation function by default. However, after I changed the activation function to "sigmoid"

nn = NeuralNetwork([2,2,1], 'sigmoid')

Now the program outputs:

epochs: 0
...
epochs: 90000
(array([0, 0]), array([ 0.45784467]))
(array([0, 1]), array([ 0.48245772]))
(array([1, 0]), array([ 0.47365194]))
(array([1, 1]), array([ 0.48966856]))

The output for the 4 inputs are all near 0.5. The result shows that neural network (with the sigmoid function) didn't learn XOR.

I was expecting the program would output:

~0 for (0, 0) and (1, 1)
~1 for (0, 1) and (1, 0)

Can somebody explain why this example with sigmoid doesn't work with XOR?


texts:
In support vector machines, I understand it would be computationally prohibitive to calculate a basis function at every point in the data set. However, it is possible to find this optimal solution due to the so-called kernel trick.

Other answers to this question use advanced math and statistics jargon to answer the question  (I assume) properly, causing it to be inaccessible to general a data science audience.  Could someone post a "big-picture" description  (i.e., not necessarily thorough or technically complete) illustrating what the kernel trick is and how it works?


texts: How can I compare classes from clusterings performed on two different data sets?

texts: I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets together into a single, bigger data set, and cluster that, I see that most of, say, class 1 of the first data set and class 2 of the second (numbers are arbitrary) fall into the same class of the total clustering, which makes sense, meaning they're probably equivalent. What would be a good rigorous way of performing this kind of analysis and giving a quantitative measure of this overlap? Thanks.


texts: Python giving memory error while fitting the text into vector using various vectorizer

texts: While training the IMDB movie review dataset for sentiment analysis, the model will give a memory error if I set the features to more than 20,000. Is there any way to pass the memory error?


texts: Reducing the Number of Training Samples for collaborative filtering recommender systems

texts:
I have the following problem: I am doing some research on the accuracy of recommender algorithms that are mostly used nowadays.

So, one way to measure their performance is by checking how well they predict a certain value under different sizes of a given dataset, meaning, sparsity in a ratings matrix.

I need to find a way to calculate the root mean square error(or mae), some metric, versus the sparsity in the dataset. As an example, let’s have a look at the picture below:

You can see that it says:

  “RMSE as a function of sparsity. 5000 ratings were removed from the training set(initially containing 80000 ratings) in every iteration. “

I’m using Python and the Movielens dataset. Do you know how can I achieve this in the mentioned language? Is there any tool to do that?


texts: I have two models that predict a person's activity (seating, walking, taking stairs, and sleeping) based on a person's motion and the video. Model 1 is trained on a video dataset while Model 2 is trained on a motion dataset. Each model can independently predict a person's activity. However, I would like to make the final prediction based on the consensus of the two models. For example, if Model 1 predict a walk class and Model 2predicts a walk then the final prediction would be walk. However, if any of the two models disagree, the final prediction would be unknown. Obviously, this can be achieved with simple if statements but I was wondering if there is any out of the box way to do this in python or if there is a known best approach to do this.


texts: Binary Classification - One Hot Encoding preventing me using Test Set

texts:
I have a preprocessing pipeline that includes replacing missing values and onehotencoding for the categorical variables.

When I try to use my model on the test set, it explains that the number of columns it expects differs. This is due to one hot encoding

One option I considered was passing the full dataset into the pipeline and then seperating into test and split. However, this causes data leakage as the missing values it capturing values from the testset.

Please let me know how to prevent this.

Thanks,


texts:
I am looking for any straightforward solution to work with Jupyter Notebooks and git since the cached data on the .ipynb files makes the diff difficult to read.

Does anyone know any standalone alternative and not server hosted?


texts: How can I see a long string in my dataframe?

texts: I have a column in my dataframe in which there are sentences which are too long. I want to see them as a whole but every time I perform even a simple iloc operation i get output like 'i am going to...'. How can I remove the ... and see the whole sentence ?


texts: algorithm to predict numeric values with sklearn

texts: I'm new to ML and trying to learn it. I scraped information (using python) from a website where people try to sell their cars. I put data in a pandas data frame. Now I'm wondering how should I predict car's price according to it's manufacturing year and its mileage! I drew some plots and there is a correlation between price-year and price-mileage!


texts: ML Algorithm for anomaly detection in paired time-series

texts:
I have many (around 40) separate time-series from different sensors, each measuring magnetic field intensity. I am looking to get an ML algorithm to identify a particular anomaly. This anomaly happens infrequently (a handful of times in 25 years of 60-second data, depending on the sensor) and is only apparent when you look at two time-series side-by-side:

The yellow circle is the anomaly, and can be described roughly like this: when the time series are particularly disturbed, the red time-series will briefly anti-correlate with the green time-series over a period of a few hours (data are sampled every 60s). This can easily be done by hand for a pair of time-series.

The Plan

I'd like to train a machine learning algorithm by manually identifying these anomalies for a sensor pair, and then set the trained algorithm on the rest of the pairs of time-series

The Problem

I have no idea what the best algorithm would be for this problem, as it involves two time-series. I messed around with gathering information in 6 hour blocks and treating it like a non-time series problem. This involved treating each 6-hour block separately (i.e., each with std, average, max, min of each time-series as well as a correlation coeff and a 1 or 0 output determined manually), but I feel like there is a neural network that will be able to better figure out the important variables.

The Question

What would be the best approach to this problem in order to train a ML algo? Is my approach above suitable?


texts:
I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it.
I tried to read the original paper, which contains so little details that it feels like it's been written explicitely not to be understood. I also read several blog posts about it but none really clarified it (one straight up admitted not undertanding it completely).
Can somebody explain it to me? I am familiar with the EM algorithm but I cannot see how it related to the loss function in order to find the subwords probabilities...


texts: With automated hyperparameter tuning available, do we still need to learn hyperparameter tuning

texts: Tools like AWS Sagemaker have capability to do automated hyperparameter tuning, even with complex algos like Neural Networks using Tensorflow. So do we still need to learn how to do hyperparameter tuning, or simply leave it to tools like Sagemaker? Thx


texts: How much does each tree of gradient boosting contribute to the global feature importance?

texts:
Let's say we are training a GBDT in the Titanic dataset. We have 3 trees in the GBDT.
You extract the first tree and calculate the feature importance (no matter if cover, gain...), and Age importance =  0.8
You go on and do the second, and you get Age importance = 0.2
And for the third Age importance = 0.5
What will be the global feature importance for Age?
0.8 + 0.2 + 0.5 = 1.5

Or weighted by a learning rate(let's say its 0.1)
0.8 + 0.1*0.2 * 0.1 *0.5 = 0.801

Or any other way.


texts: Using GANs to generate synthetic tabular data to improve supervised learning

texts:
One topic I see some people trying is using GANs to generate synthetic tabular data for supervised learning. Also as a way to oversample the minority class in a binary classification.
For me creating synthetic data is a bit dangerous.
In practice, all the experiments that I have seen to generate new training data using GANs have failed.
Is there any theoretical reason behind?


texts: Collaborative filtering when multiple items are rated multiple times by same user

texts:
When trying to model as a recommendation problem the selection of an item that can be selected (and rated) by the same user many times, I can't find references of previous work.

For example User1 can select item A (and rate it) then item A again (and give it a different rate) and then again item A can be selected n times.. then the same for item B, but item A could be again selected at any time point, and that could happen for any of the A, B, C... Z items. Could this be modelled somehow as a recommender system?

When asked in SO I was told that can't be modelled even as a multi-modal recommneder, then I think my question should be extended to how could I model this for prediction of next rate?

SO question


texts: Is there a way to calculate value from a radar (spider) chart?

texts:
So I am currently working on a project for "sales people". Essentially, the user is giving us a plethora of information related to deals they are trying to win. We have data related to the price of the deal, margin of the deal, number of people working on the deal, caliber of the people working on the deal, and the risk of the deal. I want to assign a "score" to these deals based on the given metrics and visualize how the score is calculated in a simple way.

It would be very easy to visualize this all as a radar (spider) chart (I'm aware that a traditional radar chart has limited power since the ordering of metrics is arbitrary and the area formed by the spider chart doesn't hold much value) - but why can't I take the results from a correlation matrix or regression output to weight the metrics in some way and/or oder them based on importance?

An example - let's say I've built a radar chart for NBA players based on Points per game, assists per game, player efficiency, etc. If I had a logistic regression output to determine the MVP, I would use the output to order and weight the variables (based on p-values?) to build the radar chart. If I then took statistics from the most recent NBA season and plotted each NBA player, wouldn't the player with the biggest area be one of the most valuable?

I can't seem to find much information on building weighted radar charts - although I have seen that percentiles can be used for the actual metrics. Does anybody have any advice or alternative solutions?


texts: Why large weights are prohibited in neural networks?

texts: Why  weights with large values cause neural networks to be overfitted, and consequently we use approaches like regularization to neutralize weights with large values?


texts: shape of theano tensor variable out of keras Conv2D

texts:
Being new to theano, pls bear with me. I thought the shape of the tensor variable is already well defined out of the Conv2D layer since the input is specified, as follow,

from keras.layers import Input, Convolution2D
import theano
input_img = Input(shape=(1, 28, 28))
x = Convolution2D(16, 3, 3, activation='relu', border_mode='same')   (input_img)
print type(x)
print theano.tensor.shape(x)

But the output is,

<class 'theano.tensor.var.TensorVariable'>
Shape.0

Since I'm taking the default stride of 1, and same border mode here means that padding is added so the output is the same as input. Using this information I could calculate by hand what the output shape should be.

Did I miss something here? The question is how to get the shape of the output of a convolution layer?


texts: Does batch size matter in inferencing speed?

texts:
I am reading the paper "Are Sixteen Heads Really Better than One?" and in section 4.3 it states that the inference speed varies with batch size.

How does batch size affect inference speed for this machine learning model?


texts: Difference between a log scale and linear scale : np.random.rand()

texts:
How and why will a linear np.random.rand()  (to generate a linear scale between 0.0001 and 1) not result in better distributed result but np.power(10,-4 * np.random.rand()) (log scale) will.
I am just accepting the below explanation as it is, But I can't understand the reason behind it.(simple math, i believe, which i am missing. Kindly help with some example)

Reference to my query:
Using an Appropriate Scale to Pick Hyperparameters

  To understand this, consider the number of hidden units
  hyperparameter. The range we are interested in is from 50 to 100. We
  can use a grid which contains values between 50 and 100 and use that
  to find the best value:

  Now consider the learning rate with a range between 0.0001 and 1. If
  we draw a number line with these extreme values and sample the values
  uniformly at random, around 90% of the values will fall between 0.1 to
  1. In other words, we are using 90% resources to search between 0.1 to 1, and only 10% to search between 0.0001 to 0.1. This does not look
  correct! Instead, we can use a log scale to choose the values:

This query is from Andrew Ng's Specialization Course on Deep Learning - Coursera
Course 2 Week 3


texts: Explanation behind the calculation of training loss in deep learning model

texts:
I am trying to model an image classification problem using convolution neural network. I came across a code on Github in which I am not able to understand the meaning of following line for loss calculation in the training loop.
I am omitting most of the detail and only placing the relevent code-

for batch_idx, (data, target) in enumerate(final_train_loader):
     loss = criterion(output,target)
     #Idea behind the below line
     train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

Cross-entropy loss function is being used here.


texts: Benchmarking unsupervised learning by second stage classifiers?

texts:
Suppose we have a dataset $\{x^{(i)}, y^{(i)}\}_{i = 1}^N$ where $x^{(i)} \in \mathbb{R}^n$ and $y^{(i)} \in \{0, 1\}$ for simplicity.

Our main goal is to apply some unsupervised learning algorithm on $x^{(i)}$ and interpret the results, which we can call $u^{(i)}$.  I have in mind applications where the unsupervised algorithm is designed to infer something intangible and unobservable, but nonetheless having a meaningful physical interpretation, ICA as a means of recovering independent sources for example.

Benchmarking the unsupervised algorithm algorithm in a meaningful way is difficult since we have no way to compare $u^{(i)}$ to any ground truth.  My idea to add another perspective to this problem is to train a supervised classifier on the dataset $\{u^{(i)}, y^{(i)}\}_{i = 1}^N$, i.e. just use the results of the unsupervised algorithm as features.  If a classifier on this derived dataset performs well, it would provide some evidence that the results $u^{(i)}$ of the unsupervised algorithm is actually finding some meaningful structure in each $x^{(i)}$.  If $u^{(i)}$ was basically just spurious, it wouldn't be possible to train this second classifier.

Does this sound like a reasonable means of comparison?  Is anyone aware of existing work that benchmarks unsupervised algorithms in this way?

Putting the question more generally, by what means can we try to evaluate whether or not $u^{(i)}$ is providing useful or interpretable information?  As opposed to just being some artifically constructed statistic, or something spurious.


texts: Reformer the Efficient Transformer and Image Size Limits

texts:
I'm currently trying to use the Reformer: Image Generator with my own dataset. The colab notebook for the model is here:
https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/image_generation.ipynb#scrollTo=qBvuw2h85WXE
From what I read about the Reformer here: https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html
It has a context window of up to 1,000,000 words. My question is, how does this apply to images? Would it be that the Reformer can retain context of images no larger than 1,000,000 pixels? If that is the case, do I need to reduce the size of my images? Currently they are at 1920 x 1080 pixels. Or would there be a way to preserve the detailed images and use them in the current context?


texts: What model to use for relative comparison between 3 figures?

texts:
I am working on a problem where I am given three images of different dishes (A,B,C) and the task is to figure out if figure B or figure C is closest to figure A (in terms of taste). I have a training set with ordered images (A,P,N) where P (positive) is closest to A (anchor) and N (negative). Now I figured that I somehow need to come up with an embedding that is representative of the taste, and then use a metric d(.,.) that meaningfully compares the dishes. Ideally d(embedding(A),embedding(B)) < d(embedding(A),embedding(C)) implies that dish A is closer in taste to dish B than it is to dish C.
Now I did some research and found the so-called TripletLoss, which seems to tackle a very similar problem. A nice blog post by Olivier Moindrot can be found here on the topic, and it includes the figure below.

The problem is that contrary to the implementations I found online, my dataset is not labelled. I simply have ordered triplets from which I need to extract information for the right embedding. The dishes in my dataset are all unique, and cannot be grouped meaningfully into discrete classes. Yet, I still want to come up with something like a metric that is able to compare the dishes.
My question: Am I going in the right direction? Are there any models that would suit my problem better? My knowledge of ML/NNs is still somewhat limited so any suggestion into the right direction is appreciated!


texts: What are some good resources for setting up and debugging neural networks?

texts: I am aware of Troubleshooting Deep Neural Networks by Josh Tobin and A Recipe for Training Neural Networks by Andrej Karpathy, but I am interested in other resources that can give me some guidelines or steps to setting up and debugging neural networks.


texts: Am I overfitting my random forest model (more information in description)?

texts:
First off, sorry if this a novice question! Relatively new to all this stuff. Posted this in Stack Overflow and someone sent me here! Hope it's the right place.

Anyway, I'm working with 22 datasets that each have 180 observations of "Oddball" data and 720 observations of "Standard" data. I'm trying to use random forests for classification (i.e., oddball=1, standard=0). I understand there should be approximately equal trials/observations for both factors, but if I use 75% of the oddball data, then I'm barely using over 18% of the standard data. These data are pretty variable, and I think this could be problematic.

If I make four models, still using the same training data for each, am I overfitting my model? There's a lot more I've written, but this is basically what I'm trying to do:

jj = sample(1:180,(180*75),replace = F #Take 75% of all oddball data
kk = sample(181:900,(720*.75),replace = F) #Take 75% of all standard data
jj = sample(jj); kk = sample(kk) #Mix them up
kk = matrix(kk,4) #Divide the standard data so there are 4 sets of equal numbers for jj

samp1 = c(jj,kk[1,])
samp2 = c(jj,kk[2,])
samp3 = c(jj,kk[3,])
samp4 = c(jj,kk[4,])

I would then make four models (all while not touching the out-of-sample data) using each of these sample sets and average all their predictions to give me a "master" probability (i.e., an average of .8 would be deemed an oddball).

Is this overfitting the data? Is it even possible to overfit the data when using random forests? Is there something wrong with this intuition?

Thank you for anyone that helps! Your time and expertise is much appreciated.


texts: How to custom operations in the forward function in pytorch

texts:
I am doing an experiment on images of small size as follows:
The input image has only one channel, then I apply 2 customed filters (denoted by A and B) to do the convolution with the image. The filter has the same size as the image and each of A and B have 10 channels respectively. So after the convolution we have two 1 * 10 tensors. Denote them as (a1, a2, ... a10) and (b1, b2, ... b10). Let w = (w1, w2, ..., w10) be a trainable 1 *10 tensor. Then I want to compute the following fraction: (the numerator is the norm of weighted sum of vectors (ai, bi) and the denominator is the sum of weighted norm. The norm here is the ordinary Euclidean norm.)

I am wondering how to do this computation in the forward function and where I should claim w as trainable tensor? The template of my code is as follows:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.filter_A = ...
        self.filter_B = ...

    def forward(self, x):
        x_A = F.conv2d(x, self.filter_A)
        x_B = F.conv2d(x, self.filter_B)
        ...


texts: What is the theoretical differences of Multitask learning vs Fine tuning based transfer learning?

texts:
Suppose, I have the following scenarios.

I have a bunch of fruits, i.e., apple, orange, and banana. I simply made a Multitask model, where my network first tell me which fruit it is, and then telling me the color of it. Suppose, if I give my network an apple, it tells me, (a) it is apple, (b) it is red. By doing some theoretical study, I have understood that it is one type of inductive transfer learning (TL) (correct me, if I am wrong). So here, the network is learning 2 task simultaneously.

I have a bunch of objects, i.e., cube, ball, and triangle. Here also I want my network will do the same thing like scenario 1. So it will tell me, (a) whether it is a cube or not, and (b) then tell me the color. So what I did is, I transferred the learned weight and parameters from the network of scenario 1 to this scenario. Thus I performed the fine-tuning based TL in here.

So , from theoretical point of view, I have few confusions. I need to clarity my idea, and need some ideas from experts.

If I consider the scenario 2, by definition of fine-tuning based TL, the task of scenario 1 (apple, and red) is my source task, and the task of scenario 2 (cube, and red) is the target task. From my understanding, I think that every inductive TL approach has a source task and target task. So, for scenario 2, thus it satisfies my understanding.

[REAL QUESTIONS]
2. Now the confusion starts for my theoretical understanding. For scenario 1, it has also 2 tasks - (a) identify the fruit, (b) identify the color. So here, what will be my source task, and what will be my target task. For clarifying my theoretical description or explaining my thinking into words, I need to know this.
3. As I am doing 2 TL tasks here, how to define the whole scenario ?


texts: Split time series by python or by keras?

texts:
In Python you can use TimeSeriessplit() to split a time series properly for training but you can also do the same(?) in Keras by TimeseriesGenerator.
Which one is recommendable? And/or what are certain differences?


texts: Why KL Divergence instead of Cross-entropy in VAE

texts: I understand how KL divergence provides us with a measure of how one probability distribution is different from a second, reference probability distribution. But  why are they particularly used (instead of cross-entropy) in VAE (which is generative)?


texts: How to draw convolutional neural network diagrams?

texts:
I have to draw a CNN diagram similar to this:

I tried all the tools mentioned in https://datascience.stackexchange.com/a/14900, but there is no easy way to do it.
Is there any automated way to do it? Or do I have to do it manually.

In addition, is it possible to draw this?:

I found http://alexlenail.me/NN-SVG/LeNet.html, however, in this the input to each layer is assumed to be a perfect square. Thus, I cannot draw above diagram.


texts: How to add control variable in regression using sklearn

texts:
I am trying to perform controlled regression using sklearn, I have been using sklearn for fitting dependent variable and independent variable, however, if there is a variable that I want to control for while fitting how do I do that in Python? Here is R implementation for the same.

fit <- lm(Y ~ X + C, data = ...)
summary(fit)

where Y is the dependent variable, X is the independent variable and C is the control variable.


texts: Very infrequent values in embedding layers

texts:
I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1.
I am using an embedding layer for this input and the result is not too bad.
Does such small frequency cause any instability or troubles for the neural network model?


texts: Computational aspects are typically ignored by statisticians

texts:
In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that :

  Although data science can be seen as a continuation of statistics, the
  majority of statisticians did not contribute much to recent progress
  in data science. Most statisticians focused on theoretical results
  rather than real-world analysis problems. The computational aspects,
  which are critical for larger data sets, are typically ignored by
  statisticians. The focus is on generative modeling rather than
  prediction and dealing with practical challenges related to data
  quality and size.

The bold phrase is not clear to me. In fact, since a generative model is the model that generates the data, once we obtain it we can do predictions. So, to me generative modeling and prediction are not opposing concepts. What do you think ?


texts: Is it possible to run logistic regression on data with a few longitudinal parameters?

texts: Currently, I am building a credit rating model based on logistic regression and faced a problem with inserting panel variables in it. Is it possible to do so in logistic regression and if it is, what packages and functions can deal with it?


texts: Counting the occurrence of each string in a pandas dataframe column

texts:
I'm working with a data set of movies which has various info on them. One of the columns contains the various genres a movie may belong to like so:

What I would like to do is count how often a genre occurs in each column, in above example a corresponding series would look like (created the series myself):

How can I extract this information from the original dataframe using pandas?


texts: How to convert a non gaussian distribution into a gaussian destribution?

texts:
Suppose I have a dataset inwhich there are few dimensions that distribution over them is non gaussian and this means, skewness is nonzero (possitive or negative). This is caused by some outliers in my data, which are not possible to be excluded manually.

Now my question is, How is it possible to convert this dataset into a gaussian (zero skewness) so that a learning procedure could be run on it without being biased to outliers? How should one behave this kind of datasets?