texts: Conditionally replace values in column with value from another column

texts:
I have a 5 column dataframe where one column has some NaN's. My goal is to replace these NaN's with the corresponding value in another column.

So if my DF looks like this:

YearBuilt     LastModified

1993          2004

1994          NaN

1995          2004

I would want to replace the NaN with the corresponding YearBuilt, which is 1994 in this case.


texts: How to combine different models in Keras?

texts:
I have a pre-trained network, consist of two parts, the feature extraction, and the similarity learning.

The network takes two inputs and predicts the images are same or not.

The feature extraction part was VGGNet 16 with all layers freezed.

I only extracted the feature vectors and learned the similarity network which consists of the two convolutional layers followed by four dense layers.

Note: Removed last layers from the image due to large size.

Now, I want to fine-tune the last convolutional block of VGGNet and want to use two different VGGNet feature extractors for each type of image.

I have loaded the trained model and created a new model which starts from the Merged_feature_map layer:

model = Sequential()
for layer in ft_model.layers[3:]:
    model.add(layer)

Now, the new model will only contain similarity network without the feature extraction part.

I have loaded two VGGNets for each type of image and unfreeze their last convolutional block as:

vgg_left = VGG16(weights="imagenet", include_top=False, input_tensor=Input(shape=(img_width, img_height, channels)))
vgg_left.name = "vgg_left"

vgg_right = VGG16(weights="imagenet", include_top=False, input_tensor=Input(shape=(img_width, img_height, channels)))
vgg_right.name = "vgg_right"

At that moment, I have 3 models and I want to combine them. The output from both VGG networks should be the input of Merged feature map. How to combine them and make them a single model.

bottleneck_features_r = vgg_left(left_input)
bottleneck_features_s = vgg_right(right_input)

It should be like:


texts: Python Sklearn TfidfVectorizer Feature not matching; delete?

texts:
I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.

The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error:

ValueError: X has 4877 features per sample; expecting 2799

Is there a way to delete any features in x_test that were not used in x_train?

I know if I had used a countverctorizer, i could have bypassed the error by not using fit_transform on x_test. But since it is TfidfVectorizer, it won't let me bypass.

I also tried imputation but couldn't get it to work.

Thanks


texts: Classifier that optimizes performance on only a subset of the data?

texts:
I'm working on machine learning problem where I'm only interested in getting high accuracy within a narrow band of my predicted likelihoods. Specifically, I want an algorithm that will score very accurately when it predicts a likelihood above a specific threshold.

A motivating example:

I'm presented with 1 million closed boxes, 10 percent of which contain a gold coin. The likelihood of a box containing a gold coin is associated with physical characteristics of the boxes. I'm allowed to open 500K boxes, chosen at random, to see which contain the coins.  After this, I'm allowed to choose 100 of the remaining 500K boxes and keep any gold that I find.

I'm looking to build a classifier to solve this problem.  Specifically, I'm looking for the correct way to incorporate the constraint that I am only able to open 100 boxes in the test sample. Put another way, my measure of performance is limited to a subsample of my test data, so I need a classifier that is highly accurate when it predicts the highest likelihoods. Prediction errors outside of these 100 boxes are of zero concern.

I'm aware of cost-weighted loss functions that penalize false positives/negatives differently, but I'm not familiar with any that solve this specific problem of optimizing for a localized accuracy.

Specifically, I'm trying to answer two questions:
1) Are there specific classification algorithms well suited for this type of problem?
2) Can I modify the loss function used on my training set to produce a high localized accuracy within these specific 100 boxes?


texts: Validation data shall be in broken down into batches or not?

texts:
I am using fit_generator to train the model. The training dataset is being read from a generator function which gives data in a constant batch size. Now I want to know what approach shall I adopt for validation data. Shall I make a generator for the validation set or load it completely into memory and use it in fit_generator?

Note: Validation dataset is fitting into memory quite easily therefore batching is an option for me.


texts: Using Self Organising Maps For Feature Selection

texts:
I have a time series of indicators for a single stock. Some of the features are in repeated sets e.g. simple moving average 5D, 10D , 20D, 100D.
I only want to use one feature from each set. Is there a way for SOMs to be used to select the most important feature out of a set of similar features?
My initial thought was to create a SOM with only the 4 features we are comparing, but I am not sure if this is right or how to interpret the map.


texts: Normalize data with extreme outliers for forecasting

texts: Suppose I have input values that represent the change of a stock share from each time step to the next. Now I want to feed these values into an LSTM Neural Net. My problem is that most values are between -0.000001 and 0.000001, but some values go up to 0.1 or -0.1. How can I effectively scale such a dataset. I've tried applying a log function (to the value +1, so e.g. log(1.000001)) and scaling between 0 and 1 and I've tried capping values at around 1.5. Do you have any suggestions how to deal with such extreme outliers better?


texts: Data normalization of count data for neural networks

texts:
I have a sparse matrix of count data that I'm using as input to a neural network.

I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But for features that are counts, what is a good approach? Should I $\log_2(x+1)$ transform the data and then do a $z$-score standardization? Is there another better approach?


texts: Meta Learning: how to train a model with Support Set and Query Set

texts:
I've just started to learn Meta Learning reading the book Hands-On Meta Learning with Python.

I think I know the answer for my question, but I'm a little confuse about how to implement the algorithm with Keras.

This piece of code is from an example that uses U-NET:

from sklearn.model_selection import train_test_split

# Split train and valid
X_train, X_valid, y_train, y_valid = train_test_split(train_data, test_data, test_size=0.1, random_state=42)

results = model.fit(X_train, y_train, batch_size=32, epochs=50,\
                    validation_data=(X_valid, y_valid))

My problem is with the fit method.

Reading the book I read:

  ...train on the support set and test on the query set...

So here, in a meta learning model, the train_data is the Support Set, and the test_data is the query set, isn't it?


texts: How to create a confusion matrix for one node of a decision tree?

texts:
I am doing past papers for my data science exam and was curious about one of the questions. They ask us to create a confusion matrix by hand for one node of a decision tree.
I understand how to create a decision tree for an entire model, but I am unsure on how to create one for just one variable. Should the entire first row be 0s other than the class A as it should always predict class A or am I missing something.
I have attached the decision tree and the exact wording of the question below. Thanks!
"(iv) Present a confusion matrix for the [leftmost terminal] node."


texts: passing a dataframe as a argument while applying a lambda function on a dataframe

texts:
I am applying a lambda function on a data frame and would like to add a new column to that dataframe, while applying the lambda function I am passing the data frame it self as a argument to the function.
My code

def categorical_fare_mean(data,col,cat):
    print('entered once')
    data = data[data[col]==cat]['Fare']
    fare_mean = data['Fare'].mean()
    return fare_mean

dataset['weekday_encoded'] = dataset.apply(lambda x: categorical_fare_mean(dataset,x['weekday'],0) ,axis = 1)

This is giving me an error KeyError: (0, 'occurred at index 0') I am not sure where I am going wrong. Can some one help me with it.

Thanks


texts: Possible flaw in the MDS method for dimensionality reduction

texts:
The MDS (multidimensional scaling) method is used to solve the problem of dimensionality reduction. Basically, it does the following: given $n$ points $x_1,\cdots,x_n\in\mathbb R^d$, try to find a smaller $d'$ and points $y_1,\cdots,y_n\in\mathbb R^{d'}$ such that
$$|x_i-x_j|=|y_i-y_j|\quad\quad\quad(1)$$

The reasoning goes like this: Let $D=(d_{ij})$, where $d_{ij}=|x_i-x_j|$, be the distance matrix of $\{x_i\}$. If a set of points $\{y_i\}$ is such that

$\{y_i\}$ are centered at $0$, i.e. $\sum_{i=1}^ny_i=0$;
$\{y_i\}$ satisfies the condition (1) above,

then the matrix $Y$, whose columns are $\{y_i\}$, satisfies $Y^TY=B=(b_{ij})$, where $b_{ij}$ can be determined from $D=(d_{ij})$. In other words, we only need to solve for $Y$ the matrix equation $B=Y^TY$, where $B$ can be considered known.

After this, most textbooks and online material would say: "we see that if $B=V\Lambda V^T$ is an eigenvalue decomposition of $B$, then $\Lambda^{1/2}V^T$ is a solution for $Y^TY=B$, and we can take the columns of $\Lambda^{1/2}V^T$ for our choice of $\{y_i\}$.

Questions:

How can we guarantee that $\sum_{i=1}^ny_i=0$ if we take $Y=\Lambda^{1/2}V^T$?
What if $B$ has negative eigenvalues? How can we take $\Lambda^{1/2}$ in that case?


texts: Why do people import weights for DenseNet when Keras includes them?

texts:
In several Kaggle kernels I've seen that people often import their weights into Keras' DenseNet. In the following case, I believe the weights are sourced from this github repo and contain the pre-trained weights on the ImageNet dataset.

densenet = DenseNet121(
    weights='../input/densenet-keras/DenseNet-BC-121-32-no-top.h5',
    include_top=False,
    input_shape=(im_size,im_size,3)
)

Why do people load weights from a file when you can use the ImageNet weights within Keras by specifying weights='imagenet' in the following manner?

keras.applications.densenet.DenseNet121(include_top=True, weights='imagenet', input_tensor=None, input_shape=None, pooling=None, classes=1000)

Is there a difference that I should be aware of or is it simply a matter of personal preference?


texts:
I have around 4000 customer records and 6000 user records and about 3000 customer records match leaving 1000 unmatched customers. I have created a fuzzy matching algorithm using Levenshtein and Hamming and added weights to certain properties, but I want to be able to match the remaining records without manually doing this. Ideally I want to implement an algorithm to take a customer and user and output match/no match. However, wouldn't I need to train with true negatives? Is there an algorithm that can train with just 1 label?
Thanks


texts: How do you perform basic statistical analysis on a Binary Dataset?

texts:
I have never worked with a Binary dataset (1 and 0) (True or False) so I'm unsure what kind of  statistical tests I should run to draw up simple conclusions.
I'm doing a data science project and the company wants to know/understand what features/characteristics are important to decide whether or not a customer will renew their lease (last column in Graph). They want me to just perform basic/simple probability and statistical analysis on the dataset.
I'm working in a Juypter notebook (pandas, seaborm, numpy, matplotlib, sklearn). I would appreciate it if someone could help lead me in a direction to what kind of simple analysis I can run on binary data.
Heres a sample of the table:

ID
Age 20-29
Age 30-39
Age 40-49
Age 50 >
Lease Length < 1 year
Lease Length 1-2 Years
Lease Length > 3 Years
Late Payment
No Fine Violations
Credit Score Below 600
Renews

312
1
0
0
0
0
1
0
1
0
0
1

313
0
0
1
0
0
0
1
0
1
1
0

314
0
1
0
0
0
1
0
0
0
0
0

315
0
0
0
1
1
0
0
0
0
1
1


texts: Anomaly detection - relation between thresholds and anomalies

texts:
I'm developing an anomaly detection program in Python.
Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day.
Then, using thresholds, find anomalies day by day.

I've already implemented that and these thresholds are working well:

upper threshold is equals to trimmed_mean + (K * interquartile_range)
lower threshold is equals to trimmed_mean - (K * interquartile_range)

where trimmed_mean and interquartile_range are calculated on prediction error (real curve - predicted one) and K is set to 5.

What i would like to know is if there is a rule or a method to correctly configure the value of K. Because at the moment, my approach was to adjust thresholds looking at false positive.

I'm looking for any relation between anomalies and how to set up thresholds.

I already tried to calculate for each model the AUC (area under the curve) and looking for relation, but without success. In fact, since i create a new model every day, and for the most of the days I don't have any anomalous value, i'm not able to calculate the true positive rate correctly.

Thank you

EDIT
Based on Ben's comment, I'm going to add details about my problem.
Let's start to explain better what anomalous value is for me.
So, I'm analyzing timeseries created by acquisition of kWh each 5 minutes, so 288 records per day.
An anomalous value is basically a peak, so a value that is very different from the others.
Furthermore, i need to create a model day by day because customer has not too much data and I need to catch also seasonality.
Last but not least, I working on a univariate problem, so I have just the acquisition value and time, that's it.

EDIT REPLYING BEN'S QUESTIONS
First of all yes, kWh means power consumption.
I have another type of anomalies that are blocked values (like 0, 0, 0 ecc.), but i think that i can identify them "adding a simple rule" which detect these situations.
About seasonality:
Each new model is trained using the previous 7 days, and day by day this time window moves forwarding. So, every day model is deleted and trained using just the previous 7 days.
I already managed the issue that you mentioned (train new model with anomalous data) substituting anomalous data with a new value calculated using the difference between next value and previous value.

Here a couple of plots of my data:
image 1
image 2

where green line is the prediction and blue one is the real timeseries


texts:
I have a non-conventional NLP task. I am looking to develop a sequence to a vector model. Instead of employing one-hot encoding for vectorization of the input, I am trying to see if it will be possible to learn to embed for the input text which is usually only one non-human word. For example:

1. GHJJRIDBDL7US = positive
2. LDNF3DM<VAYFI = negative
3. OFNE6NKFE1NNE = positive

ULMFiT is the state of the art now. Any idea if and how it can be applied to this problem? Any idea on how to learn the embedding for such one-word data will be helpful.


texts: Saving the hidden layers of a trained network

texts:
I have used a pre-trained VGG-19 model to build an image classifier as part of a MOOC. I have implemented a classifier like this:

classifier = nn.Sequential(nn.Linear(25088, 512),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(512,350),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(350,250),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(250,102),
                           nn.LogSoftmax(dim=1))

model.classifier = classifier

In the next step, I was asked to save the trained model so it could be reused later. My understanding of saving such a model was that I had to create a dictionary with the input size (given by the pre-trained model), the output size (given by my specific problem), the model's state dict and the hidden layers (given by my own code). I tried to do that like this:

checkpoint = {'input_size':25088,
              'output_size':102,
              'hidden_layers':[each for each in model.classifier],
              'state_dict':model.state_dict(),
              'optimizer':optimizer.state_dict(),
              'epochs':epochs,
              'class_to_index':cat_to_name}

The I tried to load it like this:

def load_checkpoint(path):
    checkpoint = torch.load(path)
    model = nn.Sequential(checkpoint['input_size'],
                          checkpoint['output_size'],
                          checkpoint['hidden_layers'])
    model.load_state_dict(checkpoint['state_dict'])
    epochs = checkpoint['epochs']
    model.class_to_idx = checkpoint['class_to_index']
    return model, epochs

test_model, _ = load_checkpoint('trained_model.pth')
print(test_model)

However, this just raises the error:

TypeError: int is not a Module subclass

What did I do wrong? As far as I can tell checkpoint['hidden_layers'] does not contain integers by themselves:

print(checkpoint['hidden_layers'])

Returns:

[Linear(in_features=25088, out_features=512, bias=True), ReLU(), Dropout(p=0.2), Linear(in_features=512, out_features=350, bias=True), ReLU(), Dropout(p=0.2), Linear(in_features=350, out_features=250, bias=True), ReLU(), Dropout(p=0.2), Linear(in_features=250, out_features=102, bias=True), LogSoftmax()]

Which is what I would expect.

Thank you for reading! Any thoughts or comments are highly appreciated!


texts: Finding point of interest in time series data

texts:
I am working on a project where I need to figure out the point of interest in time series data.

From the picture you can probably understand a bit more what I mean.
Basically, imagine this is the electricity consumption of a washing machine. From this, I want to identify these points where the consumption changes drastically. In this way I will be able to identify more or less what the machine is doing.

I only have raw data so I can't use any supervised learning algorithm, I was wondering if there are ways to do it mathematically without spending too much computation time.

Regards and thanks in advance.


texts: As far as I understand it, LDA works by assuming that a corpus was written by a set of topics ands words corresponding to that topic by a specific distribution. I'm however not enterely sure what the output of the LDA process is and more importantly, how one can (for example) model a corpus of political speeches to their sentiments.


texts:
I have a document, (purchase agreement) of approx. 100 pages. This document is sent from buyer to seller several times, and each time there is a negotiation. Negotiation could be anything.

For eg. Change in the amount of Insurance cover, or the definition of any term could be changed.
To be more specific, buyer claims the loss covered to be 10,000 dollars (Version 1 of agreement). Seller changes the agreement to cover a loss of $5,000  (Version 2). This way there is a change in the original document.

QUESTION: Is there any way by which I can predict what could be the negotiation? Can I use NLP or tf-idf to track the common changes that occurs in the document and then predict it on the new document?


texts:
Hello folks,
            Recently while going through a research paper I happened to come across a plot showing the hurricane track(figure attached). Can anyone say how it is plotted or anyone has any experience of plotting such figures? Also, I want to know what are such types of plots called.

.


texts:
I'm trying to extract resume (PDF) data. resumes always tend to follow a structure. so if you see some numbers in a cv; according to the context, we could tell whether its a telephone number, a birthday, or a date period. if I can classify/identify one entity then that would increase my ability to classify an entity near to it.

I'm still a newbie and appreciate if anyone could give me any thoughts on approaching this problem. what kind of machine learning models should i focus on?


texts: Detecting features in XY Plots using CNNs

texts: I have a simple classification problem - if two features plotted on a simple XY plot show a "kink" or characteristic turn, then the label is TRUE, otherwise FALSE.  I've been attempting to detect the "kinks" in these plots using fully-connected NN's without much success.  I thought about converting my XY plots into images, and then feeding them into a CNN, but that seems to be fairly inefficient.  Does anyone have any advice for this situation?  A kink is circled below.  Thanks


texts: Similarity measure for ordered binary vectors

texts:
I would like to ask your opinion on how to choose a similarity measure. I have a set of vectors of length N, each element of which can contain either 0 or 1. The vectors are actually ordered sequences, so the position of each element is important. Suppose I have three vectors of length 10, x_1 x2, x3: x1 has three 1 at positions 6,7,8 (indexes start from 1. Both x2 and x3 have an additional 1, but x2 has it in position 9 while x3 has it in position 1. I am looking for a metric according to which x1 is more similar to x2 than to x3, in that the additional 1 is closer to the "bulk" of ones. I guess this is a relatively common problem, but I am confused on the best way to approach it.
Many thanks in advance!


texts: Oversampling Using the Orange Data Sampler Widget

texts:
It was pointed out in the help section of the Data Sampler widget that it could be used for under or oversampling. I used the Attrition dataset where the class imbalance is 1233/237. I separated the minority class with the Select Rows widget and connected to the Data Sampler widget where I chose both Fixed sample size = 1000 and sample with replacement. I then connected the Concatenate widget to the initial dataset, the Select Rows widget and the Data Sampler widget to create a new dataset where the minority and majority classes were about the same. Classification performance (AUC) with logistic regression, neural networks, and Ada Boost improved and classification accuracy stayed about the same.
I fully understand the Python options to over or undersample but the goal of my workshop for older clinicians is to try to do as much using Orange as possible. Do you believe this is a legitimate approach to oversample the minority class? My workflow is below


texts:
Suppose I have some data set with two classes. I could draw a decision boundary around each data point belonging to one of these classes, and hence, separate the data, like so:

Where the red lines are the decision boundaries around the data points belonging to the star class.

Obviously this model overfits really badly, but nevertheless, have I not shown that this data set is separable?

I ask because in an exercise book, a question asks "Is the above data set separable? If it is, is it linearly separable or non-linearly separable?"

I would say "Yes it is separable, but non-linearly separable."

No answers are provided, so I'm not sure, but I think my logic seems reasonable.

The only exception I see is when two data points belong to different classes, but have identical features. For instance, if one of the stars in the figure above was perfectly overlapping one of the circles. I suppose this is quite rare in practise. Hence, I ask, is nearly all data separable?


texts: Best way to generate artificial handwritten text?

texts:
I would like to train a generative model that generates artificial handwritten text as output. which architectures would you recommend to use?

Training input could be either images of handwritten letters, not words, or maybe sequences of points for each letter.
I thought of using some kind of combination of GAN+LSTM/GRU.
Already found:

http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
https://distill.pub/2016/handwriting/

Would appreciate any further recommendations.


texts: What does it mean when people say a cost function is something you want to minimize?

texts: I am having a lot of trouble understanding this. Does it mean you should not use the cost function very often?


texts: Agglomerative Clustering without knowing number of clusters

texts: I want to perform agglomerative clustering, but I have no idea of number of clusters before hand. But I want that every cluster has at least 40 data points in it. How can I apply this to sklearn.agglomerative clustering? Should I use dendrogram and cut it somehow? I have no idea how to relate dendrogram to this and cutting it out. Any help will be appreciated!


texts: Python or R for implementing machine learning algorithms for fraud detection

texts:
I was wondering which language can I use: R or Python, for my internship in fraud detection in an online banking system: I have to build machine learning algorithms (NN, etc.) that predict transaction frauds.
Thank you very much for your answer.


texts: Off-Policy Estimation - Importance Sampling with Negative Rewards

texts:
Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and Swann, J) and cannot find any restrictions on the reward or value being estimated. However, it seems that there is a constraints because the calculation is not what I would expect for RL envs that have negative rewards. For example:
Consider for a given action, state pair ($a_i, s_i$) $\pi_e(a|s) = 0.4$ and $\pi_b(a|s) = 0.6$ where $\pi_b$ and $\pi_e$ are the behavioral and evaluation policies respectively. Also, assume the reward range is $[-1,0]$, and this action has a reward of $r_{\pi_b}=-0.5$. Under the IS definition the expected reward under $\pi_b$ would be $r_{\pi_e} = \frac{\pi_b(a|s)}{\pi_e(a|s)} r_{\pi_b}$. In this example, $r_{\pi_e}=-0.75$ thus $r_{\pi_e} < r_{\pi_b}$. However, assuming a change of scale of the reward to be $[0,1]$ which result in $r_{\pi_b}=0.5$, results in $r_{\pi_e} > r_{\pi_b}$.
All examples of IS I have seen in reference focus on positive rewards, however I find myself wondering if this formulation applies to reward structure that are negative. If this formulation does allow for negative reward structures, I'm not sure how to interpret this result. I'm wondering how changing the scale of the reward could change order? Is there any documentation on requirements of the value in IS? Any insight into this would be greatly appreciated! Thanks in advance.


texts: R: How to find the rows in the input files that correlate to the 2 "populations" in the plot


texts: Reframing action recognition as a reinforcement learning problem

texts: Given the significant advancements in reinforcement learning, I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into reinforcement learning problems.


texts: Manually tune tf-idf features in document classification

texts:
I am working on a multi-label document classification task with a very small data set (180 labeled documents) and a fairly large number of labels (20).
I found that - ignoring label correlations and turning the problem into 20 binary decision problems - tf-idf features fed into a logistic regression work reasonable well.
Since I have domain knowledge I would like to add manual weighs to some features to increase the prediction accuracy. For instance,  assume I train a model to recognise the category "Gender", I am thinking about something like this:
'''extract features using tfidf vecorization:'''

vectorizer = TfidfVectorizer(ngram_range = (1,2),min_df = 0.01, max_df = 0.95)
vect = vectorizer.fit(X_train)
X_train = vect.transform(X_train)
X_test = vect.transform(X_test)

'''add additional feature weight'''

weight = 10
position = vect.vocabulary_['woman']
X_train[:, position] *= weight
X_test[:, position] *= weight
position = vect.vocabulary_['gender']
X_train[:, position] *= weight
X_test[:, position] *= weight

The same weighting is then applied to the vectorised unseen data that I want to predict.
This indeed gives me a better accuracy on the test set and also leads to better predictions on the unseen data, however, I feel quite uncomfortable to "manually" change the data.
I have not found any information about such manual feature tuning. Is this something that can be  justified?
Bonus question: Tf-idf seems to work the best to build features for longer documents (>2000 characters), however, do you have another suggestion - personal positive experience - to build a feature space for long documents?


texts: Installing Orange Package with older Python Version?

texts:
I am trying to install an Add-on package from GitHub that contains prototype widgets for Orange Data Mining. I am trying to install it from the GitHub page found here.
I am using the following Terminal code to install this:
git clone http://github.com/biolab/orange3-prototypes.git

Everything then appears to install correctly and the download shows 100%. Then, however, it throws an error and says:

Orange requires Python >= 3.4

I am using Mac OS. Clearly, it is suggesting that I need to use a different version of python to install, however, I already updated my pip install. Any insight into how I can fix this?


texts:
In the context of feature relevance, I am trying to understand the meaning of the correlation method for feature selection. Can somebody please explain if the following results of the correlation coefficients arise, then should I take that feature? The rule is to select the features for which corrcoeff values are greater than 0.5. Please correct me if wrong. The way I am calculating is using Matlab's corrcoeff(target,feature) where target and feature are vectors

Case1: corrcoeff returns NaN values --

Nan Nan
Nan  1

Should the feature be selected since the value is greater than 0.5?

Case2: corrcoeff returns 0 values

0 0
0  1

In this case, I should reject the feature.

Case3:

-0.3 0
0    -0.3

Negatively correlated but absolute values less than 0.5, so reject the feature

Case4: What if there is no linear relationship at all in which case corrcoeff will not work. How do I know if there is no linear relationship and in that case how to do feature selection; is there any other function or technique?


texts: Is there a way to attach a VM instance from Azure as compute power to train your model locally?

texts:
I have a python folder locally which does the training of the model.

I am wondering, is there a way, for example to bring some GPUs locally and just "attach them" to your script or do you need to do everthing on their portal in order to train your model?


texts: What kind of neural network is the best for classifying a series of vectors

texts:
I am currently working on a project where I decided that it might be useful to use machine learning. I have a good understanding of pytorch and I know how to build my neural network within this framework.
However, currently I am struggling to find the perfect kind of neural network to solve my problem.
Let me describe the problem:
You have a game which you have developed and you want to develop an anti cheat for all sorts of cases. Now just for an example i will choose "fly-hacking".
What i am trying to do is record all of a players data over a period of time.
Let's say within 1 second i will take n "snapshots" of a current players state over e.g. 5 seconds.
This data includes the position vector in a 3-dimensional space (x,y,z), and some data of his surroundings which will be just split up in small areas (e.g. 3x3x3 matrix), now if i know there are solid objects in his surroundings they will be represented by 1 and if an area is empty it is 0.
Now my input data could be looking like this for a standing person:
(4.2, 10.2, 3.2, 0, 0, 0, 0, [....], 0, 1, 1, 1) which would just be representing a player who is standing on a plane with nothing else arround him.
And if i collect this data over 5 seconds with e.g. 3 snapshots per second i would end up with an array of 15 elements.
I can collect a fair amount of these vectors and labels them with "flying" and "not-flying" categories. I just need a fitting neural network model which would fit my problem best.
In the end I think it is just a classification problem between "flying" and "not flying" but I am struggling with the idea of which model would learn the fastest and easiest from this data.
Maybe you can help me out here and give some advice where to start :)


texts:
I have been allocated the OTB (open to Buy) budget at the start of the month for purchasing products in apparel category for online retail.

The inventory purchased on the budget should serve the following  objective:

Meet the demand forecast for the month
Ensure a certain target sell through of the inventory
Minimize the chance of mark downs of the inventory

Problem statement :

How do I allocate the budget across brands to buy products (both old and new season)?

Example output: Allocate $10k for brand A, buy 10 styles each with quantity 20 units for new season merchandise, where as buy 15 styles with 30 units for old season merchandise. Please note that I can't allocate all the budget to the best brand. It would result in lower variety available to the customers that would lead to sales loss.

This has similarities to the merchandising planning problem. I have the forecast at overall level, not at style or brand level. I also have historical purchase, product attributes , margins, customer browse and transaction data. Is there any analytical framework to solve this problem ? Any help would be greatly appreciated.


texts:
I have been learning keras and TensorFlow for some weeks now, and get confused with epoch.

I trained my network for 50 epochs, the test data and training data are randomly split (80% train, 20% test). The training data's accuracy grows nicely, but the test data's accuracy goes up and down. I am not sure why it happens like that.

Say in epoch 10, the test data's accuracy is 92%, in next epoch, how can accuracy drop? The thing I can think of is that maybe for each epoch, the test data and training data are randomly selected again, so the system sees new data which doesn't fit previous parameters?

Is it the case?


texts:
I am designing a classifier for an Imbalanced Data set. I have a queries regarding choosing the threshold for a classifier, currently I am using mean of the predicted probabilities as the threshold and I am maximizing the recall on positive classes. Is this a correct way of choosing a threshold (I.e. mean of predicted probabilities) or I should try doing something like this book by using recall as a metric on KFold cross validation.

Any suggestion would be highly appreciable. Thanks :)


texts: Train model for predicting events based on other signal events

texts:
I am trying to build a supervised ML-Model in the area of predictive Maintenance where the features are signal events, which can be represented with a gantt chart:

So my feature data consists of events with signal_code (each row in the gantt chart represents all events of a certain signal code), startTime, endTime, status (represented as the color in the chart). There can be up to a few hundred different signal codes.
I want to predict the likelyhood of an event (e.g. dark red in chart) say 30 minutes before it occures.
How could I detect "malicious" patterns prior to those events? Are there existing best practices for a use case like that?
I could find a lot information on time series (related question) but not for this kind of data.
Obviously i will ne to apply some dimensionality reduction techniques such as PCA or LDA, but what will a do beforehand in order to put the data into a one-dimensional representation?
I welcome any advice on which algorithms or feature extraction methods might help! :)


texts: What are good feature selection and engineering approaches for data with known uncertainties?

texts:
Context: I am working with a set of geological features that could have uncertainty values attached to them (for example, values come from drill holes that are sparsely distributed and must be interpolated aggressively--the distances relative to the drill holes are one type of uncertainty measures).
What I am looking for: I am looking for ways to incorporate data-error information into the feature selection and engineering steps of the pipeline. A simple starting point could be inverse weighting, but I wonder if there are other approaches out there that could also be worth experimenting with.


texts: Encoding with OrdinalEncoder : how to give levels as user input?

texts:
I am trying to do ordinal encoding using:

from sklearn.preprocessing import OrdinalEncoder

I will try to explain my problem with a simple dataset.

X = pd.DataFrame({'animals':['low','med','low','high','low','high']})
enc = OrdinalEncoder()
enc.fit_transform(X.loc[:,['animals']])

array([[1.],
       [2.],
       [1.],
       [0.],
       [1.],
       [0.]])

It is labelling alphabetically, but if I try:

enc = OrdinalEncoder(categories=['low','med','high'])
enc.fit_transform(X.loc[:,['animals']])

Shape mismatch: if n_values is an array, it has to be of shape (n_features,).

Which I do not understand. I would like to be able to decide how the labelling is done.

I considered doing this:

level_mapping={'low':0,'med':1,'high':2}
X['animals']=data['animals'].replace(level_mapping)

However, I have large number of features in my dataset which have similar categories.

Thanks.


texts: Does shuffling the training data matter in a K Nearest Neighbors Classifier model?

texts:
I am new to machine learning and I have a couple of questions about a project.
So, I created a classifier using the MNIST data set for a ML project that I was working on. I augmented the data by shifting each original point 2 pixels up, down, left, and right (so I have 240,0000 new images in addition to the 60,000 original ones). I was wondering if I should bother retraining my model with a shuffled training set, since I think (please correct me if I am very wrong) that the order of the training data in a KNN classifier does not play a factor in the final model, since predictions are made on nearest neighbors and shuffling the dataset will not change these neighbors. *As a side note, I am using the classifier/model on user drawn images, and the one with the augmented data set (unsurprisingly) performs better than the model trained only with the original data.
However, I seem to be getting the same accuracy score (the exact same to the 4th decimal place) on my test set with my augmented data set (30,000 total images) as I did with my original data set (60,000 total images). Is this something to be concerned about/did I do something wrong?
Thanks so much for your help!


texts: Binary text classification problem with small label-dataset using keras

texts:
I want to build a feedforward network with keras.
My dataset consists of youtube comments with labels 1 : spam and 0 : no spam and I use the Bag-of-words model.
The amount of the spam comments is ca. 5%.
Here is the model I use right now:

model = Sequential()
model.add(Dense(64, input_dim=len(txt_dict), activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics ['accuracy', precision, recall])
model.fit(x_train, y_train, epochs=100, batch_size=batch_size, verbose=1)

Now my problem is the model always predict 0, because of the small amount of spam labels.
I had the idea that a low recall must be punished harder, but how can I implement this?


texts: How to do feature selection after using pre-trained word embeddings?

texts:
I am working on a multiclass text classification problem. I want to use the top k features based on mutual information (mutual_info_classif) for training my model.

I started this project on ML models:

I used tfidf for feature extraction and then used mutual_info_classif for feature selection.

    svc = Pipeline([('tfidf', TfidfVectorizer()),
                   ('ig', SelectKBest(mutual_info_classif,k=1000)),
                   ('clf',LinearSVC(multi_class='ovr')),
              ])
    svc.fit(X_train, y_train)
    y_pred = svc.predict(X_test)

This was pretty straight-forward.

Next, I started to work on RNN (more specifically a simple LSTM model)and that is where I am having a problem.

I have used pre-trained word embeddings from GloVe (300d) to get features from my data.
The embedding matrix I feed into the embedding layer of my RNN has the shape (4293,300), 4293 is the number of unique words found in my data and 300 is the dimension.

My questions are: Is there any way that I can use the top 1000 words (features) out of these 4293 based on mutual_info_classif? Is it even possible to do so? If yes, then should it be done before making the embedding_matrix or after?


texts: How do I compare columns in different data frames?

texts: I would like to compare one column of a df with other df's. The columns are names and last names. I'd like to check if a person in one data frame is in another one.


texts:
I am trying to colour the points on an lmplot by a categorical variable contained within a column of a DataFrame and I am getting the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-58-e0ac8c5bf9eb> in <module>
----> 1 sns.lmplot(x='TDS', y='Li', data=df_sub, hue='Group Location', legend=True)
      2 plt.show()

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\regression.py in lmplot(x, y, data, hue, col, row, palette, col_wrap, height, aspect, markers, sharex, sharey, hue_order, col_order, row_order, legend, legend_out, x_estimator, x_bins, x_ci, scatter, fit_reg, ci, n_boot, units, order, logistic, lowess, robust, logx, x_partial, y_partial, truncate, x_jitter, y_jitter, scatter_kws, line_kws, size)
    587         scatter_kws=scatter_kws, line_kws=line_kws,
    588         )
--> 589     facets.map_dataframe(regplot, x, y, **regplot_kws)
    590
    591     # Add a legend

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\axisgrid.py in map_dataframe(self, func, *args, **kwargs)
    818
    819             # Draw the plot
--> 820             self._facet_plot(func, ax, args, kwargs)
    821
    822         # Finalize the annotations and layout

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\axisgrid.py in _facet_plot(self, func, ax, plot_args, plot_kwargs)
    836
    837         # Draw the plot
--> 838         func(*plot_args, **plot_kwargs)
    839
    840         # Sort out the supporting information

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\regression.py in regplot(x, y, data, x_estimator, x_bins, x_ci, scatter, fit_reg, ci, n_boot, units, order, logistic, lowess, robust, logx, x_partial, y_partial, truncate, dropna, x_jitter, y_jitter, label, color, marker, scatter_kws, line_kws, ax)
    787     scatter_kws["marker"] = marker
    788     line_kws = {} if line_kws is None else copy.copy(line_kws)
--> 789     plotter.plot(ax, scatter_kws, line_kws)
    790     return ax
    791

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\regression.py in plot(self, ax, scatter_kws, line_kws)
    342             self.scatterplot(ax, scatter_kws)
    343         if self.fit_reg:
--> 344             self.lineplot(ax, line_kws)
    345
    346         # Label the axes

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\regression.py in lineplot(self, ax, kws)
    387
    388         # Fit the regression model
--> 389         grid, yhat, err_bands = self.fit_regression(ax)
    390
    391         # Get set default aesthetics

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\regression.py in fit_regression(self, ax, x_range, grid)
    206             yhat, yhat_boots = self.fit_logx(grid)
    207         else:
--> 208             yhat, yhat_boots = self.fit_fast(grid)
    209
    210         # Compute the confidence interval at each grid point

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\regression.py in fit_fast(self, grid)
    228
    229         beta_boots = algo.bootstrap(X, y, func=reg_func,
--> 230                                     n_boot=self.n_boot, units=self.units).T
    231         yhat_boots = grid.dot(beta_boots).T
    232         return yhat, yhat_boots

C:\ProgramData\Anaconda3\lib\site-packages\seaborn\algorithms.py in bootstrap(*args, **kwargs)
     84     boot_dist = []
     85     for i in range(int(n_boot)):
---> 86         resampler = rs.randint(0, n, n)
     87         sample = [a.take(resampler, axis=0) for a in args]
     88         boot_dist.append(f(*sample, **func_kwargs))

mtrand.pyx in mtrand.RandomState.randint()

ValueError: low >= high

The categororical variable is a string.


texts:
I have a tensor of shape (1, M) where M is a multiple of 10. When I print the tensor it may look something like

Tensor("some_name", shape = (1, 80), dtype = float32)

This tensor is the output of a neural network which will be run in a session. I wanted to modify this tensor according to the following (broken)code:

for chunk_number in range(int(tensor.shape[1]/10)):
    this_chunk = my_fn(tensor[10*chunk_number : 10*(chunk_number+1)]) // my_fn is custom
    tf.assign(tensor[10*chunk_number : 10*(chunk_number+1)], this_chunk)

This is essentially like a custom activation function that selectively modifies the output.

However I cannot use a separate tf.Session() to run the tf.assign function because there will be a session running for the neural network. How do I go about this assignment?


texts: What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)?

texts:
I have a data set I'm trying to create a predictor model for.  The 5 features and outcome are all categorical data.  One of the features contains 5,000 unique levels. While the other 4 are all under 100 levels.  The output variable in my dataset contains 400 unique levels.

How do I handle this scenario for modeling?

My approach was going to be:

Convert the 5,000 level feature to a sequential number list via a lookup table.
Convert the remaining 4 features using One Hot Encoding.
Convert the output variable to a sequential number list via a lookup table.
Use xgboost algorithm.

I don't think I can bin the 5,000 level feature because they are user id's.  So to me, there is no way to bin them.


texts: Ordering scrambled 1D data sets by continuity

texts:
This is a cute little clustering problem that was probably solved a million times over, but I couldn't find a good reference for it.

I have 20 1D datasets with 400 entries each. In the picture they are denoted by different colors.

As you can see, they are also pretty continuous. However, for each index i the datasets have been re-ordered by magnitude, i.e. instead of nice continuous lines, the color now jumps at every intersection of each two datasets.

Is there a way to bring the the datasets to their original order? I.e. cluster the data into 20 continuous lines? This could be easily done by eye.

Thank you very much!

Ofri

P.S. Here's what I tried so far. For each index i I assumed the datasets are already ordered up to i-1. I now prepare 20 bins by extrapolating the ordered datasets from the first i-1 indices to the i index. Now I have 20 values to put in 20 bins, with exactly one value in each bin. I can try 20! combinations and find the one with the minimal error but there must be a more clever/efficient way.


texts:
The constraint that the n outputs must sum to $1$ means that only $n−1$
  parameters are necessary; the probability of the $n^{th}$ value may be
  obtained by subtracting the first $n−1$ probabilities from $1$. We can
  thus impose a requirement that one element of $z$ be fixed. For example,
  we can require that $z_n = 0$. Indeed, this is exactly what the sigmoid
  unit does.

What does it mean by "impose a requirement that one element of $z$ be fixed." I understand you can get the $n^{th}$ one if you know n-1 but "fixed"?

  Defining $P (y = 1 | x ) = \sigma(z )$ is equivalent to defining $P
(y = 1 | x) = softmax(z)_1$ with a two-dimensional $z$ and $z_1 = 0$. Both
  the $n − 1$ argument and the $n$ argument approaches to the softmax can
  describe the same set of probability distributions, but have different
  learning dynamics.

How is it equivalent to having $z_1=0$?


texts: Will setting up time series data in this way cause data leakage?

texts:
I am trying to predict future stock market values using a gradient boosted tree model. As far as I know, gradient boosted trees use the data in one row, and only that row, to predict the target variable for that row.
Therefore, I am thinking that setting up the training dataset like this would not cause data leakage?

Would this count as roll-forward partitioning in some sense, because for each row, the last year's worth of historical values are provided?


texts:
I have a big dataset with 10 columns and about a 100,000 rows. Each 5 rows represent a person being tracked and the data related to this tracking such as time, velocity, etc. the last two columns are the longitude and latitude for that person.

To test the model, the test set has the fifth row for each person missing in longitude and latitude. What's the best way to approach this problem?

for example the test set looks like:

id   time    feature2  feature3  long    lat
1      x          x        x     number  number
1      x          x        x     number  number
1      x          x        x     number  number
1      x          x        x     number  number
1      x          x        x
2      x          x        x     number  number
2      x          x        x     number  number
2      x          x        x     number  number
2      x          x        x     number  number
2      x          x        x

etc


texts:
i have labeled data with goods like:
"Double chamber refrigerator Hitachi R-WB 482 PU2 GBW".

I need to predict category like: laptops, household appliances etc.

How can i do this?


texts: Image Embeddings - Negative Sampling and Imbalanced Class Issues

texts:
I am using the negative sampling approach used in Word2Vec to train some image embeddings. From what I have read, for every positive example, we are creating a number of negative examples.

Question: Why do we use an imbalanced dataset here? Presumably we will get the normal issue where the algorithm ends up predicting the negative label to minimise the cost function? I understand that the aim isn't really to use it as a prediction model, but rather to extract the embeddings, but what is the benefit of having an imbalanced class here?


texts:
I've been using some packages in R for MCA/PCA-like analysis, yet I can't seem to find a package (or a function) within a package which allows me to implement the Kaiser-Guttman Criterion for specifying all the eigenvalues greater than 1.
Would anybody know a command in R for doing so, and the relevant packages?
There is a command at the very bottom of this page, but the authors do not specify the package used in R.
https://www.geo.fu-berlin.de/en/v/soga/Geodata-analysis/Principal-Component-Analysis/principal-components-basics/Choose-principal-components/index.html
Would be good to hear everyone's thoughts.


texts: Computation of a column-stochastic matrix with target row sums

texts:
I want to generate an $N\times N$ matrix $A$ so as to target an $N$ vector of row sums and simultaneously all column sums should sum to 1. In addition to this, I have a prefixed number of elements which are set to zero.
For example, beginning with:
$$
  \left[\begin{array}{rrr}
    0 & 1 & 1 \\
    1 & 0 & 0 \\
    1 & 1 & 0
  \end{array}\right]
$$
and the row sum vector $[1.5, 0.25, 1]^{T}$, I want to end up with
$$
  \left[\begin{array}{rrr}
    0 & a_{12} & a_{13} \\
    a_{21} & 0 & 0 \\
    a_{31} & a_{32} & 0
  \end{array}\right]
$$
under the following conditions:

$a_{12} + a_{13}  = 1.5$

$ a_{21} = 0.25$

$a_{31}+a_{32} = 1$

$a_{21}+a_{31} = 1$

$a_{12}+a_{32} = 1$

$a_{13} = 1$

While this is simplistic, in general, I have $2N$ equations in $N^{2}-Z$ unknowns, where $Z$ is the number of elements fixed to zero. So, this system of equations could be overdetermined or underdetermined, but I would like to be able to generate matrices like this such that all nonzero elements lie in $(0,1]$.


texts: The added layer must be an instance of class Layer. Found:

texts:
I don't know why BatchNormalization is giving the following error. Googled out but couldn't find any relevant answers

My code appears to be this:


texts: Finding clusters in multidimensional data

texts:
I have a set of data from 3,000 records. There are 5 attributes per individual (labelled A - E). I can use Kendall's W (coefficient of concordance) to determine the concordance between any two records.

What I require is a way to discern any clusters which exist across the entire data, and simply which of the attributes are within each cluster. e.g. for the data below I would see 2 clusters:

Cluster 1: C, D / Cluster 2: A, B

ID  | A   | B   | C   | D   | E   |
001 | 55  | 80  | 125 | 114 | 75  |
002 | 75  | 78  | 110 | 105 | 95  |
003 | 135 | 105 | 95  | 93  | 92  |
004 | 120 | 115 | 101 | 55  | 44  |

What techniques or approaches could I use to help with this?


texts: How would you - on-the-fly - prevent a neural network from overfitting using a Keras callback?

texts:
I have a neural network that starts to overfit in that the validation loss begins to increase while the training loss stays ~ flat with epochs.

Is there a generic algorithm - obvious or otherwise, well-known or not - to stop the training early if overfitting is somehow detected?

I note that catboost implements such an algorithm but I have found it nowhere else.

https://catboost.ai/docs/concepts/overfitting-detector.html

Is this all simply a matter of rolling my own callback function and stopping when the training and validation losses start to diverge..?

Preference for TF, Keras, python3, ...

Thanks as ever


texts: Proceeding with various methods for news recommendation

texts:
I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation.

I went through this paper which mentions different methods used for news recommendation (at 7th page) and most of them are using some sort of probabilistic methods (bayesian networks, latent dirichlet allocation, naive bayes model, probabilistic matrix factorization models). Also, some news recommenders  are based on multi-armed-bandit problem (e.g. yahoo's front page). I have zero knowledge about these methods.

I am very confused regarding what should be my next step. Right now, i am planning to go ahead with a very basic hybrid approach (with collaborative filtering and content based filtering). But it seems like i will have to explore these fields very soon, so what will be a good strategy (and resource) to explore these fields?


texts: How to enable GPU on GradientBoostingClassifier?

texts: Is there a way to enable GPU on GradientBoostingClassifier?


texts: conversion of dictionary to json to be sent to requests.post

texts:
I have a data frame like this.
df=pd.DataFrame({'sub1':[10,20,30,40],'sub2':[5,10,15,20],'sub3':[1,2,3,4]})

and a dictionary like this.
stud={}
stud['name']="abc"
stud['id'] ="AB10"
stud['address']="some_address"

I convert that data frame to dictionary and combine with the stud dictionary.I wrote the following code.Then this stud dictionary should be made in a json format and sent as a web service.
stud={}
stud['name']="abc"
stud['id'] ="AB10"
stud['address']="some_address"
for i in range(0,len(df)):
   data_sample={}
   data_sample_row=df.iloc[i]
   a=data_sample_row.to_dict()
   stud['data_sample']=a
   str_details=str(stud)

I would like to get the result as shown below.
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1':10,'sub2':5,'sub3':1}}"
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1':20,'sub2':10,'sub3':2}}"
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1':30,'sub2':15,'sub3': 3}}"
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1': 40,'sub2':20,'sub3': 4}}"

I would like to send this as a parameter in requests.post as following.
response=requests.post(url="some_url",data=str_details)

If I don't convert to string, gives me an error,
JSONDecodeError: Expecting value: line 2 column 1 (char 1)
If I pass json=str_details, as argument gives me the same error, How to resolve this and get the desired results.


texts: What is the use of torch.no_grad in pytorch?

texts: I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?


texts:
Why do we need to use Labelimg tool for object detection?

After labeling the bunch of training images using labelimg tool which will give CSV file How that CSV file works with TensorFlow object detection API and Keras?

Can we detect image localization without an image annotation tool? like auto annotation.


texts: PySpark: How do I specify dropna axis in PySpark transformation?

texts:
I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument axis = 'columns' in dropna(). Here an example in a GitHub post.

How do I do this in PySpark ? dropna() is available as a transformation in PySpark, however axis is not an available keyword.

Note: I do not want to transpose my dataframe for this to work.

How would I drop the furniture column from this dataframe ?

data_2 = { 'furniture': [np.NaN ,np.NaN ,np.NaN], 'myid': ['1-12', '0-11', '2-12'], 'clothing': ["pants", "shoes", "socks"]}

df_1 = pd.DataFrame(data_2)
ddf_1 = spark.createDataFrame(df_1)
ddf_1.show()


texts: How to improve a regression prediction model for data with parameters which are out of sample range?

texts:
I have been looking for some methods to improve my prediction model, but could not find any so far. I have data, including 10 numerical features, that I use for prediction. I used a random forest regression model and it works perfectly on the train set (some part of perfection comes from over fitting).

However, the model is not doing a good job for data points that some of their features are not within the range of the sample data. For example, parameter x is between 0 and 2000 in the sample but the new data point that we want to predict has a value of 3000 for x.

I know this is due to data limitation, but I wonder if there is any way that I can generate samples or improve the prediction?

I wanted to build a simple model excluding this parameter, but the problem is that this is the most significant parameter in my model.

Any hints are appreciated.


texts: Is it possible to make F1_Score differentiable and use it directly as a Loss function?

texts:
One of the metrics that is widely used in binary classification is the F1 score:

$F_1 = 2\cdot \frac{recall \cdot precision}{recall+precision}$

The problem of the F1-score is that it is not differentiable and so we cannot use it as a loss function to compute gradients and update the weights when training the model. The F1-score needs binary predictions (0/1) to be measured.

I am seeing it a lot. Let's say I am using per example a Linear regression or a gradient boosting.

Is there any way that it can be minimized directly?


texts: How to choose dimension of Keras embedding layer?

texts:
Looking for some guidelines to choose dimension of Keras word embedding layer. For example in a simplified movie review classification code:
# NN layer params

MAX_LEN = 100        # Max length of a review text
VOCAB_SIZE = 10000   # Number of words in vocabulary
EMBEDDING_DIMS = 50  # Embedding dimension - number of components in word embedding vector

text_model = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_length=MAX_LEN,input_dim=VOCAB_SIZE,
                              output_dim=EMBEDDING_DIMS),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(6, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
text_model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])

Embedding vector has 50 components in this example. Trained on 17500 and tested on 5625 reviews this model reports:
              precision    recall  f1-score   support

           0       0.87      0.86      0.87      2802
           1       0.87      0.88      0.87      2823

    accuracy                           0.87      5625
   macro avg       0.87      0.87      0.87      5625
weighted avg       0.87      0.87      0.87      5625

With 10 and even 2 dimensions I get similar values in classification report!
Then what guiding principle really works when choosing word embedding vector dimension? When select 50, 10,100, 200, ... etc. dimensions?


texts:
I have about 6 months of experience in building and using Neural Networks with no prior/formal training.  As I explore this field further, I see a lot of discussions about determining how many layers/neurons to use and some rules of thumb of where to start.  In development of my network models, I use a brute-force approach by incrementing the layers and neurons by 1 for several cycles of epoch training and then select the "best" model out of those.  My understanding at this point, is that the layers/neurons represent the relationship between the NN inputs and the NN outputs, and while training my networks, I can determine the optimal number of layers to be 5 or more.  I determine optimal based on using cross validation data the network was not trained on.

Given the above, I have read from various sources statements like "you never need more than 2 layers" and "you don't get better performance out of more than 2 layers", etc.  I have also read comments that indicate more than 2 layers is expected.  Is the idea of 2 layers could be "too much" really correct and I should be focusing on expanding the number of neurons used while capping the number of layer at 2 for my brute-force determination of optimal layers?

EDIT:

Is it is true that in the vast majority of cases no more than 2 layers is warranted as it seems some are claiming?  My specific interest is in the area of numeric prediction, however, comments about other segments are welcome as well.


texts: What methods can be used to detect anomalies in temporal texual data?

texts: I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I find a context in the data set and figure out contextual anomalies?


texts: Is this a valid way of training a model ? training loss in 1.1921e-07?

texts:
While training a keras model, i got this logs.

Epoch 49/50
2/2 [==============================] - 2s 943ms/step - loss: 1.1921e-07 - acc: 1.0000

As per my knowledge, i know that the ideal loss should be btw 1 to 10 (not sure) while training a model, but i faced this loss value while training in e-07.

Is this a valid training? Or I'm doing wrong somewhere ??


texts: What's an efficient way to compare and group millions of store names?

texts:
I'm a total amateur as far as data science goes, and I'm trying to figure out a way to do some string comparison on a large dataset.

I've a Google BigQuery table storing merchant transactions, but the store names are all over the board.  For example, there can be 'Wal-Mart Super Center' and 'Wal-Mart SC #1234', or 'McDonalds F2222' and 'McDonalds #321'.

What I need to do is group ALL 'Wal-mart' and 'McDonalds' and whatever else.  My first approach was doing a recursive reg-ex check, but that took forever and eventually timed-out.

What's the best approach for doing that with a table of 20 million+ rows?  I'm open to trying out any technology that would fit this job.


texts: Can we add other features along with Text in sentiment analysis

texts: Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given some extra features along with text from tf-idf vector matrix. Can we input other features along with tf-idf vector matrix to a ML model or DL model.


texts:
Question
Is R Forumula is to train a linear regression model?
If a response value "cancel probability" is a function of (age, height), then R Formula is to generate a model (w-age, w-height, w-bias)?
cancer_probability = w-age * age + w-loc * height + w-bias

And this is written as "RFormula":
cancer_probability ~ age + height

Is this RFormula about?
Background
Trying to understand what RFormula is. Went through multiple articles but they do not give a simple definition.
RFormula has fit and transform, hence it seems to me it is just a training algorithm, but could not find any article/book which simply says so. Hence not sure what it is for sure.

R Formulas and an Example

Suppose you’re interested in how the temperature varies with the month. Having lived through many Mays through Septembers in one place, you might guess is that the temperature generally increases in this data frame from month to month.
In this example, let’s say that Temperature depends on Month. Another way to say this is that Temperature is the dependent variable and Month is the independent variable.
An R formula incorporates these concepts and serves as the basis for many of R’s statistical functions and graphing functions.
Read the tilde operator (~) as “depends on.” Here’s how you can address the relationship between Temp and Month:
analysis <- lm(Temp ~ Month, data=airquality)

R for Data Science 3.4 Formulas and model families

In R, formulas provide a general way of getting “special behaviour”. Rather than evaluating the values of the variables right away, they capture them so they can be interpreted by the function.
The majority of modelling functions in R use a standard conversion from formulas to functions. You’ve seen one simple conversion already: y ~ x is translated to y = a_1 + a_2 * x.

R manual - Formula

The generic function formula and its specific methods provide a way of extracting formulae which have been included in other objects.

Spark 2.4.4 - RFormula

RFormula selects columns specified by an R model formula. Currently we support a limited subset of the R operators, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. The basic operators are:
~ separate target and terms
+ concat terms, “+ 0” means removing intercept
- remove a term, “- 1” means removing intercept
: interaction (multiplication for numeric values, or binarized categorical values)
. all columns except target
Suppose a and b are double columns, we use the following simple examples to illustrate the effect of RFormula:

y ~ a + b means model y ~ w0 + w1 * a + w2 * b where w0 is the intercept and w1, w2 are coefficients.
y ~ a + b + a:b - 1 means model y ~ w1 * a + w2 * b + w3 * a * b where w1, w2, w3 are coefficients.


texts: For SOLOv2: Dynamic, Faster and Stronger , could anyone explain how both decay factor and decremented functions make matrix-NMS parallel in terms of implementation ?


texts: feature engineering in test and train sets (on combined data or separately on train and test)

texts:
Background: As part of prediction analysis, I am given a train and test dataset. Both train and test data have numerical and categorical predictor variables and additionally, train data has a numerical target variable. The objective is to predict target in the test.

train = [c1,c2,x3,x4, y] = [Xc,X, y]
test = [c1,c2,x3,x4] = [Xc,X]

Xc, X denotes categorical and numerical predictor variables. I am trying to generate additional features from categorical variables Xc such as count features, count_mean, count_variance and similar features from a combination of categorical variable and a numerical variable (mean, variance etc).

Problem: Is it better to generate features on a combined dataset train+test or is it better to generate features separately on train and test datasets?

What are the implications when the distribution of a categorical variable are different in train and test and what happens when they are similar?


texts: Best approach for image recognition/classification with few training data

texts:
After research, I've decided to take advice on the approach to use for our machine learning problem as I'm still not sure which to take.
Our problem:
We have our labels as a set of website names (Google, Twitter and Facebook for example). We need to train our model on a set of images that belong to each website, and later be able to give the model any image and predict which website it might be. Also note that the image could belong to none of the websites trained so I need some sort of None output.
One of the problems we're facing is the small number of data for each set (around 40 images for Facebook login forms that could be pushed to 200 with color transformations if that makes a difference while training.)
What we need:
Some sort of image classifier/predictor/recognizer (still not sure which) that will detect the class of the input image (Facebook, Google or Twitter for example) or tell us that it's none.
Problems we might face:

I'm not sure if our model is subject to over-fitting since our set of input images are limited and we should already have trained the model on. Take the facebook login page for example, they all look the same with small variety in the location of the fields. So our training and out-of-sample data are almost the same.

What I've done so far:

I've already started working on a Convolutional Deep Network using keras, but I'm worried that deep networks would be too complex for our data, or our data is too simple(alike) for the model to be trained correctly.
One Shot Learning and Siamese Networks, but this needs to have an output that must belong to the data we've trained on (which disregards the None possibility)
Sklearn's different classifiers, but a classifier must have an either or output for one of the labels, which again disregards the None possibility.
Finally, and what sounds most logical personally, is to use sklearn's OneClassSVM on each label (so if I have Google, Facebook and Twitter then I would have 3 models, one that corresponds to each label) and then pass my input image to each model to determine to which it belongs. The problem with this is if I have a lot of classes (which could be the case) then I'd have to run each one.

One final approach could be generating probabilities for each class after inputting an image to our model, assuming that these probabilities should not sum to 1 as to keep the None possibility.
Any suggestions or advice would be helpful. Thank you!


texts: How does Keras calculate the categorical cross-entropy?

texts:
From https://towardsdatascience.com/cross-entropy-for-classification-d98e7f974451 I understand that categorical cross-entropy is -log(q(x))
When using tf.keras.losses.CategoricalCrossentropy I assume that loss is calculated per batch and per epoch as the weighted sum of the losses of the training instances or is there something else?
(So with k-fold the loss would be the sum of weighted losses of the training folds and the validation loss would be the sum of weighted losses of the validation loss?)
How does Keras deal with a loss of inf?


texts: How to find the mean of a column relative to another column?

texts:
I am working on the Boston house price prediction.  I have a column named GarageYrBlt that holds the year the garage was built for a specific house.  My assumption is that the garage would most likely be built at the same time as the house so I want to fill the missing value with the median of the column GarageYrBlt relative to the column YearBuilt.

To explain my idea further:  While I was working on the Titanic problem, I filled the missing Age column with the median relative to the Sex column.  So all female passengers with the missing age would get the value that is the median age of all female passengers.

This is what I did:

train['GarageYrBlt'] = train['GarageYrBlt']
     .fillna(train.groupby('YearBuilt')['GarageYrBlt']
     .transform("median"), inplace=True)

And when I do print(train['GarageYrBlt']), this is my output:

0       None
1       None
2       None
3       None
4       None
5       None
6       None
7       None
8       None
9       None
10      None
11      None
12      None
13      None
14      None
15      None
16      None
17      None
18      None
19      None
20      None
21      None
22      None
23      None
24      None
25      None
26      None
27      None
28      None
29      None
        ...
1430    None
1431    None
1432    None
1433    None
1434    None
1435    None
1436    None
1437    None
1438    None
1439    None
1440    None
1441    None
1442    None
1443    None
1444    None
1445    None
1446    None
1447    None
1448    None
1449    None
1450    None
1451    None
1452    None
1453    None
1454    None
1455    None
1456    None
1457    None
1458    None
1459    None
Name: GarageYrBlt, Length: 1460, dtype: object


texts: intuition behind the difference between likelihood function of discriminative and generative algorithms

texts:
I am currently studying the online material of Stanford CS229 and I came across the likelihood function for discriminative(for example, regression) and generative algorithms(for example, naive bayes):
+Discriminative:

+Generative:

In both cases, m is the number of training examples and all training examples are independent of each other. What I am wondering is why in discriminative likelihood function, the formula is the product of conditional probability of y given x and in generative likelihood function, the formula is the product of joint probability of x and y? Is there some reasoning behind this choice?


texts: Is information lost when converting an image to a single vector for analysis?

texts:
On several tutorials on object and pattern recognition in images (such as the typical mnist digit recognition problem), when working with a sample of $m$ images of $n \times n$ pixels each, the first step is to build a matrix of $m \times (n * n)$, where each image is converted into a single row-vector with all of its pixels, one next to the other. Different techniques are then applied to this matrix, such as PCA.

My question is: when converting a two-dimensional image to a single-dimensional vector by placing each image row next to another, the fact of which pixels are neighbours (in the same column, different row) seems to be lost. For instance, in image:

a b c
d e f
g h i

Which is converted to:

a b c d e f g h i

The fact that b and e were next to each other seems to be lost.

Is that conclusion correct? Does that fact even matter for analysis? Are there other ways to represent the image that do preserve that piece of information?

Apologies in advance if the question is too vague or basic, I'll be glad to reformulate if needed. Thank you.


texts:
I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document output={d10,d11,d1,d8,...}. I eyeballed the results and found it to be satisfactory, the question I have is how do I measure the performance, accuracy of the system.
I did some research and found that recall, precision, and F1-score are used to evaluating the recommendation systems that predict user ratings. For this, we should no the original ratings and then the system should predict the ratings later we can plot the confusion matrix and then compute the aforementioned metric. However, in my case, I don't predict anything instead I measure the cosine similarity score sort it in descending order and pick the top N.
In this use case, how do I evaluate the system?
Thanks


texts:
I am trying to fit a LR model with an obvious objective to find a best fit. model which can achieve lowest RSS.

I have many independent variable so i have decided to yous Backward selection (We start with all variables in the model, and remove the variable with the largest p-value—that is, the variable that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached.) to fit the model.

This is a preview of my model fit

after fitting my model i started eliminating all the variables with with high p values.

The Adjusted R squared and RSE are almost the same in both cases indicating little to no improvement.

How should i approach it further??


texts: LSTM for text with different sentences size, but same input-output sizes

texts:
Hello fellow Data Scientists

I'm trying to use a LSTM (using word embeddings) to generate a system that can tag each word of a sentence. For this, I give it a set of sentences of different sizes and using padding I make the sentences all the same size, so that the LSTM can process them.

The problem is, the LSTM results, after excluding the tags that correspond to the padding portion give outputs of different size than the input.

So my question is, is this an architecture problem or it might be the case that more training (epocs word bigger batches) is needed?

Thanks for the help!

Edit: More details:

In the following picture I show my LSTM Architecture

Upon running for a sentence of size n (using one hot encoding and word embeddings) I get a result, that is usually of diferente size; by result I mean final output of the lstm and corresponding translation into words.

Also, there are the variables I have at the moment, with which I've played around:

BATCH_SIZE = 164
EPOCHS = 50
LSTM_NODES = 128
NUM_SENTENCES = 3000
MAX_NUM_WORDS = 250000
EMBEDDING_SIZE = 300

Sample input:

1  Cada
2  obra
3  consome
4  1,5
5  tonelada
6  de
7  aço
8  (
9  US$
10 6
11 mil
12 )
13 mais
14 US$
15 10
16 mil
17 de
18 mão-de-obra
19 .

Sample output:

1  c-am-prd*
2  c-am-prd*
3  (c-v*)
4  (am-rec*)
5  (c-v*)
6  (c-v*)
7  (c-v*)
8  (c-v*)
9  am-dis*


texts: Adaptive learning of user's IoT setting preferences

texts:
We are working on a project to Predict the settings of the IoT devices(fan/light/AC) the user is using, based on his: location, outside temperature, humidity, time of the day etc...

Training data is not available and the model should start building/adapting itself every time the user uses the IoT device, to predict his preferred settings in any new environment he goes. This model will be unique for each user. Possibly we can start off with a demo model but it should adapt itself as per each user's preferences.

Which (machine?)learning algorithm can be useful here? Any links to the same and some tips on implementation would be appreciated.


texts: What are the 'protos' in TF Object Detection?

texts:
I am struggling to understand what are the 'protos' in TF Object Detection?
Why do we need them here?
Also, while setting up the TF API we need to download and compile protocol buffers.
There is also a 'protos' folder when one downloads the object detection module - could anyone please explain me what are those and what is the relationship between them?


texts: Algorithm query for bank customer segmentation

texts: I've been using k-means clustering for bank customer segmentation up until now and I'm looking to explore other clustering algorithms in the banking domain. Is it a good idea to use affinity propagation algorithm for bank data? It will also be great if you can recommend me other algorithms frequently used for clustering in the banking domain.Thanks in advance.


texts: Can we calculate iou and mean iou with only images and labels and without train and test csv file

texts: Are train and test CSV files necessary for calculating IoU (Intersection over Union) and Mean IoU. Can I calculate IoU with just images and labels?


texts:
I have a multi year dataset. Each time frame  of the data has different predictor importance.

Say for example, I am slicing the data into two partions as follows:

a dataset for the year 2014 (whole year)
a 2015 Jan.

When i look for the predictor importance, the predictor variables are different for both the partions. (1) Hence i am not able to arrive at one  unique decision tree which can explain the model better. (2). I am not able to train a model which can predict the new data correctly.

Is there anything I am going wrong here.


texts:
I have a dataframe where one of the features is the Mileage expressed in some cases in $\frac{km}{l}$, while in others is expressed in $\frac{km}{kg}$, according to the combustion type of the car (so I can't just convert some and have all expressed in the same unit because that'd make no sense).

I think it makes no sense to just ignore this and treat all numbers equally. The first approach I can think of consists of creating two separate columns for each case, in which case for each register in the dataset, one column will have a number while the other will be empty. I could then fill nans with the average value of the column, thinking of it as 'the most neutral value it could get'.

Another option I can think of consists of saving all the values in the same column, and have another column to indicate the type of units used in each case, e.g. $\frac{km}{l}$ maps to $1$; $\frac{km}{kg}$ maps to $0$.

I'm not convinced by any of the two approaches, but at the same time can't think of anything better. If anyone can help me think of a better approach to deal in this cases, I'd be very thankful.

Edit1:

Here are the types of fuel and their respective units.

CNG: ['km/kg']
Diesel: ['kmpl']
Petrol: ['kmpl']
LPG: ['km/kg']
Electric: [nan]

Although it would be possible to convert cng kg to l, it would be a different meaning in each case.

One thing I thought of was to express this information in terms of price per km... Until now, it seems the most reasonable thing to do.


texts: What is the favored discriminant analysis package in R?

texts:
I have been using the LDA package for R, but it is missing quite a few features especially those that can assess the output.

Are the any preferred packages that have some of the following?

Univariate Test Statistics
Canonical Analysis
Multivariate Statistics like Wilk's lambda


texts: What is the best way to deal with image sizes in object detection using YOLO v. 4?

texts:
I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios.
Examples of using Darknet / YOLO from the command line such as in this Colab notebook simply throw images at the darknet executable without any apparent considaration for image size or aspect ratio. This makes me think YOLO does some scaling/cropping/insertion of blank parts automatically.
Then when I look at the darknet_images Python module that comes with the Darknet/YOLO framework, the approach seems to be to use cv2 to scale images to the size of the network prior to detection in YOLO. In particular, this seems to generally skew the aspect ratio. Now I am confused: should I scale or not prior to detection via darknet.detect_image()?
I can create image objects with darknet.make_image() using the image's native size and feed them to darknet.detect_image() without any complaints, so what does Darknet do about the size in that case?
I am also wondering if it would be better to scale a, say 1000x500 image to my Darknet network's size of 608x608 as a 608x304 image in a 608x608 "frame", leaving the remainder of the frame blank (white or black)? Or should I flat out scale the 1000x500 image to 608x608, distorting the aspect ratio heavily? I suppose YOLO's training takes some amount of distortion into account via data augmentation, but I guess there must be limits to how much distortion it can tolerate?


texts: Using a loop to predict longer vectors with a LSTM

texts:
I have a model based on LSTMs that can predict a vector output based on a vector input. I can't increase the size of the output because :

I would need a larger network to obtain good results
It would take more time to train
The behaviour of my timeseries can be captured in the size of the vector I'm already predicting

So I tried to predict an output, and then use that prediction as a new input in my model and predict a new vector (without re-training). I iterate 5 times and I get this result :

As you can see, the first prediction is pretty good. Then, my model is lost and loses its accuracy. Do you know how to fix it ?

Thanks


texts: SKLearn NearestCentroidClassifier score with predict_proba

texts:
I'm using the NearestCentroidClassifier combined with TF-IDF for classification of documents. The are linked to a growing number of document groups. I've set sklearns TfIdfVectorizer and the NearestCentroid classifier up and got it going.

But instead of a hard classification which spits out only a single label I'm more interested into having 'suggestions'. For example the 3 document groups that are most relevant for a document should be chosen and given as result.

The NearestCentroid classifier does not implement a 'predict_proba' function which I'd need for this. But I guess you can add it by hand (https://stackoverflow.com/questions/52592434/decision-boundaries-for-nearest-centroid/52593267)

So far so good, my problem is now that the 'score' function of the NearestCentroid classifier is using the 'predict' function and not the 'predict_proba' (which I would also need to give a number of 'suggestions'). I guess I could use the DIY 'predict_proba' function and build my own testing suite around that.

But I just would like to know if there is a more elegant way of dealing with this.


texts: Interactive dashboard for time series data

texts:
I'm searching for some tools that allow me to build a dashboard in order to visualize information about time series. This dashboard needs to be interactive and allows to be integrated into a web site (like a web application).

I made some projects with D3.js, but for this one I would prefer something faster in term of implementation (python+matplotlib like).

What kind of options do I have?


texts: How does the SciKit-Learn version of Naive Bayes optimize its prediction?

texts:
I am working on implementing a Naive Bayes Classifier for sentiment analysis from scratch following the example given in Jurafsky's book. When evaluating the performance of my version of the classifier following this formula (using +1 / laplace smoothing):

My model has about 66% accuracy, whereas sklearn's implementations get around 75%-80% accuracy.
Is there any reason why the sklearn models are so much better, or is there something wrong in my implementation?


texts: Taking average of multiple neural networks?

texts:
I'm fitting a neural network using a very small data set,
so try splitting the data into training and validation sets.
(there is a separate test set)
If I split training/validation randomly multiple times,
construct a neural network for each training/validation split,
and take average of predicted values of the neural networks on the test set,
can it be called a ensemble model?
Or is there a specific name for such a technique?

Edit: I just found that a similar technique is called 'repeated random sub-sampling validation,' but the RRSSV splits the data into training and test set (though it is called 'validation data' according to Wikipedia, it's actually test data). My method splits the given data into training and validation set, and use separate test data.
I think my method can also be called RRSSV.


texts:
I'm building convoluted neural networks with TensorFlow. I've got the latest Mac Pro, use all of my cores, and yet it still takes me several hours to train a single network. I need to do grid search on many different tuning parameters, so this is not ideal.

This is for a personal project, so I want to save money by using the cloud. I have used AWS extensively, and I like the idea of using their spot market, however I've heard that 1) Amazon's GPU instances aren't compatible out of the box with TensorFlow (AWS's GPUs have CUDA 3.0 and TensorFlow requires 3.5) and 2) Amazon's GPUs are a bit slow.

Does Microsoft's Azure or Google's Google Compute Engine have easy-to-use GPUs when using TensorFlow? How do their GPUs compare to those of Amazon in terms of the relevant neural-net-training tech specs, prices, and ease of use? Are there smaller cloud providers that you would recommend instead?

Also, roughly how much faster can I expect training to take on one of these virtual GPUs compared to a top of the line Mac Pro?


texts: Why does my cost function return double the expected value?

texts:
I tried to implement a function propagate() that computes the cost function and its gradient knowing that :

Forward Propagation:

You get X
You compute $A = \sigma(w^T X + b) = (a^{(0)}, a^{(1)}, ..., a^{(m-1)}, a^{(m)})$
You calculate the cost function: $J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(a^{(i)})+(1-y^{(i)})\log(1-a^{(i)})$

Here are the two formulas I was using:

$$ \frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T\tag{7}$$
$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})\tag{8}$$

Which I coded as :

def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    Tips:
    - Write your code step by step for the propagation. np.log(), np.dot()
    """

    m = X.shape[1]

    # FORWARD PROPAGATION (FROM X TO COST)
    ### START CODE HERE ### (≈ 2 lines of code)
    A = sigmoid(np.dot(w.T,X)+b)                                    # compute activation
    cost = -1/m * np.sum(np.dot(Y.T,np.log(A)) + np.dot((1-Y).T,np.log(1-A)))                                # compute cost
    ### END CODE HERE ###

    # BACKWARD PROPAGATION (TO FIND GRAD)
    ### START CODE HERE ### (≈ 2 lines of code)
    dw = 1/m*np.dot(X,(A-Y).T)
    db = 1/m*np.sum(A-Y)
    ### END CODE HERE ###

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())

    grads = {"dw": dw,
             "db": db}

    return grads, cost

With the following example :

w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1,0,1]])
grads, cost = propagate(w, b, X, Y)

My output was :

dw = [[ 0.99845601]
 [ 2.39507239]]
db = 0.00145557813678
cost = 10.6046359582

Whereas the expected output was


texts: Saving and loading keras.callbacks.History object with np.save and np.load

texts:
I have been saving my training history in keras as follows:

history = model.fit(X_train, Y_train, epochs=700, batch_size=128,validation_data=(X_cv, Y_cv))

np.save('./history_sim#', history)

I am then trying to load the training history from the various simulations in order to print figures like loss vs. epoch, etc. as follows:

history = np.load('history_sim#.npy')

When I try to load the training history, I am receiving the following error message:

"ValueError: Unknown metric function:precision"

I am very worried that I have lost all of the training history now. Training takes several days and I am on a bit of a time crunch. Is the data lost or is there some way to get the data from the .npy file where I have saved it?

I already figured out that it works perfectly if I save/load as follows:

np.save('./history_sim#', history.history)
np.load('history_sim#').item()

I will do that for the rest of the simulations, but there are already a few that were saved the first way and I do not have time to re-run them.

Thanks!

Additional Info:

import keras_metrics

# Calculate precision for the second label.
precision = keras_metrics.precision(label=1)

# Calculate recall for the first label.
recall = keras_metrics.recall(label=0)

opt = Adam(lr=0.05, beta_1=0.9, beta_2=0.999, decay=0.0)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=[precision, recall])


texts: Find differences of two protein sequence files

texts:
I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in A but not in B. Both files are in text format. Each ID and Sequence is separated by new line character. I tried making pandas data frame and finding the difference but couldn't make it work.
File A:
>Q63544|9
----------------------MDVFKKGFSIAREGVVGAVEKTKQGVTEAAEKTKEGVMY
>Q63544|51
KTKQGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKT
>Q63544|54
QGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEE
>Q63544|67
VMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRK
>Q63544|72
TKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEP
>Q63544|73
KTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEPP

File B:
>Q63544|51
KTKQGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKT
>Q63544|54
QGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEE
>Q63544|67
VMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRK
>Q63544|73
KTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEPP

What I need: A-B
>Q63544|9
----------------------MDVFKKGFSIAREGVVGAVEKTKQGVTEAAEKTKEGVMY
>Q63544|72
TKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEP

Any help or suggestion would be appreciated.


texts: What is the standard terminology for the output matrix of a hidden layer in a CNN?

texts: The output of a hidden layer $h_i$ in a convolutional neural network is (generally) a 3D grid of values. These values are the outputs of the neurons of layer $h_i$. Is there a standard way to refer to this matrix of values?


texts:
Related to this paper, could it also be applied to convolutional neural networks? I ask this because FNN and CNN share many characteristics
This is the formula:
$(N_e+1)\cdot N_C + (N_C+1)\cdot N_S < 0,1N_{ej}$
Where Ne it is Entry Layers, Nc it is Hidden Layers, Ns it is Output Layers and Nej it is the minimum number of samples needed for training.


texts: Can an ordinal variable with a wide range be considered as a continuous integer variable in regression?

texts:
Just as a trivial example:

I have a sample of 25 children and I have gathered some information example: weight, age etc - to use as predictors.

Now, I don't have the height of the children, but I have an ordinal variable with the shortest child having a value of 1 and tallest child having the value of 25 - "classification wrt height" variable

If I wish to build a regression model, with the "classification wrt height" variable as the response variable, shall I consider it as ordinal or continuous integer variable please (to select the type of regression needed to be applied accordingly)?

Thank you!


texts:
XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem?

This is how I applied the parameters for a recent Kaggle problem:

param <- list(  objective           = "reg:linear",
                booster = "gbtree",
                eta                 = 0.02, # 0.06, #0.01,
                max_depth           = 10, #changed from default of 8
                subsample           = 0.5, # 0.7
                colsample_bytree    = 0.7, # 0.7
                num_parallel_tree   = 5
                # alpha = 0.0001,
                # lambda = 1
)

clf <- xgb.train(   params              = param,
                    data                = dtrain,
                    nrounds             = 3000, #300, #280, #125, #250, # changed from 300
                    verbose             = 0,
                    early.stop.round    = 100,
                    watchlist           = watchlist,
                    maximize            = FALSE,
                    feval=RMPSE
)

All I do to experiment is randomly select (with intuition) another set of parameters for improving on the result.

Is there anyway I automate the selection of optimized(best) set of parameters?

(Answers can be in any language. I'm just looking for the technique)


texts:
I finished my Economics thesis using RStudio, but my script was very slow due to massive RAM consumption during the process.

My Case

I had a massive dataset (stock prices in daily frequency for 10 years, ~700 stocks i.e. $3500\times700$) and I was picking each stock as a vector to decompose it into wavelets and CF filter (2 datasets $28000\times700$) and apply benford's law (two datasets $9\times700$).

The Problem

RStudio was storing my datasets in memory and they were consuming a significant proportion just by touching them.

Question

I started learning basic SQL commands and I found out that I can call specific columns from a certain table. Would my script be more efficient if I was calling my stocks one by one as vectors from there instead of picking them directly from RStudio? In other words, do queries call the whole dataset and then retrieve the requested values or do they follow a kind of shortcut to be memory efficient? If not, what's the purpose of using databases for domestic use?


texts: Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

texts: I would like to implement a model based on some cleaned and prepared data set. I already have a bit of experience with PySpark, but from a data scientist's perspective it can be cumbersome to work with it. Therefore I would like to try Koalas. I have a lot of experience with Pandas and hope this API will help me to leverage my skills. My question is: does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow ? As the Koalas API is relatively new, what draw-backs can be expected besides the lack of documentation ?