texts:
I am training a Mask RCNN model with a train dataset that has been generated from some simple computer vision operations (color thresholding) and some morphological filtering.
The train set captures around 2387 instances over 1157 images and there is only one class.
The validation dataset is human annotated with 1381 instances over 410 images.
I estimate the data pollution rate in the train set to be around 40%.

The model appears to be training well, albeit very slowly as indicated by the always decreasing val_loss.
However a strange thing happens with the training loss; at every drop in learning rate, the loss increases significantly (see below).

Can anyone provide insight or references as to why:

The model trains so slowly
The training loss increases while the val loss decreases
Any tips on better training models with noisy train datasets.

Parameters:

L2 loss on Conv2D layers: 1e-4 / size(kernel)
Optimisation strategy:

Nadam optimiser, LR 1e-3, scheduled to divide by 10 on plateau (patience 50)

Training from scratch


texts:
I'm totally naive to Data Science - that is, the relatively new, somewhat hyped-field that is so popular at the moment. But I'm not naive to data ... as a scientist and researcher I've worked with all sorts in different roles in the past.

Now I'm in the lamentable position of having dug a lot of shallow holes, using different software systems and different types of data, and not really being professionally competent in anything.

My question is, if I want to "get up to speed" with data science, and perhaps leverage the different experiences I've had, how do I approach it? Ideally I'd like to make my research skills marketable - that is, become a data scientist of sorts, but with a greater emphasis on the research/reporting side.

Assuming I'm coming from scratch but have demonstrated capacity - I say that because, for example, I've used R before for some projects but after a break of a year or so I need to relearn it every time... Where do I start; how do I unify all these bits and pieces?

And what claim can I make to work in this field? (I've worked on all sorts of data, from gigabytes of climate data and earth science, to health registers to longitudinal surveys ... but none of it under the moniker of a data scientist).

Specifically, what tool(s) do I learn and what theory do I need to grasp? (Keeping in mind that all my coding and statistical competencies are mostly self-taught.)

Unlike this (fascinating) question, I don't have a business background and don't necessarily want to move towards the business analyst path - I still want to play with psychical (earth science) or social data. Neither do I want to work on the data management side so much - I want databases and coding to be a means, not an end. And finally, I'm not inclined much towards the theory and mathematics. Perhaps the best way to summarise my inclination and position is that I don't want to become a data science expert, but want to be able to become an expert in given subjects through data science.

My inclination is perhaps to concentrate on something like Python, and use it to exploit R and other functionality?

Tools I've used in the past (in order of exposure) -

SAS (for statistics and research, not the warehouse side)
VBA/VB6/Excel/Access (data manipulation, reporting)
GIS (ArcGIS for analysis/research, not database management)
R (stats...)
Some HTML/JS
Some Python

One thing I find is that my existing competencies don't provide me with a useful tool for bringing together different data and getting it into the state I want for analysis (ETL I suppose?), hence the inclination to re-learn Python.

Thanks for your thoughts!


texts:
I was reading image to image translation papers.
Can someone explain to me in simple language what cycle consistency loss is and why it works in image to image traslation


texts: Activation method and Loss function for multilabel multiclass classification

texts:
I am using CNN for Sentence Classification code by Yoonkim. This is used for text classification. I noticed that he uses softmax layer and negative log likelihood error. This is optimal for single label multiclass classification. I now have a dataset which is for multilabel-multiclass classification.

What are the ideal activation methods and error functions to use?
I felt that, using softmax layer will give the sum of probabilities as 1 but in this case since it belongs to different labels, at each instance we might have high probability for each class. So I believe that Softmax is not an ideal option. Am I right?
I find data for images but nothing prominent for text. Can anyone refer me to some source?


texts: How each layer of a neural net is responsible for one feature

texts: Through my study of neural networks, I came across the idea that each layer of a neural network is responsible for recognizing one feature of the input data. For example, if we build a neural network that classifies cars, buses, vans and bicycles, a layer will be responsible to recognize the tires, another one will be responsible to recognize the size of the vehicle. The question is, why is this true? i.e. each layer appears to perform similar to the others and there is no special design for each one. Is there any way to assign each layer a specific feature or it is done implicitly?


texts:
Let's say that I receive lists of coordinates (latitude,longitude), each representing a 2D route. Later I want to use this data for route prediction, but first I have a preliminary problem to solve: I was expecting these lists to be sorted and, although the majority of the coordinates of each route are indeed sorted, some are not. For example:

a) expected:

b) actual:

where t is a timestamp.

I am looking for solutions to address this problem, for example remove all coordinates like b.t3 or sort the lists appropriately. Does anyone have any suggestion?


texts: Can word embedding be used for text classification on a mix of English and non-English text?

texts:
I'm doing text classification on text messages generated by consumers and just realized even though most of the replies provided by consumers are in English, some are in French. I've used Keras word embedding, conv1D and maxpooling to learn the structure in the text and didn't use any other text preprocessing techniques such as stop words removal etc.

In this case, I think it should be fine to use word embedding on both languages since word embedding learns the meaning of individual words regardless of languages...Is this reasonable? Or maybe I do need to separate the languages and build different models for each language?


texts:
I am using 3-fold cross validation and a grid search of the C and gamma parameters for a SVC using the RBF kernel I have achieved a classification score of 84%.

When testing against live data the accuracy rate is 70% (1500 samples used). However, when testing against an un-seen hold out set the accuracy is 86% (8800 samples, 20% of the original dataset).

The training and holdout data set have even distribution of the 3 classes.

What could be the cause of this large discrepancy? It does not seem to be over fitting judging by the performance of the model with the hold out set?

EDIT:

How did you split the data set?
The data was originally in sequential order. I wrote a script to randomly split each sample between the train and hold out set, making use of a CSPRNG. Then at the end a report was automatically generated to display the distribution of each class in each set. The distribution very near equal.

How did you do the grid search?
Through the SKlearn SVC grid search method (GridSearchCV).

Is there any overlap between the data points used during grid search and the un-seen hold out set?
No overlap, they are all from unique time stamps in the initial set.

Does the live data come from the same distribution as the other?
Yes the live data comes from the same source and the distribution is roughly the same.

How do you know?
I have a script to count up the occurrences of each class in the data set.


texts: Benchmarks based on neural networks libraries to compare the performance between different GPUs

texts:
I am looking for benchmarks based on neural networks libraries (Theano/TensorFlow/Torch/Caffe/…) to compare the performance between different GPUs.

I am aware of:

https://github.com/jcjohnson/cnn-benchmarks (CNN in Torch)
https://github.com/jcjohnson/neural-style#speed (CNN in Torch)
https://github.com/glample/rnn-benchmarks (vanilla RNNs and LSTM in Theano, TensorFlow, and Torch)

What are some other benchmark codes?


texts: How the term "R-squared" in VIF(variance inflation factor) is different from normal R-squared calculation?

texts:
In normal calculation of R2 , more the value of R2 , it indicates variable represents more variance across the dataset.
But in the calculation of VIF (variance inflation factor), higher the value of R2 , more will be the multi-collinearity present in the variables, hence more unstable the dataset is.
My question is whether R2 is the same on both cases or will it be different calculation? If different, how? If its the same, why R2 value treated as good and bad in both cases? Can someone explain this for me?
Thank you!


texts: How do you know if your Autoencoder network is fully connected?

texts: I am new in deep learning and I am confused about fully connected network. Is an Autoencoder with more than one hidden layers a type of deep neural network (DNN)? Is DNN always fully connected? Let's say an Autoencoder network with three hidden layers was built using H2O R package, how do you know if the Autoencoder network that you build is fully connected?


texts: How to remove layers from a TensorFlow2 model?

texts:
I have a CNN model which has a lambda layer doing One-Hot encoding of the input. I am trying to remove this Lambda layer after loading the trained network from a h5 file. So far I have tried to create a new model using the output layer of the old one, and old_model.get_layer('Conv1D-First-layer-after-onehot') but I get the following error:

Traceback (most recent call last):
  File "bin/extract_signatures", line 57, in <module>
    outputs=output
  File "/home/ariana/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 146, in __init__
    super(Model, self).__init__(*args, **kwargs)
  File "/home/ariana/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 169, in __init__
    self._init_graph_network(*args, **kwargs)
  File "/home/ariana/.local/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/base.py", line 457, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/home/ariana/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 272, in _init_graph_network
    base_layer_utils.create_keras_history(self._nested_outputs)
  File "/home/ariana/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 187, in create_keras_history
    _, created_layers = _create_keras_history_helper(tensors, set(), [])
  File "/home/ariana/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 213, in _create_keras_history_helper
    op = tensor.op  # The Op that created this Tensor.
AttributeError: 'Dense' object has no attribute 'op'

I also tried to tie that layer to a newly created Input(), but no change.


texts: Error Metric for Prediction of sales quantity (by using Attributes of item)

texts:
I trying to predict sales quantity of particular item by using attributes of item.Not by using Time series.I have currently 3 years of past data.I am performing this by using Neural Networks.
What is the Error metric need to follow
in this case ?


texts: Pros and cons of using the zscore of a dataset before normalizing it during feature engineering?

texts:
Normalization is a common feature engineering technique. However, this post used standardize(zscore) on the dataset before normalizing it.

I think that would result in losing some of the information in data.

What are the pros and cons of doing this?


texts: Multivariate noise variance in Gaussian process prediction

texts:
In GP regression, we predict using $\mu^* = ... (K(X,X)+\sigma^2I)^{-1}...$
This is fine when the noise $\sigma$ is a scalar, but I am confused about what happens when $\sigma$ is Multivariate/anisotropic.
$K(X,X) \in R^{m\times m}$, does $\sigma$'s dimension not depend on the width of our prediction vector $f_\ast$? If so, how does the above section of the prediction work?


texts:
Here is my model. Why is it giving a val_loss of nan?
I am splitting the data in this way:


texts: Difference between RFE and SelectFromModel in Scikit-Learn

texts: What is the difference between Recursive Feature Elimination (RFE) function and SelectFromModel in Scikit-Learn? Both seems exactly similar.


texts:
I am using sklearn and python, to build a "malicious" login identifier. Reading some documents and examples, I chose the RandomForest classifier, then I decided to use the following features:
Timestamp (hour)
IP (this is expanded in feature creation adding geolocation info like Coordinates)
Username (hashed)

In order to train the model, I got some data from a real log. I'm assuming here that these are GOOD data. Then I need to train the BAD (malicious) login.
For this I got some IPs from some public blacklist and then I create a BAD training file with the same format.
But since I don't know the username for BAD data (since these are not real login), I decided to set it to 0 and than train the model.
I think that this is messing everything up. The model does not work very well and it is not able to detect BAD login, unless I try to query it with a '0' login name. In other words, if I query the model with a  BAD IP and a valid username, I always get a "GOOD" result.
May be this is completely expected in this case, but I'd like to understand some things:

is this way of creating the BAD list actually wrong?
do I have a way to consider a feature like a "wildcard" without replicating the line in the training for any good username?
is it possible to set a "priority" in features evaluation, so that some are more important than other? But this should be done by the algorithm itself....or not?


texts: Should I build a different model for each subset

texts:
I have a dataset which has categorical variable class. I am trying to solve a regression problem

I am not understanding whether I should build a model on entire dataset and consider variable class as one of the input variable or for each class should I build a different model altogether.

What are general rules which can help me deciding between two approaches.

This is a sample of how my data looks like

+------------+----+-----+-----------------+
|   Class    | X1 | X2  |     Speed       |
+------------+----+-----+-----------------+
| Class1     | 12 | 123 |              10 |
| Class2     | 14 | 120 |              32 |
| Class3     | 15 |  34 |              12 |
|   .        |  . |   . |               . |
|   .        |  . |   . |               . |
|   .        |    |     |                 |
|  Class 300 | 23 |  13 |              45 |
+------------+----+-----+-----------------+

Class is the input categorical variable and I have around 300 classes. The output variable is Speed. I am trying to predict the speed with variables Class, X1, X2.

Should I build a model for each Class separately. So when I know input type is Class1 I will select model built for Class1. When input type is Class2 I will use model built for Class2 and so on. Also the values in Class variable can repeat meaning Class1 can come 4 times, Class2 can come 8 times etc

Other way I was thinking is to include Class itself as a variable and just build one model.

I don't know which would be the correct way for it


texts: In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence?

texts:
I want to train an LSTM model with variable length inputs. Specifically I want to use as little padding as possible while still using minibatches.
As far as I understand each batch requires a fixed number of timesteps for all inputs, necessitating padding. But different batches can have different numbers of timesteps for the inputs, so in each batch inputs only have to be padded to the length of the longest input-sequence in that same batch. This is what i want to implement.
What I need to do:

Dynamically create batches of a given size during training, the inputs within each batch are padded to the longest sequence within that same batch.
The training data is shuffled after each epoch, so that inputs appear in different batches across epochs and are padded differently.

Sadly my googling skills have failed me entirely. I can only find examples and resources on how to pad the entire input set to a fixed length, which is what i had been doing already and want to move away from. Some clues point me towards tensorflow's Dataset API, yet I can't find examples of how and why it would apply to the problem I am facing.
I'd appreciate any pointers to resources and ideally examples and tutorials on what I am trying to accomplish.


texts: Visualization techniques best suited for building non-linear models

texts:
What's the recommended approach for identifying new predictors when building gradient boosting or support vector machine predictive models? I know that visualizations are often used to come up with new predictors and/or transformations for linear models because linear models aren't able to capture non-linear behaviors or interactions in the same way that gradient boosting can. Are there visualization techniques that provide insight that you wouldn't get by just adding the candidate predictor to the model and seeing if it improves out-of-sample error?

I'm just using gradient boosting as an example, but I'm actually interested in
visualization techniques that could be applied to other powerful models like
support vector machines too.


texts: Multiclass classification of textual data

texts: I have a problem statement in which I have to classify the text data into various classes, but the training data is very less (250-300 data points for 4 classes). I am confused about what approach to use? Also is there any way that deep learning could be used for this problem with such a small tarining data?


texts: Is it necessary to learn Python from square one if your ultimate goal is to use it for data science? I have used R a lot for data science. But could I learn Python sufficiently well just by looking at other data science problems and their Pythonic approaches?


texts: logistic regression : highly sensitive model

texts:
I am a newbie to data science and ML. I am working on a classification problem where the task is to predict loan status (granted/not granted).

I am running a logistic regression model on the data. The accuracy of my model is 82%. However, my model is more sensitive (sensitivity = 97%) and less specific(specificity = 53%).

I want to increase the model's specificity. At this stage, after referring to a bunch of internet resources, I am confused about how to proceed.

Below is my observation :
In Testing data,

a percentage of 1's in the class label is 73.17073170731707.

Testing data has more 1's than 0's in the class label. Is this the reason behind model being highly sensitive.

I am attaching my data file and code file.  Please take a look at it.

Data sample :

Process :
 Data --> missing value imputation -->distribution analysis-->log transformation for normal distribution ---> one hot encoding --> feature selection --> splitting data --> model selection and evaluation

Code snippets :

Here I have selected "3 best features": Credit History, Property Area

How should I proceed? Any help (even if it's just a kick in the right direction) would be appreciated.


texts: Importance of variables in RandomForest in R

texts: I'm using varImpPlot() to evaluate the importance of variables from my rf model, and I can't decide wich variable I have to delete from my model. I'm not sure about the diference between "MeanDecreaseAccuracy" and "MeanDecreaseGini", I guess the last one can measure the purity of the tree nodes, but as an example of my model: I have a variable that has a MeanDecreaseGini of about 100 and at the same time a MeanDecreaseAccuracy of about zero.! Because of this I can't decide if this variable is good or bad.


texts:
I'm trying to use an autoencoder to reduce dimensionality of my features. My features are of dimension 2048. I tried to train an autoencoder to reduce the dimensionality to 50. I'm using a single hidden layer. But the loss decreases to certain extent and saturates. It's not even overfitting on the training data.
To test, I changed the dimension of dimension of hidden layer to 2048. My expectation was that now reconstruction is a trivial task and network would learn it easily. But to my surprise, the loss is even higher than in the first case. Any idea what is going wrong?
My network:
class MyAE:
    def __init__(self) -> None:
        flat_inputs = Input(shape=(2048,), name='flat_inputs')
        bottleneck = Dense(units=2048)(flat_inputs)
        flat_outputs = Dense(units=2048)(bottleneck)

        self.model = Model(inputs=flat_inputs, outputs=flat_outputs)
        optimizer = Adam(lr=0.001)
        self.model.compile(optimizer=optimizer, loss='mse')
        self.model.summary(print_fn=print)

I'm training my network as
num_samples = inputs.shape[0]
self.model.fit(x=inputs, y=inputs, batch_size=num_samples, epochs=self.num_epochs, verbose=1)


texts: How do you visualize Decision boundaries?

texts: My features for a problem are just a single feature, so X_train has just a single dimension. Now, I have used SVM , naive Bayes, and Logistic Regression. So now How do I draw the decision boundaries of these classifiers? Is there is a way to draw a one dimension feature vector with the decision boundary using python? I have done many searches but I couldn't find any results.  The X_train shape is (489, 1) (489 instances each with a single value)


texts: Is adding geo information to zip codes redundant in feature Engineering?

texts:
I was wondering if it is redundant to add geo information like elevation and distance between two points (between supplier and purchaser) as features to a model, if you already have country code and zip codes included?
I am afraid the Model (XGB gradient boosting) should already learn the relation using solely the zip and country codes?
Do I may face a multicollinearity issue here?
The Data set is a time-series and the mentioned features do not change over the time as you can guess.
Appreciate any thoughts!


texts: Which classifier performs better when using 'class_weight'?

texts: I have used the 'class_weight' method to balance my multi-class classification problem, using Logistic Regression, Random Forest, and XGBoost classifiers. Among these three methods, logistic regression's performance for the minority classes is substantially higher than the other two models. Could someone please explain why does LR beat decision-tree-based classifiers in this scenario? Thank you.


texts: How to use a classification model to choose the best one from three survival models?

texts:
I have a survival data like this:

Sum    Days
 4      10
 5      12
 2      7
 3       8

Let's say this is a ink purchase history of one credit card. Sum means how many bottles of ink the credit card buys at that time. Days means how many days are before this credit card makes another purchase. So, the first two lines means this credit card buys 4 bottles of ink at first time. Then 10 days later, this card buys 5 bottles. So, I consider this to be a survival model. All the data except for the last line is the training set. The last line is the test set. The model trained by the training set is used to predict the value of Days of the last line given the value of Sum. In this example, I think the model is satisfying if the predicted value is 9 as 9 is pretty close to the real value 8.

I have like 1000 credit cards each with its own ink purchase history. I trained three survival models seperately: weibull, average and ctree. We can expect that one model performs better on these credit cards while the other is more suitable for other cards.

So, I want to use a classification method to identify which data (credit cards) suits which survival model more so that my system can automically decide which survival model will be used when a particular credit card data comes in.

However, it I take the purchase data as a time series, then in my data, this is very sparse. Thereforely it is probably not good for time series classification. So, what I want to ask is if ther is a classification model which can do this?


texts: Why neural networks do not perform well on structured data?

texts:
I was recently working on some classification problem where decision trees performed better than neural networks. I had tried various combinations with neural networks altering the number of neurons / hidden layers with an objective to beat the decision tree classifier accuracy on the test set. But the best accuracy I could achieve with neural networks was that of 0.42 and decision tree was at 0.50.

I had asked a question here, as what could be the case and someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image). In the comment linked to the same answer, it was pointed out that :

  Well you can take a look at kaggle competition winners. In competitions containing structured data by far the most popular algorithm is xgboost (along with other similar algorithms lightgbm, catboost, etc.). On the other hand Neural Networks are rarely used in these competitions because they are not so strong with these types of data. This is also evident by the near 20-year disappearance of neural networks, until deep learning made them relevant again. During these years trees and SVMs on top.

That could generally be true but I do not have the intuition/reason as to why neural networks do not work well with the structured data? Could someone help me reason that? It will also be great if you could point me to some paper/post that explains this.

One feeling that I have is, it could be because of the less volume of data. Neural networks might not generalize well with fewer data points as compared to other classifiers like decision trees, svms etc. But then I am not really sure about this.


texts:
I have the following types of data for clustering - Numeric, Categorical and Latitude Longitude data for a location in one dataframe in python. I would like to know how can I go about doing clustering when the data is mixed to this degree.

Can I DBSCAN or hierarchical clustering and what do I need to do to convert categorical data to numeric.  Same with geo location data.


texts:
I wish to accomplish the following task in PyTorch-

I have the COCO dataset, wherein each data sample is used in training YOLO v3. After being processed by the model, the sample is to be deleted if it satisfies a certain condition. The data sample is thus no longer used for training in further epochs.

I now have two questions regarding implementation -

1) How do I process each sample individually? Do I go about this by setting batch size = 1? Or is there any advantage to disabling automatic batching. If so, how do I go about this.

2) How exactly do I delete the sample from the dataset for further epochs. Is there any way to skip this sample in the DataLoader?


texts: How can we show that one model might have higher accuracy than another model but at the same time lower AUC?

texts: Assume that we have two classiﬁcation models M1 and M2 that are evaluated on ﬁve test instances. How to show with an example that M1 can have a higher accuracy than M2, while at the same time M2 has a higher area under the ROC curve (AUC) than M1?


texts: How should I choose n_features in FeatureHasher in sklearn?

texts: How should I choose n_features for FeatureHasher in scikit-learn? Assume that I have 1000 categories in feature "case" and I would like to hash them.


texts:
Let's say I have a dataset with the following format:

customerid
product
orders_in_last7days
orders_in_last6days
orders_in_last5days
orders_in_last4days
orders_in_last3days
orders_in_last2days
orders_in_last1days
orders_currentday

This dataset could have multiple customers and some customers could place $n$ numbers of orders on different days. How can I flag customers that have unusual number of purchases on the current day, by looking at the distribution of orders on the previous day for that specific customer?


texts: Why are separable convolutional layers not recommended after layers having too few channels?

texts:
Since separable convolutional layers only have one spatial filter per input channel, you should avoid using them after layers that have too few channels, such as the input layer.

from "Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (p. 475). O'Reilly Media. Kindle Edition. "


texts: Why is video classification still not that accurate?

texts: I was wondering, with the advent of deep learning, many tasks related to images have been solved to near human accuracy such as classification, object detection etc., however in videos, traditional methods still compare to deep learning methods and both are not as good. I wanted to understand what and how is it that with videos that understanding them makes it much harder.


texts: How to Convert a Pandas Column having duration details in string format (ex:1hr 50m) into a integer column with value in minutes

texts:
Lets say i have the following data like below:

  import pandas as pd
    import numpy as np
    df = pd.DataFrame({'Duration': ['1h 50m', '50m', '3h', '2h 30m', '5h', '60m']
                  })
    df

Generated Output:

Column Created in pandas data frame contains duration details in string format i.e (1h 50m,50m,3h etc). I need to know how to convert this column into an integer with the value displayed in minutes (110,50,180).


texts: Scalling features for competition participants

texts:
Hello there and Happy Holidays.
I have a data set with each row representing a race with 6 participants, with each participant having its own column for each feature. The target variable is winner with values from 1 to 6(representing the # of the participant in the race).
Among others, I have 3 features for each partic: races won(win_rec), total races competed in(tot_rec), score rating(rtg). the header of df looks like this:
    | 1_win_rec | 2_win_rec |...| 6_win_rec | 1_tot_rec |...| 6_tot_rec | 1_rtg |...| 6_rtg | winner

I've scaled the win_rec cols using win_rec = win_rec/tot_rec, but I got stuck at how to scale the other two.
tot_rec distribution for all runners is shown below(is it Poisson distr?):

Should a 0-1 range normalization do the thing?
rtg, being a normal distribution, I was thinking about standardising the values to have a mean of 0 and a standard deviation of 1. But I have a dilemma: should I standardise each race on its own(std-ise the rtg values of the 6 racers in the 1st race, then the next 6 in the 2nd and so on), or std-ise the rating values in the dataset all at once?


texts: Estimating effort of building machine learning model

texts:
I know it depends on the problem and various other factors like data availability, the complexity of the use case, the workload of developer ..etc, but can someone suggest effort estimation of building a minimum viable product.

I have taken the following parameters into consideration while preparing effort estimation(close to 3months) for building a price prediction model based on two problem definitions(no sample data supplied by client), which would be integrated with web interface.

Data loading & pre-processing - 1 week
Development of model - 2.5 month
Deployment - 2 weeks.

not sure whether it's overestimated or underestimated.

Note: Please do not close with the comment "too broad" as I do not have much/can not provide more information. Requesting expert advice.


texts:
I have 2 set of numbers: one big (300.000+ numbers) and one little (50- numbers). Assuming that the elements of the little one form a cluster I want to identify the elements of the big one that belong to this cluster.
I use a one-class SVM but tuning gamma by hand is extremely difficult: very often almost all the point of the big set belong to the positive case or to the negative case.
I need an intermediate situation where the number of points of the big set in the positive case is between n_min and n_max.
I wrote this: (I'm new to python)
def check_set(big_set, small_set, mid):

clf = OneClassSVM(gamma=mid).fit(small_set)
return np.count_nonzero(clf.predict(big_set) == 1)

def binarySearch (l, r, big_set, small_set, n_min, n_max):

low = l
high = r

while low <= high:

    mid = (Decimal(high) + Decimal(low)) / Decimal(2.0)

    x = check_set(big_set, small_set, mid)

    if x > n_max:
        low = Decimal(mid)

    elif x < n_min:
        high = Decimal(mid)

    else:
        return Decimal(mid)

and I call it with:
binarySearch(0.1, 1000, big_set, small_set, n_min, n_max)

It works but I have some questions:
Are there libraries to do the same job?
Any suggestion to improve the search? (I will add a stopping criterion soon)
[0.1, 1000] is large enough for any practical purpose as interval for gamma?


texts: Fine tune the RetinaNet model in PyTorch

texts:
I would like to fine the pre-trained RetinaNet model available in torchvision in order to create my own object detection.
I'm trying to replicate what is done for the FastRCNN at this link:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#finetuning-from-a-pretrained-model
What I have done is the following:
model = model = torchvision.models.detection.retinanet_resnet50_fpn(pretrained=True)
num_classes = 2

# get number of input features and anchor boxed for the classifier
in_features = model.head.classification_head.conv[0].in_channels
num_anchors = model.head.classification_head.num_anchors

# replace the pre-trained head with a new one
model.head = RetinaNetHead(in_features, num_anchors, num_classes)

The model is declared, and the training doesn't break.
However the performance are so bad that neither a very stupid detection works.
My question is, the code that I wrote is okay to retrain the RetinaNet model?


texts:
I have found out the most profitable products in my dataframe by using:

df.groupby('ProductName')['ProfitPerOrder'].sum().sort_values().tail()

It gives me the output below. each ProductName has a ProductCategory. How do I display the category next to the product name in the output below?

ProductName
WFS Shoes           29033.887659
RDL Suit            45845.253318
Tennis Suit         46848.513342
Davenport Shoes    127103.915707
Halter Dress       314155.742025
Name: ProfitPerOrder, dtype: float64


texts: LinearRegression with multiple binary features sometimes performs poorly

texts:
I have a dataset comprising a number of binary features which are the dummies (as in, pd.get_dummies()) of categorical features. SalePrice is my target variable.

I'm literally just fitting a sklearn LinearRegression model with that data a thousand times to get an average of the score, and I'm getting a weird result. The relevant bit of my code looks like this:

import numpy as np
scores = np.array([])

for i in range(1000):

    x3_train, x3_test, y3_train, y3_test = train_test_split(
            df3.drop('SalePrice', axis=1),
            df3.SalePrice,
            test_size=0.33
    )

    lr3 = LinearRegression()
    lr3.fit(x3_train, y3_train)

    scores = np.insert(scores, 0, lr3.score(x3_test, y3_test))

print(scores.mean())

Now the weird result is that the average result is super poor, because every so often the model just tanks completely but most of the time performs "reasonably" (still terrible but that's not a surprise as it's incredibly basic and not tuned at all, I'm just comparing the effect of treating a set of features in different ways). For example the first 30 runs generated these scores:

0     5.907010e-01
1     6.044523e-01
2     5.178049e-01
3     5.622240e-01
4     5.810432e-01
5     5.131722e-01
6     5.772946e-01
7     4.674152e-01
8     4.962015e-01
9     4.887872e-01
10    5.144772e-01
11    5.676829e-01
12    5.122566e-01
13    5.453985e-01
14    5.355022e-01
15    5.888459e-01
16    5.552912e-01
17    5.615658e-01
18    5.472429e-01
19    5.810185e-01
20    5.334900e-01
21    5.493619e-01
22    5.567195e-01
23    5.514374e-01
24    4.916478e-01
25    4.580718e-01
26    5.286095e-01
27    5.761865e-01
28    5.638573e-01
29   -1.809208e+24
Name: lr3, dtype: float64

I guess my question is what is likely to be happening on that 30th run through such that the model performs so poorly? I'm comparing this model to others that treat the data differently (e.g. simply encode using .astype('category').cat.codes) and whilst there's relatively minor variations in the "usual" range of scores (they're all sort of 0.44 - 0.63) those other models don't have this occasional complete tanking.


texts:
I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName"

Doing this in Pyspark:

display(df.filter(df.AppName=="23954ec32332dfgd"))

only yields 1 row as opposed to at least 1400 rows.

Any ideas ?


texts: Should the data be shuffled on a translation dataset

texts:
As reference to Why should the data be shuffled for machine learning tasks I was curious if this is the case for neural-machine-translation.

I would like to analyze why I think this is the case for the 3 components (train, test, val) parts where the data (as a reference dataset think of the europarl-dataset) is split:

for the training part, when doing mini-batch gradient descent it is better that for each update we collect data that are closer semantically, so that the learning gets focused each time on a certain direction. For example is better to have a lot of topics for let's say coffee together for a single update, than topics about drugs or coffee, dogs etc in a single update
for the test part, we would like to see how the model behaves in a big text of continuous sentences, as the goal of translation is that of documents. The same should hold for validation

I would like your view on that if you have any empirical or theoretical insight of why this is not the case for neural machine translation.


texts: Number of leaves for lightgbm is smaller than categories in one feature

texts:
I was looking at a notebook someone posted for a Kaggle competition.  They use lightgbm with the number of leaves set to 40.  If I understand right, that's setting a limit on the size of the weak learners (decision trees) that are used in the boosting; the trees can't have more than 40 leaves.

However, after training, we see that the feature with the greatest feature importance is a categorical variable with 1000+ categories! If a branch were ever used in a decision tree for that variable, wouldn't it necessarily have at least 1000+ leaves?

How is this situation handled? When the number of leaves on the weak learners is smaller than the number of categories within one of the variables?


texts: Using a part of a trained model in a custom loss function -Tensorflow

texts:
I want to write a custom loss function that uses the intermediate result of a trained discriminator.
the loss function compares images.
the loss function is for recovering the latent vector of an image from a gan.
im relatively new to this.

visual representation

im using a reference code to test it out.
https://github.com/utkd/gans/blob/master/cifar10dcgan.ipynb
for full reference code im using

below is an example
https://m.youtube.com/watch?v=dCKbRCUyop8
Watch at 17:30

below is the discriminator code

def get_discriminator(input_layer):
  '''
  Requires the input layer as input, outputs the model and the final layer
  '''

  hid = Conv2D(128, kernel_size=3, strides=1, padding='same')(input_layer)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

  hid = Conv2D(128, kernel_size=4, strides=2, padding='same')(hid)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

  hid = Conv2D(128, kernel_size=4, strides=2, padding='same')(hid)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

for my loss function i want to use the intermediate result from the layer above

  hid = Conv2D(128, kernel_size=4, strides=2, padding='same')(hid)
  hid = BatchNormalization(momentum=0.9)(hid)
  hid = LeakyReLU(alpha=0.1)(hid)

  hid = Flatten()(hid)
  hid = Dropout(0.4)(hid)
  out = Dense(1, activation='sigmoid')(hid)

  model = Model(input_layer, out)

  model.summary()

  return model, out

below is the code im planning to use

zp = tf.Variable(np.random.normal(size=(1,l_size)), dtype=tf.float32)

start_img = Image.open(folder + "foo_00.png")
start_img.resize((img_x, img_y), Image.ANTIALIAS)
start_img_np = np.array(start_img)/255

fz = tf.Variable(start_img_np, tf.float32)
fz = tf.expand_dims(fz, 0)
fz = tf.cast(fz,tf.float32)
# variable 'generator' = trained model that is loaded.

# Define the optimization problem
fzp = generator(zp)
loss = tf.losses.mean_squared_error(labels=fz, predictions=fzp)

here is where i want it to go something like

fzpD= discriminator_intermediate(fpz)
fzD= discriminator_intermediate(fz)
loss = tf.losses.mean_squared_error(labels=fzD, predictions=fzpD)
```


texts: How is Amazon using Deep Learning in the new Amazon Go?

texts:
In the Amazon Go page they claim to use "deep learning".

Where is deep learning used in the "Go" service? Is it computer vision or inventory management?


texts: Using neural network regression for skewed data

texts:
I'm using a feed forward network for a regression problem where the response variable is a ratio that can be negative and is very heavily skewed.

As the response can be negative, I can't just log transform the raw data. I've thought about doing log(1+x), rank(x) and standardize(rank(x)). What would be the best way to deal with this while preserving as much feature of the data as possible?


texts: Unable to figure out the linear embedding layer in the convolutional neural network?

texts:
I have the network architecture from the paper "learning fine-grained image similarity with deep ranking" and I am unable to figure out how the output from the three parallel network is merged using the linear embedding layer.
The only information given on this layer, in the paper is

  Finally, we normalize the embeddings from the three parts, and combine them with a linear embedding layer. The dimension of the embedding is 4096.

Can anyone help me in figuring out what exactly does the author mean when he is talking about this layer?


texts: How to asses employees performance using their data?

texts:
I have some employees' data consisting of three variables - the name of the employee, a value denoting their positive behavior and a value denoting their negative behavior.

How to score employees using this data with a 70% weight on negative behavior and a 30% weight on positive behavior?


texts: What are the pros and cons of zero padding in a convolution layer?

texts:
TensorFlow's conv2d() operation lets you choose between "VALID" (without padding) and "SAME" (with zero-padding). I suppose all other frameworks let you do the same. I'm trying to understand the pros and cons of zero padding: when would you want to use it, or not to use it?

So far, my understanding is that if the filter size is large relative to the input image size, then without zero padding the output image will be much smaller, and after a few layers you will be left with just a few pixels. So to maintain a reasonably sized output, you need zero-padding + stride 1. Is this the main reason for using zero-padding? Is it preferable to avoid it when you can, for example when the filter size is small relative to the input image size?


texts: How can I preprocess multi-page image inputs in a theano/lasagne network?

texts:
I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no  photographs or icons), and different documents may have a different shape (height and width). I'd like my classification approach to use all pages in each document, rather than just the first page.

As far as I know, the input to a CNN (like theano) needs a standardized shape. My first thought was to create a single image array that has all the pages concatenated. But I would then have to resize/zero pad all concatenated pages to match the length of the document with the largest page count, and use that as my lowest height and width for the set of concatenated pages. If I don't use this strategy I risk losing resolution on the words in the image, but that is a giant input vector.

I feel like splitting each document's pages into separate samples would be a better approach to standardize input preprocessing, but I'm at a loss for how to classify the whole document if I'm just training on loose pages of each document. Can anyone advise on me on this?


texts:
I am using convolutional nets for a physics application. I am trying to figure out how to structure my raw data as an image for input into the CNN.

I have $N$ samples. Each sample consists of the following:

(1) Temperature values at discrete times and at discrete spatial locations.
I.e., temperature is a function of time, $t$, and space, $(x,y,z)$.

(2) TBD, but it should consist of some other spatial and temporal data. I haven't figured this out yet. For this question and for simplicity, we could just limit it to working with (1) only.

The examples that I've looked at deals with images, and they're inherently 3D. e.g., 100 pixels wide by 50 pixels high by 3 RGB channels. So the input can be structured as a 3-D tensor.

In my case, temperature depends on 4 independent variables (3 for space and 1 for time). What would be the best way to wrangle it for use as input into a CNN? A 4-D tensor? Or would it be better to map the 4-D tensor to a lower-order tensor, say, 3-D or 2-D?

Also, somewhat of a side note, I am a little bit confused on how an input image relates to the number of nodes in the input layer of the NN. Say, we have an image that is 100 pixels wide by 50 pixels high with 3 channels. Does this mean there should be 100x50x3=15,000 nodes in the input layer?

I'm relatively new to this, so I don't really know what I'm doing. Any insight would be greatly appreciated.

-----------------Edit 1---------------

After some thinking, this is one format that I'm considering. This format represents my temperature values in an image-like matrix:
$$
\begin{bmatrix}
T(x_1,y_1,z_1,t_1) & \cdots & T(x_1,y_1,z_1,t_n) \\
\vdots             & \ddots & \vdots \\
T(x_m,y_m,z_m,t_1) & \cdots & T(x_m,y_m,z_m,t_n) \\
\end{bmatrix}
$$
where $m$ is the total number of locations and $n$ is the number of discrete time intervals. Each column is constant in time, but varying in space. Each row is constant in space, but varying in time. The matrix elements closest to each other (i.e., closer in space and time) have temperature values that are similar, and elements further from each other have temperature values that are less related.

In this format, the number of channels would be 1. And my input into the first convolution layer would be $m \times n \times 1$. Does this format fit the definition of a 2-D topological order? It seems to me that it does since this matrix abstraction is structured such that elements close to each other should be more similar than the elements further apart (see bolded statement above).

Another format I am considering is similar to the one suggested by Antonio in his answer. In this format, the number of channels would be equal to the number of discrete times, so $n$, and my "image" becomes a 1-D column (or I guess it could be a row vector too):
$$
\underbrace{\begin{bmatrix}
T(x_1,y_1,z_1,t_1)  \\
\vdots              \\
T(x_m,y_m,z_m,t_1)  \\
\end{bmatrix}}_{\text{channel 1}},
\  \ \underbrace{\begin{bmatrix}
T(x_1,y_1,z_1,t_2)  \\
\vdots              \\
T(x_m,y_m,z_m,t_2)  \\
\end{bmatrix}}_{\text{channel 2}},
\ldots
,\underbrace{\begin{bmatrix}
T(x_1,y_1,z_1,t_n)  \\
\vdots              \\
T(x_m,y_m,z_m,t_n)  \\
\end{bmatrix}}_{\text{channel n}},
$$
So each sample input is now $m \times 1 \times n$, and for each channel we have a $m \times 1$ column vector.


texts: How can I encode a 'Name' so that similar names are represented by vectors close in n-dimensional plane?

texts: I want to encode names of people for similarity comparison between them such that a name like 'Sarah' is closer when represented in vector to a name like 'Sarah connor', something very similar to what word2vec does but it uses sentences to train but I only have list of words. Using string matching algorithms like Levenshtein distance and Jaccard Index, I can find similarity between them but this can't be used to derive vector embeddings of these words satisfying above conditions, or can they ? If not is there a way to encode these list of names such that the condition of similar names (based on characters and other conditions) are closer in n-dimensional space, n being the vector length of these embeddings.


texts: How to resolve too many indices for array Index Error

texts:
I'm performing a binary classification in Keras and attempting to plot the ROC curves. When I tried to compute the fpr and tpr metrics, I get the "too many indices for array" error. Here is my code:

#declare the number of classes
num_classes=2
#predicted labels
y_pred = model.predict_generator(test_generator, nb_test_samples/batch_size, workers=1)
#true labels
Y_test=test_generator.classes
#print the predicted and true labels
print(y_pred)
print(Y_test)
'''y_pred float32 (624,2) array([[9.99e-01  2.59e-04],
                                 [9.97e-01  2.91e-03],...'''

'''Y_test int32 (624,) array([0,0,0,...,1,1,1],dtype=int32)'''

#reshape the predicted labels and convert type
y_pred = y_pred.argmax(axis=-1)
y_pred = y_pred.astype('int32')

#plot ROC curve
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(num_classes):
    fpr[i], tpr[i], _ = roc_curve(Y_test[:,i], y_pred[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])
fig=plt.figure(figsize=(15,10), dpi=100)
ax = fig.add_subplot(1, 1, 1)
# Major ticks every 0.05, minor ticks every 0.05
major_ticks = np.arange(0.0, 1.0, 0.05)
minor_ticks = np.arange(0.0, 1.0, 0.05)
ax.set_xticks(major_ticks)
ax.set_xticks(minor_ticks, minor=True)
ax.set_yticks(major_ticks)
ax.set_yticks(minor_ticks, minor=True)
ax.grid(which='both')
lw = 1
plt.plot(fpr[1], tpr[1], color='red',
         lw=lw, label='ROC curve (area = %0.4f)' % roc_auc[1])
plt.plot([0, 1], [0, 1], color='black', lw=lw, linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristics')
plt.legend(loc="lower right")
plt.show()

The shape of y-pred and Y_test are:

y_pred float32 (624,2) array([[9.99e-01 2.59e-04],
                                 [9.97e-01  2.91e-03],...

Y_test int32 (624,) array([0,0,0,...,1,1,1],dtype=int32)


texts: How to get model attributes in scikit learn (not hyper parameters)

texts:
How to get model attributes list (not hyper parameters passed to Estimator's class)?
For ex:
kmeans = KMeans(n_clusters=5)
kmeans.fit(X)
kmeans.labels_

how to get list of the attributes like labels_ from model object (ending with _)?


texts: How is the linear regression cost function evolved?

texts: A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. Can anyone help me by telling how that cost function has been evolved?


texts: How to perform feature engineering on unknown features?

texts:
I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.

People are performing a lot of feature engineering on these features. I am wondering how exactly one is able perform feature engineering on features which are unknown? Can somebody please help me understand this and some tips on how can I perform feature engineering on unknown features?


texts: Classifying whether a chess board square contains a piece or not?

texts: I have an image of a chessboard which I segment into its constituent 64 squares. I want to construct a binary  classifier which can detect which square contains a piece or not. I know how to do this via CNN's, but I feel like that might be a bit overkill. I was wondering what would the simplest approach be to this problem? Thank you.


texts: Does it make sense to use TF-IDF to extract most important tokens from a corpus?

texts:
I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus.

My understanding of TF-IDF is that it is calculated per token per document, so the calculated weights are relative to a given document in the corpus. Is there a way to use TF-IDF to recover the most significant terms in the entire corpus, or is this the wrong approach? If the latter, what would be a more appropriate NLP approach?


texts:
I'm looking for ways to extract sentences from paragraphs of text containing different types of punctuations and all. I used SpaCy's Sentencizer to begin with.
Sample input python list abstracts:
["A total of 2337 articles were found, and, according to the inclusion and exclusion criteria used, 22 articles were included in the study. Inhibitory activity against 96% (200/208) and 95% (312/328) of the pathogenic fungi tested was described for Eb and [(PhSe)2], respectively. Including in these 536 fungal isolates tested, organoselenium activity was highlighted against Candida spp., Cryptococcus ssp., Trichosporon spp., Aspergillus spp., Fusarium spp., Pythium spp., and Sporothrix spp., with MIC values lower than 64 mug/mL. In conclusion, Eb and [(PhSe)2] have a broad spectrum of in vitro inhibitory antifungal activity."]

Code:
from spacy.lang.en import English

nlp = English()
sentencizer = nlp.create_pipe("sentencizer")
nlp.add_pipe(sentencizer)

# read the sentences into a list
for doc in abstracts[:5]:
    do = nlp(doc)
    for sent in list(do.sents):
        print(sent)

Output:
A total of 2337 articles were found, and, according to the inclusion and exclusion criteria used, 22 articles were included in the study.
Inhibitory activity against 96% (200/208) and 95% (312/328) of the pathogenic fungi tested was described for Eb and [(PhSe)2], respectively.
Including in these 536 fungal isolates tested, organoselenium activity was highlighted against Candida spp.,
Cryptococcus ssp.,
Trichosporon spp.,
Aspergillus spp.,
Fusarium spp.,
Pythium spp.,
and Sporothrix spp.,
with MIC values lower than 64 mug/mL. In conclusion, Eb and [(PhSe)2] have a broad spectrum of in vitro inhibitory antifungal activity.

It works fine for normal text but fails when there are dots (.) present in the sentence elsewhere other than at the end, which breaks the whole sentence as shown in the above output. How can we address this? Are there any other proven methods or libraries to perform this task?


texts: How to select algorithms for ensemble methods?

texts: There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far away are the algorithms from each other? What are the best combinations?


texts: Building predictive model with low correlated data

texts:
I have been working on a project with low features and only few entry fields ( 4 to be exact ). All the data in the dataset is barely correlated to each other. Is there some organized way or approaches to tackle such data and thus build a predictive model out of it. To give you a little glimpse of the dataset, here are a few columns from the

+----+----------+-----------+---------+
| ID | OAG CODE | CONSIGNEE | SHIPPER |
+----+----------+-----------+---------+
| 11 | 665      | 1001      | 20100   |
| 11 | 665      | 1006      | 20105   |
| 13 | 667      | 1023      | 20110   |
| 13 | 669      | 1015      | 20104   |
| 13 | 669      | 1006      | 20105   |
+----+----------+-----------+---------+

I want to perform EDA over the data and also build a predictive model from the same. Please point out some of the standard techniques and methods for tackling such a problem.


texts: Coursera's course "How to Win a Data Science Competition"

texts:
The Coursera Course "How to Win a Data Science Competition".

Has onyone seen it? How is the quality of it? Would you recommend it?


texts: Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

texts: Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not when the 'penalty' argument is set to a value other than none (i.e. 'l2', 'l2', or 'elasticnet')?


texts: Can RNN learn for each `t` in time from a whole new dataset (many entries)


texts:
I was asked in an interview why do we use the binomial distribution in logistic regression and how is it related to the class that we are predicting?

Could anyone explain, without any mathematical equations, why do we use binomial instead on any other distribution?


texts: Reinforcement Learning algorithm for Optimized Trade Execution

texts:
My question deals with the algorithm described in the paper:
Reinforcement Learning for Optimized Trade Execution

This paper uses reinforcement learning technique to deal with the problem of optimized trade execution. They divide the data into episodes, and then apply (on page 4 in the link) the following update rule (to the cost function) and algorithm to find an optimal policy:

(T is the total time units, I is the volume, L is the possible number of actions, x represents the state, and c represents the cost function, and c_im is the immediate reward at a certain state and a certain action. n is the number of times that the state-action pair were visited)

Here are my questions:

If I understand correctly, the algorithm is basically a dynamic programming, when we move backwards in time.

Why do we need n in the cost function update rule. Aren't we visiting each state exactly once?
If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes). In such case, how do we combine the results from all the episodes? That is, each episode provides an optimal policy. How do we combine all these policies to one final policy?


texts: What kind of data visualization should I use?

texts:
I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" button take up some of the space of the "w" button:

 (screenshot from here)

In order to figure out how often I make specific typos, I'll need to collect data and store it in some kind of graphic organizer to help me visualize it. Right now, I'm thinking of something like a cluster map. Does that sound like a good plan, or do you have any other suggestions?

Here's an example cluster map (from Microsoft PowerBI). Instead of pictures, they would say things like "KI", to show how often I type "K" instead of "I", and the circle would be larger or smaller depending on how often I make this mistake:

Thanks!


texts: How to convert VGG VIA Instance segmentation annotation to COCO/PASCAL for Tensorflow Object Detection?

texts:
I have VGG VIA JSON annotations for instance segmentation for counting diamonds from a given image. The annotations are mixture of Circle, Polygons, Polylines. However, I am planning to use the Tensorflow Object detection with EfficientDet. In order to use it, it needs to have the annotations in either COCO or PASCAL format so that it can be converted to TFRecords. I guess COCO/PASCAL only need Rectangular box and not the mixture of shape annotations which my VIA Json file has. I tried the Roboflow for conversion, but it doesnt have this feature.
How do I convert it so that it can be used with TF Object Detection.
Thanks


texts: How do I set/get heap size for Spark (via Python notebook)

texts:
I'm using Spark (1.5.1) from an IPython notebook on a macbook pro. After installing Spark and Anaconda, I start IPython from a terminal by executing: IPYTHON_OPTS="notebook" pyspark. This opens a webpage listing all my IPython notebooks. I can select one of them, opening it in a second webpage. SparkContext (sc) is available already, and my first command in the notebook is help(sc), which runs fine. The problem I'm having is that I am getting a Java heap space error that I don't know how to address. How do I view my current Java heap setting, and how do I increase it within the context of my setup. The error message I'm getting follows:

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 19 in stage 247.0 failed 1 times, most recent failure: Lost task 19.0 in stage 247.0 (TID 953, localhost): java.lang.OutOfMemoryError: Java heap space


texts:
I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum message size:

    scala> val mod = pipeline.fit(df)
    mod: org.apache.spark.ml.PipelineModel = pipeline_936bcade4716
    scala> mod.write.overwrite().save(modelPath.concat("model"))
    18/01/08 10:00:32 WARN TaskSetManager: Stage 8 contains a task of very large size
    (755610 KB). The maximum recommended task size is 100 KB.
    org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task
    2606:0 was 777523713 bytes, which exceeds max allowed: spark.rpc.message.maxSize
    (134217728 bytes). Consider increasing spark.rpc.message.maxSize
    or using broadcast variables for large values.

Making the following assumptions about the problem area:

It's not possible to decrease the size of the model  AND
It's not possible to increase the maximum message size to a point where the pipeline would fit in a single message.

Are there any methods that would allow me to save the pipeline successfully to HDFS?


texts: For very simple linear regression can we quantify the prediction accuracy hit between using one hot encoding and simple numerical mapping?

texts:
Suppose I had a simple linear regression model that had the following input or X variable:
[North]
[East]
[West]
[South]
[North, East]
...
[North, East, West, South]

and I decided to numerate them like:
[North] - 0
[East] - 1
[West] - 2
[South] - 3
[North, East] - 4
...
[North, East, West, South] - 15

I had someone take a look at my model and tell me to use One Hot Encoder or One Hot Binary Encoder instead of assigning inputs like this.
My question is from a linear regression perspective what is the advantage for using OHE over my simple numerical mapping? If we can quantify an accuracy loss would it be substantial? If I had 10 model variables that I had to map like this would the loss be more substantial?
I want to know what sacrifice I'm making not using OHE


texts: How to shift rows values as columns in pandas?

texts:
Input: I have csv file like below as input....

ID, Year,Specialty,AgeRange,PlaceSvc,Count, Group
101,2009,Internal,  20-29,  Office,     0,  PRGNCY
101,2010,Emergency, 20-29,  Urgent Care,0,  GIOBSENT
101,2011,Internal,  20-29,  Office,     0,  GYNEC1
102,2010,Other,     30-39,  Office,     1,  PRGNCY
102,2010,Laboratory,30-39,  Independent,1,  MSC2a3
103,2009,Laboratory,30-39,  Independent,1,  MSC2a3
103,2011,Other,     30-39,  Office,     0,  PRGNCY

Output: I want output like below...

ID,Year,Specialty_Internal,Specialty_Emergency,Specialty_Labrotory,Specialty_Other,Age20_29,Age30_39,PlaceSvc_Urgent,PlaceSvc_Office,PlaceSvc_Independent,Count,GroupPrgncy,GroupGiobsent,GroupGynec1,GroupMsc2a3

101,2009,1,0,0,0,1,0,0,1,0,0,1,0,0,0
101,2010,0,1,0,0,1,0,1,0,0,0,0,1,0,0
101,2011,1,0,0,0,1,0,0,1,0,0,0,0,1,0
102,2010,0,0,1,1,0,1,0,1,1,2,1,0,0,1
103,2009,0,0,1,0,0,1,0,0,1,1,0,0,0,1
103,2011,0,0,0,1,,1,0,1,0,0,1,0,0,0

How can i do this by pandas? or is there any other techinque to do this?


texts: Is this a correct way improving a statistical model?

texts:
I am building a fair amount of statistical models - text classifiers and sequence taggers. The statistical models are linear in features - Logistic Regression and linear chain first order CRF.

The models undergo several stages starting from a rough, initial version all the way to a mature production model. Given a specification and some dataset (often with incomplete annotation, noisy labels and features), I build the first model from a small, hand labeled dataset. This dataset is then grown iteratively by adding more labeled examples (e.g. using crowdsourcing).

Then comes the test phase. The statistical model is actively evaluated within a larger system by testers. I, as a developer, start getting the failure reports on their input. I am supposed to fix the model, update it in the system and deploy it.

Every individual error report is fairly small to manually go through and fix every misclassification. Obviously it is incorrect just to simply add the failure cases with correct labels to the training set. You never know if the model learned that new pattern.
I do this by comparing the failure cases with a vast unlabeled dataset of examples (real usage data but no labels) and extract similar examples. I label those similar examples, add to the training set and re-train the model. Then I check if the new model fixes the originally reported issues.

My question is: Is this a correct approach to build and continuously improve the statistical models? May be there are some hidden caveats that will bias my model in some sense?


texts: Deep RL: Visualizing/Analyzing the gradient

texts:
I am testing different RL methods, and I know e.g that policy gradient method is supposed to have a high variance gradient which can cause trouble. I want to run a few different Deep RL algorithms, and see if I can see any patterns in the gradient. How can I visualize the gradient, and analyze it?

So far I have been using the average standard deviation, of the gradient, for every layer in my neural network and visualized it as a scalar with epoch on the x-axis. Is there a better way?


texts: What are the differences between logistic and linear regression?

texts: I know that linear regression does "regression" and logistic regression does "classification". When we implement these two methods, the only difference I could notice is the loss function: linear regression uses a loss function like mean square error and logistic uses cross-entropy. Is there any other difference that I am not aware of?


texts: Test accuracy of neural net is going up and down

texts: I am using a CNN to classify medical images. I am using a four convolutional layers with ReLU activation followed by a softmax layer. I am using rmsprop as the optimizer. The problem I am facing is the network's test accuracy increases and then goes down. This continues till the maximum number of epochs is reached. I read somewhere that this could be due to the fact that the network shifts its weights in favor of one class, and then has to do so in favor of the other class. I shuffled the test data, to see if this would help, but no such luck. Is there any remedy?


texts: The most informative curve for imbalance datasets

texts:
For the imbalanced datasets:

Can we say the Precision-Recall curve is more informative, thus accurate, than ROC curve?
Can we rely on F1-score to evaluate the skillfulness of the resulted model in this case?


texts: Autoencoder neural network for dimensionality reduction of many-to-many regression problem

texts:
I want to find the relationship between a continues input-output using neural networks. The input consists of time series with more than 1000 data points. The output of each input time series is the corresponding time series of the same size. So it is a many-to-many regression problem.

The data points in each time series are highly correlated. So, I used a LSTM network but I did not get good results.

When I use a feed-forward network, only one layer with a handful of units (less than 50) is sufficient to get fairly good results. This is not surprising, because the input size is relatively large, and thereby one layer with an small number of units provides sufficient capacity (parameters) to learn.

Now I want to go deep. I don't have that much data to support the capacity of a deep network. So I thought that I can use an autoencoder network to reduce the dimension of the input.

Ideally I can reduce the input size to just one without loosing any information. But using input size of one in a feed-forward network seems to be losing too much information.

What am I missing?


texts: Dealing with high dimensionality datasets

texts:
I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for classification purpose. Is there any other way I can model this data? I flattened the 2d matrices into 1d arrays with shape (50000, ) and applied PCA on the (25000, 50000) dataset, I am getting around 85% explained variance for 400 components. Are PCA and LDA still good for such high dimensional datasets? (Other similar questions on stack do not address this problem)
Is there any other way I can extract 1 dimensional features so I can apply models like SVM, XGBoost? (for example: CNN to extract flattened 1-d features is one way)


texts: Classification of OLS regression coefficients

texts:
A variable $A$ (reaction time) is log-normally distributed, i.e. $\log(A) \sim \mathcal{N}(0,\sigma^2)$ and is linearly dependent of $n$ variables $X = (X_1,\ldots,X_n)$, i.e.
\begin{align}
A &= \alpha_1 X_1 + \alpha_2 X_2 + \ldots + \alpha_n X_n + \epsilon,
\end{align}
where $\epsilon \sim \mathcal{N}(0,\tau^2)$. An observation of variable $A$ belongs either to class 1 or to class 0, denoted as $A^1$ or $A^0$. I splitted my training observations on $A^1$ and $A^0$ (balanced ratio) and used an OLS model to find two sets of coefficients $\hat{\alpha}^1$ and $\hat{\alpha}^0$ such that $\hat{\alpha}^1 X$ and $\hat{\alpha}^0 X$ estimate $A^1$ and $A^0$ with minimal errors.

Now, I want to make a class prediction for new observations $A,X$. How to classify such an observation?

One way forward could be a Bayesian approach, since I know the class priors $p(D^i)$ equal roughly $0.5$. But how to compute the conditional class distributions $p(A|D^i,X)$ and the posterior $p(\hat{\alpha}^i|A,X)$ $i=0,1$?

What would you suggest?


texts: How to improve model performace when model shows a systemic pattern in residues

texts:
I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots when I plotted residues. My model is overpredicting for small target value (near zero) and underpredicting for large target value (near 1). How can I improve my model performance? The model is not overfitting and I'm doing an exhaustive hyperparameter search and basic feature engineering.
I'm trying to understand why I have a systemic pattern in residuals and how to address them. I went through some resources on residuals here with limited success.
I suspect the model is trying to do its best in the region with most samples (~0.2-.5) and neglecting other areas. Assigning higher weights to regions with low-performance doesn't seem to help much.
n_samples = ~10k
Target mean = 0.3457


texts:
I would like to test an experimental algorithm for string classification. More precisely, the dataset should be split into a set GOOD of good strigs and a set BAD  of bad strings. The algorithm should learn a model that is consistent with the dataset, in the sense that the output model always answers Yes for strings in GOOD, No for strings in BAD. The algorithm can behave arbitrarily on strings that are neither in GOOD nor in BAD.
Note that I'm not interested in prediction for strings not in the dataset. The only important thing is to obtain a model that is consistency with the given dataset. What makes the problem interesting is the fact that the model can be much smaller than the dataset. For instance, if the model is an automaton, then in some cases one can find automata that are exponentially smaller than the number of bits necessary to represent the dataset.
I would like to test our heuristics against known algorithms/datasets.

What are standard datasets used for testing new experimental heuristics in this area?
What are the standard tools used for such classification tasks on strings.

Note: The goal is to classify strings, not large texts. In particular, a typical size for the strings being considered is 1000 characters (at most). Additionally, one can assume these strings have the same length.


texts:
I've been doing some research on ensemble learning and read that for base models, model with high variance are often recommended (can't remember which book I read this from exactly).
But, it seems counter-intuitive because wouldn't having base models with low variance(doing good on test set) be better than having multiple bad base models?


texts: How to select the learned model using $k$-fold cross validation?

texts:
Let us consider a case where $1000$ data is given, i.e., the data set $U=\{x_1, \ldots, x_{1000}\}.$

When we want to use $k$-fold validation scheme, we first divide the data set into $k$ groups.

With out loss of generality, the parameter $k$ is assumed to be $10$.

Hence, we have $S_1=\{x_1, \ldots, x_{100}\}$, $S_2=\{x_{101}, \ldots, x_{200}\}$, $\ldots$, $S_{10}=\{x_{901}, \ldots, x_{1000}\}$.

I can obtain models, $f_k$, by learning a data set $U \setminus S_k$ for $k=1, 2, \ldots, 10$.

I can obtain error rates, $r_k$, by testing a data set $S_k$ with $f_k$.

Hence, I can obtain the error rates, $r$, by averaging $r_k$'s, i.e., $\sum_{k=1}^{10}r_k/10$.

I understand the $k$-fold cross validation so far.

Most materials I've seen just say the error rate averaged by $k$ scenarios in $k$-fold cross validation. However, they do no t say about $f_k$'s.

However, in this case, which model do I have to use among all $f_k$'s?


texts: Correct LSTM model to predict shuffled data

texts:
For a year I've been collecting data from my RPi:
[0 core load, 1 core load, 2 core load, 3 core load, environment temperature, fan speed, CPU temperature]
Now I want to build a model with Keras and LSTMs to make a prediction of core temperature based on a vector
[0 core load, 1 core load, 2 core load, 3 core load, environment temperature, fan speed].
The main problem is that I can not compose this data into time series, because I've sorted it by 0 core load unintentionally. So this data can be considered as shuffled.
Current model does not work at all:
model = Sequential()

model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(LSTM(128))
model.add(LSTM(128))

model.add(Dense(1, activation='sigmoid'))

Training summary:
Epoch 1/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140033.1094 - val_loss: 119426.7812
Epoch 2/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140032.5156 - val_loss: 119426.7812
Epoch 3/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140034.2500 - val_loss: 119426.7812
Epoch 4/128
6483/6483 [==============================] - 45s 7ms/step - loss: 140040.8438 - val_loss: 119426.7812
Epoch 5/128
5071/6483 [======================>.......] - ETA: 9s - loss: 140302.2812

How can I predict such shuffled data with LSTM? I'm currently stuck.


texts:
I'm studying PCA and my professor said something about finding the linear regression by doing the dot product of both axis. Could someone explain to me why? The dot product returns a number. What's the relationship between that number and the linear regression?
In my example, I have two vectors
$stat\_grade = [0,1,3,7,10]$
$physics\_grade = [1,5,8,10,10]$
The first step is normalizing them:
$ \frac{stat\_grade - mean(stat\_grade)}{std(stat\_grade)}  = [-1.69131435 -0.52489066  0.34992711  0.93313895  0.93313895]$
$ \frac{physics\_grade - mean(physics\_grade)}{std(physics\_grade)} = [-1.11613741 -0.85039041 -0.3188964   0.7440916   1.54133261]$
Computing the dot product between the normalized vectors will return $4.355129045906131$
I can't understand how a single number can help me to find the linear regression
Thanks for the response!


texts: Alternative distance to Dynamic Time Warping

texts:
I am performing a comparison among time series by using Dynamic Time Warping (DTW). However, it is not a real distance, but a distance-like quantity, since it doesn't assure the triangle inequality to hold.

Reminder:d:MxM->R is a distance if for all x,y in M:

1 - d(x,y) ≥ 0, and d(x,y) = 0 if and only if x = y
2 - It is symmetric: d(x,y) = d(y,x)
3 - Triangle inequality: d(x,z) ≤ d(x,y) + d(y,z)

There is any equivalent measure that ensures the condition of distance in a matemathical sense? Obviously, I am not looking for a Euclidean distance, but one that ensures the proper classification of my series in a future clustering.
If so, there is any solid implementation in a R or Python package?


texts: Significant testing - repeated observation over multiple days

texts:
I work in mobile gaming, and want to analyze A/B test groups, but I believe I'm introducing errors in my calculations.

The metric I'm looking at is:
num of unique players who engaged in battle that day/ num of unique players who were active that day.

I currently have my data that with each row as active players for the day:
date, player_id, group A/B, boolean 0,1 if engaged in battle that day
I split the groups A/B and take the array of booleans and do a t-test.
I feel that I'm overstating the sample size.

For example, I might have 1,000 people in Group A, and each day 300 are active from that group, of those 300 most of them are the same 200 people who are active everyday.
I'm feeding 300 active players x 7 days = 2,100 rows of data into a t-test, but most of this is data from the same 200 people.

What is the correct way to do this?

Thank you!


texts: How do you calculate how dense or sparse a dataset is?

texts:
I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf

In order to select which CF algorithm should be used the paper refers to the density of the dataset. What it doesn't do is explain how you actually calculate the density of your dataset.

So in the context of that above paper can anyone help explain to me how I would calculate the density of a dataset? The paper refers regularly density in the 1-5% range.


texts: Using pre-trained word2vec vs re-training a word2vec model

texts:
I am relatively new to using word2vec. I am interested in solving the topic-word intrusion introduced here by using the vector spaces of words generated by word2vec and SVC.
I have a corpus with a vocabulary of 8000 words, the vocabulary is perfectly contained in Google's word2vec trained model. I was wondering which model would provide a better representation of the words, the pre-trained model on 3M words or a model trained only on the 8000 words appearing in my corpus?


texts:
Say you have a binomial distribution with $p$ very small ($\approx 0.001$).

You are asked to predict the conditional success rate $SR=S/T$ with $S$ successes out of $T$ trials given a set of conditions $X$.

One would expect (correct me if I'm wrong, though I ran simulations and am quite confident) $SR$ to have a downward with increasing $T$ when $T$ is still small, and to approach $p$ when $T$ is large.

The distribution of trials and successes is not uniform across $X$, so the train set tends to give higher $SR$ to conditions with smaller $T$ values.

How would you treat this data skew when constructing a model?


texts: Image segmentation with large class imbalance leads to zero precision/recall

texts:
I have a binary semantic image classification problem where only very small parts of the images are positive, most of it is negative. In the training data I have a positive rate of around 0.023, which is a large imbalance (factor 43). Now when I train, the output is just that every single pixel is negative, which leads to a high accuracy. However, it does not locate the particular symbols that I want to find on the pictures. Increasing the training data won't remove this imbalance because the glyph I want to find covers less than half of the bounding box, therefore even an image full of that glyph would be imbalanced.
I've read that one can set weights for the categories, but Keras Model.fit does not support class_weight when the output has three or more dimensions (samples, pixels, classes) but only when classifying samples as a whole (samples, classes).
Is there a way to reduce the false-negatives and increase the true-positives?


texts: Ignore unseen columns with OneHotEncoder

texts:
I'm working in a problem in which I'm OneHot encoding a set of feaures from a dataframe, for instance:

from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y)
oh = OneHotEncoder(handle_unknown='ignore')

print(X)

      a     b     c
1    one    m     y
2    two    m     n
36   three  f     n
113  one    f     n
31   two    m  other
....

oh.fit(X_train)

However, it could be that not all features are present in the test set. For this example, say I only have the two first columns. The encoder, in this case will raise an error:

oh.transform(X_test.loc[:,:'b'])

  The number of features in X is different to the number of features of the fitted data. The fitted data had 3 features and the X has 2 features.

Is there some way around this? Ideally I'd like to have the missing columns, either ignored (not present in the output), or set to NaN.


texts: Object detection stops predicting well according to how I collect the images

texts:
On my problem I have to use 4k resolution (3840x2160) to record images and detect qrcodes at the same time. The images I end resizing it to 1920x1080 to save it on disk.
The problem is that if I train using this resized data the model stops predicting well compared to when I record the video on native 1920x1080.
I can't have good results nor if I predict with resized to 1920x1080 neither native 1920x1080 with this train.
If I train using native recorded at 1920x1080 it predicts well both with native 1920x1080 and resized 1920x1080.
Is there some explanation about why this is occurring?
I always try to save the images as jps with quality=95 and the camera I'm using compresses the image before sending to opencv (directly affects the fps).
I can't go back to 1920x1080 because the qrcode reader stops detecting.
How I record the video without resize (working method):
for z, frame in enumerate(frames):
    try:
        self.out.write(cv2.cvtColor(frame2, cv2.COLOR_BGR2RGB))
    except AttributeError:
        self.out = cv2.VideoWriter(ANNOTATION_RECORD_VIDEO_FILE, cv2.VideoWriter_fourcc('m','p','4','v'), 60, (conf.cameras.phase1[0].resolution.width, conf.cameras.phase1[0].resolution.height))
        self.out.write(cv2.cvtColor(frame2, cv2.COLOR_BGR2RGB))

With resize (frames are coming with 4k res):
for z, frame in enumerate(frames):
    frame = cv2.resize(frame, dsize=(conf.annotation.resolution_to_save['width'],
                                        conf.annotation.resolution_to_save['height']),
                                interpolation=cv2.INTER_CUBIC)

    try:
        self.out.write(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    except AttributeError:
        self.out = cv2.VideoWriter(ANNOTATION_RECORD_VIDEO_FILE, cv2.VideoWriter_fourcc('m','p','4','v'), 60, (conf.annotation.resolution_to_save['width'], conf.annotation.resolution_to_save['height']))
        self.out.write(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))


texts: How to determine my pacf and acf in ARIMA with daily time series data?

texts:
I have daily time series data for revenue sales ranging from 2018-04-01 to 2021-02-19. I want to implement an ARIMA model and the plot of the data looks like this:

Based on my augmented-dickey fuller test, the original data seems stationary

To determine my p and q values, I plotted my pacf and acf plots but the first value to go above the confidence band is around ~92 for both. I know this has a quarterly trend (3 months) so it makes sense that I see this pattern, but it's taking far too long to run in my model and I wanted to know if I was analyzing the plots properly.

Here is the code I'm using to implement the model:
model = sm.tsa.statespace.SARIMAX(train,
                                 order = (92,0,92),
                                 enforce_stationarity = False,
                                 enforce_invertibility = False)
model_aic = model.fit()
print(model_aic.summary().tables[1])


texts: Trying to implement a "smart compose" feature

texts:
I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself.

https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html

The text is super vague though - the only part that gives any direction at all is "we combined a BoW model with an RNN-LM".

I remember that BoW is an unordered representation of the input corpus where only relative magnitudes are important. But I just don't have enough experience to know how an unordered representation like this could help me predict an ordered suffix.

Does anybody have any thoughts on how this might more specifically work? Or any references for further reading? I wasn't able to find much unfortunately.


texts: Seminal papers about analysis and application based on publicly released data

texts: The US government releases a lot of varied data here. Data science and data analytics have greatly matured over the last decade varying from Cambridge Analytica on one extreme to Google/Amazon Ads recommenders on another. A great number of startups have sprouted with regards to crunching big data which is already growing bigger every minute and computing some models using which prediction or recommendation services are designed. I am interested in knowing about seminal academic papers where researchers picked a societal dataset and their analysis launched big ideas and startups or at least gave novel, intelligent insights into societies. Could you list some such papers or guide where to look?


texts: Why is ReLU used as an activation function?

texts:
Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.

Which I am able to understand intuitively for the activation functions like sigmoid.

I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation. However, I am not able to understand why is ReLU used as an activation function if its output is linear?

Doesn't the whole point of being the activation function get defeated if it won't introduce non-linearity?


texts: in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

texts:
general Infos about my dataset: I have 40k data points and 5 features.
I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS is making an error of 10 meters and you want to correct it. So I brought another super GPS which is very accurate and measured 40k data while driving so in my dataset I have some vehicle informations which are speed, Acceleration, yaw rate, timestamp and wheel angle and I have position Informations which are the ground truth longitudes latitudes and the false longitudes and latitudes from my normal GPS. I'm transofrming those latitudes and longitudes to an x and y just to know how much should I shift my false longitudes and latitudes so that my position can be more accurate and similar to the Ground truth values. Can my data be bad in this case? I'm trying to predict the error in longs and lats that the GPS make so that I can later correct it so it's a regression problem and I'm using those features above to do that which I think they are informative since speed, acceleration, yaw rate and wheel angle are related somehow to a position(am I wrong?)

I'm asking this generally, I read some articles in the internet, that say that data is sometimes bad or the quality of the data is bad but I don't know what the mysterious sentence really mean.

I also had the problem when training neural networks that my loss start to decrease for the first 10-20 epochs and then it stuck on some high value and the network stops learning like if it were struggling to go out of that loss value but it can't. I tried to use only 100 data points instead of all the 40k and I noticed that it worked good, the NN achieved to fit those but as I increase the number of data points the performance become worse(do you have any ideas on this?)

some people suggest that I don't have many data and many features and in this case it would be better to use some machine learning approach since it outperforms NNs in case of small datasets or if I have few features like in my case so I also tried using random forest and I noticed that it gives better results than neural networks but it also doesn't generalize well, even if it gave me good results on train and validation sets, when I try it on test data(data that the random forest have never seen), it perform really bad.

so I was reading in the internet what can cause those problems and I noticed that I sometimes saw people or articles that claim that maybe the quality of the data is bad! but what does this really mean? I thought neural networks can map any kind of data, if I have one feature and one target then neural networks can map those two together, at least it can overfit the data right?

so can someone please tell me what is bad data or better how do I know if my data are bad? well if there is a way to know that then I would probably save time and not start working on a project that will take me a month to complete and then figure out my data is bad. Also can you tell me whether my case make sense ? I mean I find it weird that NNs gives very bad performance way worse than random forest. at least my NN should overfit the data or am I wrong?


texts: Nearest neighbors search for very high dimensional data

texts:
I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. Given the size of my dataset and some initial tests I performed, my assumption is that the method I will use will need to be either parallel or distributed. So I'm considering two classes of possible solutions: one that is either available (or implementable in a reasonably easy way) on a single multicore machine, the other on a Spark cluster, i.e. as a MapReduce program. Here are three broad ideas that I considered:

Assuming a cosine similarity metric, perform the full multiplication of the normalized matrix by its transpose (implemented as a sum of outer products)
Using locality-sensitive hashing (LSH)
Reducing first the dimensionality of the problem with a PCA

I'd appreciate any thoughts or advices about possible other ways in which I could tackle this problem.


texts: When to consider examples as duplicates in a classification problem?

texts:
I'm trying to train a model for a binary classification problem and I'm trying to understand if my model is biased from my data or if my results are valid.
I have a pandas dataframe as output of my preprocessing/cleaning/transforming phase.
This dataset is composed by the following columns:
[ID, Column_X, Column_1, Column_2,....,Column_n, Target]
For a single ID there are multiple examples and these examples differ only for Column_X.
When I train the model (after the split phase in train/validation/test) I use all the above columns without the ID and Target ones.
Should I consider the examples with the same ID as duplicates and then my results on test set invalid due to the fact that the model has probably already seen a very similar example (different only for the column Column_X) during the training phase?
I am currently using CatBoost algorithm because it seems to have the best results after having tried with also Logistic Regression, Random Forest and Decision Tree algorithms.


texts: I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I need character-level embeddings. The only character-level embedding library I've been able to find is chars2vec which does not seem well maintained. Is there a way to get character-level embeddings with either spacy or a more popular package than chars2vec?


texts: Visualization of three-dimensional report

texts:
I have a visualization problem.

Creating a comparison report of PR event efficiency. Say, show or exhibition.

There are two dimensions of comparison:

compare vs the same event performance in the past years
compare vs another type of analogical/competitive events

There is also a number of comparison aspects:

Audience
Media Coverage
Social Buzz
ROI
.... etc

Each aspect is a set of some final KPI-s (just numbers, which can be compared vs another "dimensions"), plus maybe some descriptive text and pictures (which couldn't be a metric but should be attached to the report).

So finaly it looks like a three-dimensional coube:

Years
Another Events
Aspects

If I put it in plain Word or PPT it will look like a document with dozen of slides/papers and linear structure.

Any ideas how to compile an elegant user-friendly report?


texts:
I'm using Flask where i load some pre-trained machine learning models once. I'm also using Gunicorn usually with 2 or 4 workers to handle parallel requests.

Every request contains some texts that i want to analyze.

I'll explain my problem with a example:
My Flask server with Gunicorn and 2 workers is up and loads my models once for every worker.
Then i send two parallel requests. The first will run analysis on the 1st worker with 500 texts and the second on the 2nd worker with 2000 texts.
The problem is that the second request will stop the analysis after some period of time and reload the models for this worker.

Does Gunicorn contains a default worker timeout and how it can be solved?


texts: Should I create a tfidf on a subset of a data set or use the whole corpus?

texts:
My goal in this project is to see if businesses on a list are currently customers within my organization.  One piece of this involves producing a similarity score using cosine similarity on the names of the businesses

My initial process uses TfidfVectorizer  from sklearn.feature_extraction.text to tokenize the names across all of the customers.

Is it incorrect to subset my organization's customers to those that are in the same state as the customers in the new list?

The next question is should I produce a tfidf across all of the customers within the organization, store the data, and build a new one at runtime that incorporates the names from the new list (as described in this answer)? Or should I only consider the customer names in my organization that are in the same state as those on the new list?

Thank you for reading


texts: Product of dot products in neural network

texts:
In a neural network, it is common to compute a dot product of the form

$$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$

and use it as argument to some activation function. This is done for several different weights $w$ and activation functions (one per neuron). The results are used as input to the next layer, where we compute again several dot products and use them as argument to more activation functions.

Using many layers leads do several theoretical difficulties of deep learning and one has to make some effort to make everything works. But at the end, hopefully it will end up working.

I was thinking in a way of "cheating" the use of many layers. The idea is to use just one layer in a way that the argument to the activation function is complex enough so the program will learn properly. Instead of a dot product I was thinking in using products of dot products:

$$\prod_{k=1}^L\langle w^{(i)}, x \rangle = (w_1^{(1)} x_1 + w_2^{(1)} x_2 + \ldots + w_n^{(1)} x_n) \cdot \ldots \cdot (w_1^{(L)} x_1 + w_2^{(L)} x_2 + \ldots + w_n^{(L)} x_n)$$

The result is them used as argument to some activation function. Since this argument already has several weights, my hope is that the need of several layers of activation functions wouldn't be necessary anymore. I even though about using sum of these products and tested it. I'm getting $\sim 80\%$ of accuracy in a classification problem where I know it is possible to get more than $90\%$. Does my approach has a fundamental limitation I'm not aware of?

Thanks.

Extra: Above I gave a simplified description of my idea (which I think is enough), but it may be helpful to put the whole thing here. My final model of learning is given by the following formula

$$ = \textbf{f}\left( \sum_{r=1}^R \left( \prod_{i=1}^{L-1} \langle \textbf{w}^{(i,r)}, \textbf{x} \rangle \right) \left[
\begin{array}{c}
    w_1^{(L,r)} \\
    \vdots\\
    w_m^{(L,r)} \\
\end{array}
\right] \right) = $$

$$ = \left[
\begin{array}{c}
    f\left( \sum_{r=1}^R \sum_{i_1, \ldots, i_{L-1}=1}^n w_{i_1}^{(1,r)} \ldots w_{i_{L-1}}^{(L-1,r)} x_{i_1} \ldots x_{i_{L-1}} w_1^{(L,r)} \right) \\
    \vdots\\
    f\left( \sum_{r=1}^R \sum_{i_1, \ldots, i_{L-1}=1}^n w_{i_1}^{(1,r)} \ldots w_{i_{L-1}}^{(L-1,r)} x_{i_1} \ldots x_{i_{\ell-1}} w_m^{(L,r)} \right) \\
\end{array}
\right]$$

$f:\mathbb{R} \to \mathbb{R}$ is the activation function and $\textbf{f} = (f, \ldots, f)$ is the "vectorized" version of this function. Note that I tried to come up with a very flexible formulation before the activation function. The idea is that this would avoid the need of more layers. I hope everything is clear at this point. Thanks again.