texts: What could be a dataset in which the presence of an outlier dramatically affects the performance of Ordinary Least Squares (OLS) regression?

texts:
I am tasked with giving an example of a dataset in which the presence of an outlier dramatically affects the performance of
Ordinary Least Squares (OLS) regression. I've searched and searched the web and I understand that OLS has a hard time dealing with outliers, but I'm having a hard time figuring out why, and finding a dataset to prove this.


texts:
I’m currently building sequence models for forecasting, and have tried using RNNs, LSTMs, and GRUs.

Something unusual I noticed was the highly unstable loss curves, where the loss sometimes goes back to the loss level in the first few epochs. Interesting, the severity of this decreases from RNNs to LSTMs to GRUs.

Would anyone have an idea why this occurs?

For reference, here are the loss curves for the following models, across 500 epochs.


texts: How can I classify text considering word order, instead of just using a bag-of-words approach?

texts:
I've made a Naive Bayes classifier that uses the bag-of-words technique to classify spam posts on a message board. It works, but I think I could get much better results if my models considered the word orderings and phrases. (ex: 'girls' and 'live' may not trigger a high spam score, even though 'live girls' is most likely junk). How can I build a model that takes word ordering into account?

I've considered storing n-grams (check-out-these, out-these-live, these-live-girls), but this seems to radically increase the size of the dictionary I keep score in and causes inconsistency as phrases with very similar wording but different order will slip through.

I'm not tied to Bayesian classification, but I'd like something that someone without a strong background in statistics could grok and implement.


texts:
From what I have read, Denoising during preprocessing for image classification tasks seems to be a bit controversial.
While on one hand it might improve classification accuracy, the computational complexity seems to discourage a lot of people. Rather I've seen a lot of claims, but hardly anything I can call concrete evidence, which argue that noise "would not matter" for a sufficiently deep neural network.
That being said, assuming that I have decided to perform some kind of Denoising on my data, prior to classification, I'd like to ask what are some good practices?
I've recently gotten into auto-encoders (and their potential in denoising) and I've seen models which either derive features from encodings (in deeper layers) or concatenate encoded features on top of the original data. So essentially, one approach is to embed the "denoiser" into the classification network itself.
Are there any other techniques I can look into? I'm asking in general, so feel free to answer on any other ML models which aren't neural network-based. Thanks in advance


texts: What does coefficient mean for a binary independent variable in multiple linear regression?

texts:
Let's say I have a multiple linear regression model where my dependent variable, Y, is an integer. And, one of my independent variables --x1-- is binary --let's say either 0 or 1.

We know that sign of the coefficient for x1 in the model, positive or negative, demonstrates its correlation with Y. My question though is, is there anyway for us to know that what the correlation of x1 values independently, 0 or 1, is with the dependent variable? For instance, do we know if that 0 is the reason for the sign of coefficient and the more 0 will lead to more Y (not sure if this is the correct language but hope you understand what I am saying.)

A real-world example, I have a multiple linear regression model to find the correlation of a set of numeric indexes in tweets, as independent variables, with the number of retweets as dependent variable. One of the independent variables in this model is the tweet's fact checking label, "true" or "false" which we call it "truth_label." I have trained a multiple linear regression model and the coefficient of the truth_label is positive meaning that it has a positive correlation with the number of shares. But, I would like to know if that positive correlation is because of the "true" values or "false" values. Hope this example made my question a bit more clear.


texts: Training Object Detection model on just 10 images

texts:
I am trying to train an object detection model using Mask-RCNN with Resnet50 as backbone. I am using the pre-trained models from PyTorch's Torchvision library. I have only 10 images that I can use to train. Of the same 10 images, I am using 3 images for validation. For the evaluation, I am using the evaluation method used in COCO dataset which is also provided as .py scripts in the TorchVision's github repository.
To have enough samples for training, I am oversampling the same 10 images by a factor of 100 i.e. I end up with 1000 images that I can use to train my model. Similarly, I end up having 300 images that I can use for validation.
Now, the problem is that I am getting 0% mAP after train and 0% recall. I have two questions:
Q1. Why would it return 0% mAP?
If it has something to do with the fact that I am oversampling to a large extent, then my next question is
Q2.  Shouldn't the oversampling just cause the model to Overfit and instead provide a higher training as well as validation accuracy for my case (since I have picked the validation data from the training data itself?


texts: Classifying objects based of a varying number of the same type of feature vector for each object

texts:
For a congressional session, I have created a doc2vec model of speeches made. Using the vectors from this model, I have a dataset of each congressperson, their political affiliation, and a list of the vector representations of each speech they made. Each of these document vector representations is a 300 element vector.

I am now trying to classify each congressperson by party using these document vectors representing their speeches. So far I have tried using the mean vector for each speaker, but I was looking for ways to use the whole set of vectors.


texts: What is the best way for synthetic data generation while maintaining privacy?

texts:
For one of the projects where we are working as third party contractors, we need a way for the company to share some datasets which can be used for data science. It is not possible for the company to share the real data as that would be a privacy issue.
We are exploring ways so that the company can either share the data while maintaining privacy or else ways to generate fake data that matches the statistics/demographics of the actual data.
We are currently looking at a couple of options:

Using differential privacy to add noise to the data and then sharing the transformed data with us. Can this approach lead to any privacy issue? I am concerned about reverse engineering. Does "privacy budget" apply here? How should it be tackled?
Using encoder-decoder neural networks to learn vector embedding of the real data. Once the vector embedding is learned, the decoder can be destroyed and the encoder's output can be shared with us.

Is there any other approach that can be used for synthetic data generation that resembles the actual data in terms of demography and statistics. Or else what would be the best way to access the real data without violating privacy?


texts: How can I plot a line for time series data with categorical intervals in R

texts:
I am working with single time-series measurements that I want to plot for the time window of about 1 week.
This is the data I am working with.
This is my R script:
library(tidyverse)
library(ggplot2)

filesource <- "C:/ ... /testData.csv"
df <-read.csv(filesource, header = TRUE)

ggplot() +
  geom_line(data = df, aes(x = date, y = value, group = 1), color = "red") +
  ggtitle("Some Measure over Time") +
  xlab("Time") +
  ylab("Some Measure in %")

This produces this plot.

What I want is to have the individual unique weekdays show on the x-axis like this, as if I would plot the days as individual categories but only show the first one of each day. I cannot really hardcode this, because I am working with different participants, days and value amounts per day.
Expected Outcome:

So I created a new variable with the weekdays:
df$day <- weekdays(as.Date(df$date, '%d-%m-%Y'))

However, when I want to use this column as the x-axis variable, The days are not in the right order, and all values of one day obviously get plotted on top of each day:
geom_line(data = df, aes(x = day, y = value, group = 1), color = "red")

I have seen this being somewhat solved in python: Visualizing Time Series Data
However, I really want to use R and Markdown to create automated participant reports.
If this is easier to accomplish with another plotting function in R, I am open ears. I just like the customizability of ggplot.
I hope my example is clear. I guess this can be solved with the right ggplot() parameters and settings.
Does anyone have a solution for ending up with something more like the expected outcome montage?


texts: Very basic question: what is an accepted term for "linear order distance"

texts:
In data science we have "Manhattan Distance" as a slang term for Level 1 Distance and "Euclidean Distance" as a slang term for Level 2 Distance. Is there an accepted term for linear distance in memory of cells in different rows in a matrix? That is, given an 8x8 matrix, the "linear distance" from [3,2] to [4,5] is:
(4-3) * 8 + (5-2)

This is the distance in memory addressing. "Level 0 Distance" doesn't really work as a technical term, because computer memory layout is not a native concept in mathematics. Is there a standard term for this? It reminds me of the old flyback transformers in cathode ray tubes, so "flyback distance" makes sense as a slang term.


texts:
In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot.

The value prop is that you move the low value (and less used) data from EDW to Hadoop and then save on expensive EDW storage by using Hadoop.

So in my company say we keep 10 years of data on EDW and say we don't use anything below 2 years very actively. So I move 8 years of data from EDW to Hadoop.

I can setup an Impala (or equivalent) product to query the 8 years of data as well.

But the problem is how do you order, sort a query which requires some data form EDW and some data from Impala?

because this kind of grouping, sorting ordering etc will have to be done in memory and the apps which queried the EDW were not written for operations and don't have the capacity as well to sort, group and process so much of data.

So how are people implementing the Active Archive?


texts: Python library to implement Hidden Markov Models

texts:
What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before.

Alternatively, is there a more direct approach to performing a time-series analysis on a data-set using HMM?


texts: Is the usage of the "momentum" significiantly superior to the conventional weight update

texts:
The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller).
Is it significiantly superior?
Weightupdate:
$$
w_{i+1} = w_i + m_i
$$
With momentum:
$$
m_0 = 0
\\
m_1 = \Delta w_{1} + \beta m_0 = \Delta w_1
\\
m_2 = \Delta w_{2} + \beta m_1 = \Delta w_2 + \beta\Delta w_1
$$
So the momentum already contains the actual weightupdate and the momentumhistory.
beta is like alpha a number between 0 and 1 (beta diminishes older momentum items).
Is there a common consent, that the usage of the momentum approach improves the learning quality, in terms of stability and speed?


texts:
I want to calibrate probability outputs of a model. I'm using Isotonic Regression. After calibration, when I called calibration_curve function of sklearn calibration module I got this error: ValueError: 'list' argument must have no negative elements. However when I checked results that I obtain from Isotonic Regression there are no negative values and all values are in [0,1] range. There are no problems in targets too.
from sklearn.isotonic import IsotonicRegression

i_reg = IsotonicRegression().fit(X_train, y_train)
res = i_reg.predict(X_test)
prob_true, prob_pred = calibration_curve(y_test, res, n_bins=200, strategy="quantile")

Error log:
ValueError                                Traceback (most recent call last)
<ipython-input-36-5a2bd10e64b2> in <module>
----> 1 prob_true_model_test , prob_pred_cal_test = calibration_curve(y_test, res, n_bins=200, strategy="quantile")

/environment/lib/python3.6/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     70                           FutureWarning)
     71         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 72         return f(**kwargs)
     73     return inner_f
     74

/environment/lib/python3.6/site-packages/sklearn/calibration.py in calibration_curve(y_true, y_prob, normalize, n_bins, strategy)
    590     binids = np.digitize(y_prob, bins) - 1
    591
--> 592     bin_sums = np.bincount(binids, weights=y_prob, minlength=len(bins))
    593     bin_true = np.bincount(binids, weights=y_true, minlength=len(bins))
    594     bin_total = np.bincount(binids, minlength=len(bins))

<__array_function__ internals> in bincount(*args, **kwargs)

ValueError: 'list' argument must have no negative elements


texts: .h5 file size is same before and after training?

texts:
learner = ConvLearner.pretrained(arch, md, ps=0.5) #dropout 50%
learner.load('ResNet34_256_1-2')
learner.fit(lr,1)
learner.save('ResNet34_256_1')

h5 file in load and save is having same size. Should it increase after training?
How do I know that saved model is better than loaded on?


texts: Is there any optimal way on feature selection for more than one classification algorithms?

texts:
I have a wine dataset with 13 features that indicates 3 different wine classes (target), and k-NN, SVM with linear kernel and SVM with rbf kernel algorithms to be tried with this dataset.

My goal is to obtain the best classification accuracy, and to obtain this accuracy:

Which classification algorithm (kNN, SVM with linear kernel or SVM with rbf kernel) should I choose?
Among all the features, which ones of them (based on backward elimination, maybe according to p-values) should be chosen?

I have thought of using GridSearchCV with 3 estimators for the algorithms above. But in this case, the problem is feature selection part as you guess. Is there any optimal way to achieve both? Thanks!


texts: Limitations of StempelStemmer, Polish stemmers

texts:
I am trying to understand what is an acceptable output of a stemmer.

I have tried to stem a few words using StempelStemmer, an algorithmic stemmer for Polish from Apache Lucene. However, it looks it cannot handle some words properly, e.g.

  joyce -> ąć

  wielce -> ąć

  piwko -> ąć

  royce -> ąć

  pip -> ąć

  xyz -> xyz

I'm surprised it cannot handle Polish words ("wielce", "piwko" and "pip"). Is this a limitation of (a) the stemming algorithm or (b) the training dataset the algorithm used or (3) something else?
I am surprised that for non-Polish words it returns "ać". I would
expect that for words it has not be trained for it will return their
original forms, as it happens, for instance, when stemming words like
"xyz". I understand that compared to lemmatizers, stemmers are more crude: they can cut too much. But it looks like instead of cutting it introduces completely different word. Is it a bug?


texts: Econometrics Thesis methodology Suggestions?

texts:
I just picked my topic for my econometrics course thesis. I decided to create an algorithm that forecasts NBA players' performance. My inspiration is Silver Nate's PECOTA and CARMELO algorithms, but I have no empirical experience with Data Science techniques, so I am trying to avoid any misleading transitions from theory.

The algorithm is going to compare the performance of a current player with other historical players and then forecast his performance based on the of his "neighbors".

So, I'm going to pick all player statistics per season. Then, I'll create indexes to establish informative dimensions for kNN algorithm (eg. Offense, Defense, Physics). As next step I thought to use naïve Bayesian algorithm to find the likelihood of having the same performance with his neighbors and then pick the weighted (depended on likelihood) sum of the neighbors' stats as a forecast.

Empirically speaking, do you see any bias on this methodology?

Thank you in advance


texts:
I am trying to build a distance matrix for around 600,000 locations for which I have the latitudes and longitudes. I want to use this distance matrix for agglomerative clustering. Since this is a large set of locations, calculating the distance matrix is an extremely heavy operation. Is there any way to opimize this process while keeping in mind that I am going to use this matrix for clustering later. Below is the code I am using.

from scipy.spatial.distance import pdist
import time
start = time.time()
# dist is a custom distance function that I wrote
y = pdist(locations[['Latitude', 'Longitude']].values, metric=dist)
end = time.time()
print(end - start)


texts: Creating queries on the fly and general manipulation for dataset of half a million data records

texts:
What would one use for manipulating data of the kind below ?

a) Data is bio-markers of different globs

GlobA 3 4 5 ....
GlobB 2 1 1 ....
GlobC 3 2 1 ....

b) Manipulations are queries like:
- show me the Globs where average of each Glob in file Efficiency is greater than 50%
- or show me sorted Glob list where first and second sort criterion are xx and yy)
- Construct a chart of globs that differ in criteria x by integer 3 (show me globs whose average over 5 runs for Structure and Efficiency differ from their nearest neighbors by 3)

Currently, this data is stored in a 100MB Excel file that is painfully slow to load on the speediest computer our lab can afford.

Ideally, there would be some open source program that accepts csv files of this data and has ability for user to construct queries that can be stored in a library for easy pulling up, charting abilities would be great too.
Here are 2 files (real data would be around 40 such files each file containing 20K rows):

Efficiency File:

Glob,Run1,Run2,Run3,Run4,Run5
SigX,6.2,4.8,2.4,4.32,5.59
SigY,8.44,8.16,5.99,0.98,9.6
SigZ,0.00,0.00,0.00,0.01,0.20

Structure File:

Glob,Run1,Run2,Run3,Run4,Run5
SigX,3.2,3.8,2.4,7.32,6.32
SigY,2.4,5.16,6.99,0.98,9.6
SigZ,1.02,0.00,2.23,0.01,0.20


texts: Data set of vectors of SVG paths for digits


texts:
I am trying to understand kmeans clustering and I read a article where kmeans is used for clustering the features generated in network logs. This clustering is followed by a supervised classification.

They mention in article :

"Due to extensive repetition of input and output network ip addresses, portnumbers, and protocol information in the collected data, these were deemed scalar entities for analysis and excluded while clustering. The remaining attributes  were chosen as input to clustering  algorithm"

I fail to understand what scalar entities mean here? Does this mean that these input and output ip address and port numbers are not considered as features during feature extraction stage? If the ip addresses arent considered, how are the features mapped to the input data of network logs?


texts: Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

texts: I have a binary classification problem which in the test set, the number of data in both classes are equal (the test number of class 0 and class 1 are equal). Since we know  that the number of samples from every class are equal, I use median on the probabilities of logistic regression output (probabilities of samples for class 1) and map probabilities to zero and one and then calculate f1-micro and f1-macro. But they are absolutely equal and I don't know is it odd or not and why is this happening. I would be grateful if you have any idea what's going on and what is wrong.


texts:
I am trying to train a model to predict the location of a storm at a given time. The dataset includes the longitude and latitude of the storm at the given "timestamps" but I am not sure if that is the best way to represent the location as it doesn't likely have a linear relationship.

Is there a way to combine the longitude and latitude into a feature that can be used for training? I was thinking about creating "grids" to represent spaces but I'm not sure how I would go about creating these grids or converting them back to a long/lat range.

Any help is appreciated. Thanks.


texts:
I am working on a deep learning CNN project. The dataset contains more than 500 classes and the classes have different numbers of items (images). For example, some of the classes have 5 images and some of the classes have 10 and some of the classes have 20  images and some of the classes have more then 20 images.
Can I use this dataset to create the CNN model?
Should the number of the images in each class be the same number?
Note: I will use VGG to train the model.


texts: On interpreting the statistical significance of R squared

texts:
I have performed a linear regression analysis to two series of data, each of which has 50 values. I did the analysis in SPSS and as a result got a table which says that my adjusted R squared is 0.145 and its significance is 0.004.

Being 0.004 < 0.05, I assume my adjusted R squared is significant.

1) Does it mean my adjusted R squared is credible?

2) What does happen if you get a significance which is > 0.05? Does it imply the adjusted R squared can be trusted with credibility but also that the two datasets are not or poorly correlated?


texts: Getting a transition matrix from a Adjacency matrix in python

texts: I have a 3*3 Adjacency matrix and I'm trying to sum the elements of each column and divide each column element by that sum to get the transition matrix. Can you please help me code this part? Thanks in advance.


texts:
I am using a CNN to predict large changes in my target variable X. I am classifying several "set-up" states visually from my images.

I am only interested in big changes, or maybe no change. So, I can classify in 3 ways:

x is future period absolute change in X

Label BigChange when x > A%. Label Flat when x < A%.
Label BigChange when x > A%. Label Flat when x < B%, where B < A.
Label BigChange when x > A%. Label Flat when x < B%. Label SmallChange when x > B && x < A.

Which of these approaches is likely to give me the best predictive accuracy for BigChange?

  EDIT 1: Additional clarification

Input data is a set of images (60x60) with connection to Target variable (Float) X.

x is the delta between X values, and a big value might be 2-3% change. I am interested when the next x > some A (maybe 2%) i.e. there is a BigChange, but if x < B ( maybe 0.25% ) then I call that Flat. I want to classify my images so that I can predict a big change in x or when change is small. But I am NOT interested in predicting anything else (like changes > 0.25% but < 2%).

So, I am asking whether I should just classify and train on images that I have previously labelled BigChange or Flat. BUT this means when I run model on new data it will received lots of images that represent changes in x that are NOT Big or Flat. Is this problematic? Should I label things I  am not interested in aswell for completeness?


texts: How to improve my f1 score in stories analyze

texts:
I got an assignment to build a model that identify the gender of the text writer.
The assignment score will determine by my model f1_score, to get the maximum points, T need it will be at least 0.7.
I'm given a 360 stories and their writer gender, and another 150 to check on (without their gender).
I tried a few different ML modle, a few normalization method and a few stop words corpus. and yet to reach my destination.
if you have any tips/information I would appreciate it!!
** the stories languages is NOT English, but my native language (Hebrew)**


texts: How to visualise multidimensional categorical data with additional time dimension


texts: How does the seed value work in Weka for clustering?

texts:
I have recently been learning about the various clustering methods, and I decided to apply Furthest Point clustering in Weka with varying seed values.

The seed value dictates the initial choice of points in the data set. It can't be totally random, since repeatedly doing the clustering with, say, seed = 3 gives the same result.

How does it select data points based on the seed?


texts: How to use future holiday/promo days as input in multivariate LSTM sales forecasting

texts:
Im building a forecast using an LSTM in tensorflow 2.
My data consists of 7 columns: date (daily), gross_sales (the target), daily_total_inventory, avg_daily_order_value, daily_total_new_customers, is_holiday (1/0), is_promo_day (1/0).
My aim is to predict gross sales based on the historical values of past gross sales and all other variables (inventory, avg order value, new customers, whether it was a holiday, and whether there was a promotion going on).
However, I also have a planned schedule for future promotional days that I want my model to take into account when predicting gross sales. For example, every time there is a promotion going on, sales increase significantly compared to non-sale days. So when my model is making predictions, it needs to consider whether or not that future day will have a promotion or not. However I'm not sure how to shape this data, or appropriately load it into an LSTM.


texts: Is there any difference between a weak learner and a weak classifier?

texts:
While reading about decision tree ensembles Gradient Boosting, AdaBoost etc.
I have found the following two concepts weak learner and weak classifier.
Are they the same?
If there is any difference what is it?


texts: Finding synergies among observations of equal length

texts:
Assume we have a set $I$ with 20 different items (we call them $I_0$, $I_1$ up to $I_{19}$). Also we have $n$ observations $O \in I^{n\times 8}$; so each observation is a subset of $I$ with exactly 8 items and is labeled with a score.

Just as an illustration here are some made up observations with their score:

$O_1=\{I_0, I_8, I_9, I_{10}, I_{14}, I_{15}, I_{16}, I_{17}\};s_1=0.995$
$O_2=\{I_0, I_1, I_2, I_3, I_4, I_5, I_6, I_7\};s_2=0.667$
$O_3=\{I_2, I_3, I_9, I_{15}, I_{16}, I_{17}, I_{18}, I_{19}\};s_3=0.1$

The goal now is to identify those subsets (of any length $\leq 8$) that have the best synergies based on the observations; using an efficient algorithm.

Strictly speaking I would want the first $t$ subsets with length $l$ (for each $1 \leq l \leq 8$) of a complete list, ordered (descending) by the average score of observations containing the subset divided by the average score of observations without the subset.

Using brute force (over all subsets of $I$ of length $l$) would be easy to implement but since $8$ and $20$ are only placeholders for much larger numbers, the performance would not be acceptable. Any suggestions?


texts: Creating a pie chart with historical motion in Rstudio

texts: So, I created several pie charts showing voting percentages for republicans and democrats, from 2008 to 2016 with plotly package in Rstudio. What I would like to do, to make it sleek, is create a menu for a user to click on(2008, 2012, 2016), and the pie chart would motion into the correct vote totals. Would I need to create a shiny app to do this?


texts: Is active learning able to detect challenging cases?

texts:
Let's say we have a set of data points that need to be labelled for a classification task. In the pool-based active learning, if we go with the uncertainty measure, is the AL approach able to detect challenging cases? By challenging cases I mean samples that receive a high prediction score for $\hat{y}$ (e.g. >90%) but, most probably, $\neg\hat{y}$ is the correct prediction.
The rationale behind my question is: does adding more samples to the training set always improve the performance of a classifier?


texts: High validation loss, high validation accuracy

texts:
I'm just getting started into the field of deep learning, and I completed my first model training using PyTorch.
I decided to use a pre-build model from torchvision, more specifically the mobilenet_v2 (https://pytorch.org/docs/stable/_modules/torchvision/models/mobilenet.html#mobilenet_v2) on a custom dataset for binary classification.
I manage to get 100% accuracy for both training and test sets (this particular dataset is not a difficult one, so nothing weird about that)
What I don't understand is why the test loss seems to be several orders of magnitude higher than the train loss, while the accuracy is 100% in both cases. Could someone here explain me what is happening here?
I show you the training stats below:
Epoch: 1/15, Train Loss: 0.22990, Train accuracy: 88.48%, Test Loss: 2.10275, Test accuracy: 99.22%
Epoch: 2/15, Train Loss: 0.03988, Train accuracy: 99.41%, Test Loss: 2.08563, Test accuracy: 99.22%
Epoch: 3/15, Train Loss: 0.02211, Train accuracy: 99.41%, Test Loss: 2.05521, Test accuracy: 100.00%
Epoch: 4/15, Train Loss: 0.01173, Train accuracy: 99.80%, Test Loss: 2.05332, Test accuracy: 100.00%
Epoch: 5/15, Train Loss: 0.00499, Train accuracy: 100.00%, Test Loss: 2.04989, Test accuracy: 100.00%
Epoch: 6/15, Train Loss: 0.00381, Train accuracy: 100.00%, Test Loss: 2.04952, Test accuracy: 100.00%
Epoch: 7/15, Train Loss: 0.00171, Train accuracy: 100.00%, Test Loss: 2.04999, Test accuracy: 100.00%
Epoch: 8/15, Train Loss: 0.00072, Train accuracy: 100.00%, Test Loss: 2.04971, Test accuracy: 100.00%
Epoch: 9/15, Train Loss: 0.00045, Train accuracy: 100.00%, Test Loss: 2.04938, Test accuracy: 100.00%
Epoch: 10/15, Train Loss: 0.00035, Train accuracy: 100.00%, Test Loss: 2.04932, Test accuracy: 100.00%
Epoch: 11/15, Train Loss: 0.00029, Train accuracy: 100.00%, Test Loss: 2.04920, Test accuracy: 100.00%
Epoch: 12/15, Train Loss: 0.00025, Train accuracy: 100.00%, Test Loss: 2.04922, Test accuracy: 100.00%
Epoch: 13/15, Train Loss: 0.00022, Train accuracy: 100.00%, Test Loss: 2.04906, Test accuracy: 100.00%
Epoch: 14/15, Train Loss: 0.00020, Train accuracy: 100.00%, Test Loss: 2.04914, Test accuracy: 100.00%
Epoch: 15/15, Train Loss: 0.00018, Train accuracy: 100.00%, Test Loss: 2.04905, Test accuracy: 100.00%


texts: Visualize graph network with more than 30k edges

texts:
I have a dataset with users connection and I want to create a directed network graph.

Nodes: ~20.000
Edges: ~33.000

I used to work before on NodeXL and Gephi, but now I am on Mac and I don't have access on NodeXL.

I tried Gephi and another tool called Cytoscape. Both of them were lagging and I could barely do some statistics. Visualizing the graph was a dream.

Is there any software (or maybe Python library) that can handle such a big network Graph?


texts: ValueError: operands could not be broadcast together with shapes while using two sample independent t test

texts:
I am trying to perform two sample t test. My data set consists of 744 rows and 186 columns for which I have calculated total sum and mean. I need to perform two sample t test. My csv looks like this from which I have to calculate ttest and rank sum test for each row as individual row denotes separate ID and have the corresponding values :

SRA ID  ERR169499            ERR169498           ERR169497
Label   1                    0                   1
TaxID   PRJEB3251_ERR169499  PRJEB3251_ERR169499 PRJEB3251_ERR169499
333046  0.05                 0.99                99.61
1049    0.03                 2.34                34.33
337090  0.01                 9.78                23.22

The labels 0 and 1 are for case and control respectively. So far I have done this:

import pandas as pd
import numpy as np
from scipy.stats import ttest_ind
from scipy.stats import ranksums

def transposer(filename):
file = open(filename, 'rt')
pd.read_csv(file).T.to_csv(str(filename).split("/")
[-1].split(".")[0]+'_transposed.csv',header=False)

pd.read_csv('project.csv').T.to_csv('transposed.csv', header=False)

file = open('transposed.csv', 'rt')
out = open('final_out.csv', 'w')
meta = open('Meta3251.csv', 'rt')
contents = {}
for ids in meta:
    contents[ids.split(',')[1]]=ids.split(',')[-1]
count = 0
for row in file:
    if count == 0:
    out.write('SraID, Label,'+row)
    count=1
else:

    try:
        pid = row.split(',')[0].split('_')[1]
out.write(pid.replace('\n','')+','+contents[pid].replace('\n','')
+','+str(row))
        out.flush()
    except:
        print(pid)
        pass
file.close()
out.close()
transposer('final_out.csv')
file1 = open('final_out_transposed.csv','rt')
label = []
data = {}

x = open('final_out_transposed.csv','rt')
for r in x:
    datas = r.split(',')
    if datas[0] == ' Label':
        label.append(r.split(",")[1:])
label = label[0]
label[-1] = label[-1].replace('\n','')
counter = len(label)
for row in file1:
    content = row.split(',')
if content[0]=='SraID' or content[0]== 'TaxID' or content[0]==' Label':
    pass
else:
    dt = row.split(',')
    dt[-1] = dt[-1].replace('\n','')

    data[dt[0]]=dt[1:]
keys = list(data)
sum_file = open('sum.csv','w')
sum_file.write('TaxId,sum_case,sum_ctrl,case_count,
ctrl_count,case_mean,ctrl_mean,\n')
for key in keys:
    sum_case = 0
    sum_ctrl = 0
    count_case = 0
    count_ctrl = 0
    mean_case = 0
    mean_ctrl = 0
for i in range(counter):
    if label[i] == '0':
        sum_case=np.float64(sum_case)+np.float64(data[key][i])
        count_case = count_case+1
        mean_case = sum_case/count_case
    else:
        sum_ctrl = np.float64(sum_ctrl)+np.float64(data[key][i])
        count_ctrl = count_ctrl+1
        mean_ctrl = sum_ctrl/count_ctrl
sum_file.write(key+','+str(np.float64((sum_case)))+','

+str(np.float64((sum_ctrl)))+','+str(np.float64((count_case)))
+','+str(np.float64((count_ctrl)))+','+str(np.float64((mean_case)))
+','+str(np.float64((mean_ctrl)))+'\n')
sum_file.flush()
sum_file.close()

df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])
case = df.xs('0', axis=1, level=0).dropna()
ctrl = df.xs('1', axis=1, level=0).dropna()
(tt_val, p_ttest) = ttest_ind(case, ctrl, equal_var=False)
print (tt_val)
print (p_ttest)

I am getting the error:

  ValueError: operands could not be broadcast together with shapes (92,) (95,)

How can I handle this error. I cannot change my data.


texts:
I'm trying to write a model with keras, that is built as shown below:
  |                 +-----+
  +->+------+       |     |
+--->|  NN  |------>|     |
|    +------+       |     |
| |                 |     |
| +->+------+       |     |
+--->|  NN  |------>|  L  |
|    +------+       |  S  |------+-----> Output value
|                   |  T  |      |
|      ...          |  M  |      |
| |                 |     |      |
| +->+------+       |     |      |
+--->|  NN  |------>|     |      |
|    +------+       |     |      |
|                   +-----+      |
|    +-------+                   |
+----| Delay |-------------------+
     +-------+

I have several simple sequential models (marked as NN), that receive two numerical value as input, they calculate some other numeric values (one per each network). These values are passed to LSTM network, which produces a single value as an output and this value additionaly is passed to initial networks (possibly with some delay) as one of two inputs. I work with time series, so calculated final value is passed to network alongside with the next time series value.
I use LSTM to store some "state". It is not quite difficult to build separate "sub-models", but I don't realize, how to join them together as I need, i.e. how to make a final output to be passed to initial networks and how to stack them in the described way (not as a chain).
What I've found: I found keras.layers.Concatenate, but it seems not to be what I'm looking for... But maybe (I hope) I'm mistaken.


texts: why model's training is faster on windows than ubuntu?

texts:
I'm training a model of object detection with Tensorflow object detection API
on windows 10 it looks around 3-4 times faster than ubuntu 18.04 and I don't know why
I'm using same batch size, same PC and same dataset
what could be the problem here ?
I'm using gtx 1050 4gb
the model is SSD mobilnet v2
windows:
Step 100 per-step time 0.729s loss=0.841
ubunto:
Step 100 per-step time 2.859s loss=0.841


texts: Why autoencoders work well for outlier detection?

texts:
Apart from the fact that they are neural networks, which usually is a reason for outperforming other algorithms, is there other reason helping auto-encoders perform well in outlier detection?
I know that autoencoders work by encoding a sample into lower-dimension representation, then decoding the representation to reconstruct the sample. As outliers usually have higher reconstruction, they can be detected. However, this does not convince me why auto-encoders can beat other methods. Is it because the outliers errors are very high, so that they can be spotted easily? If so, then what makes the reconstruction errors so high?
Both intuitive and/or theoretical explanations are welcomed.


texts: Problem when using Autograd with nn.Embedding in Pytorch

texts:
I am in trouble with taking derivatives of outputs logits with respect to the inputs input_ids. Here is an example of my input:

# input_ids is a list of token ids got from BERT tokenizer
input_ids = torch.tensor([101., 1996., 2833., 2003., 2779., 1024., 6350., 102.], requires_grad=True)

content_outputs = self.bert(input_ids,
                        position_ids=position_ids, token_type_ids=token_type_ids,
                        attention_mask=attention_mask, head_mask=head_mask)

# content_outputs[1] is sentence embedding
logits = F.linear(content_outputs[1], self.W_s, self.b_s)

# Notes:
# - input_ids.dtype = torch.float32
# - input_ids.is_leaf = True
# - input_ids.requires_grad = True
# - Torch version: 1.0.1

To compute the gradients for input_ids, input_ids.dtype must be float; otherwise, I will get the following error:

  RuntimeError: Only Tensors of floating point dtype can require gradients

However, my model is using an embedding layer which requires the input with dtype=long causing another problem if I use the input_ids initialized with float type above:

  RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.FloatTensor instead (while checking arguments for embedding)

I have also searched on Stack Overflow as well as Stack Exchange but I found nothing related to this problem. Please help me with this. Any comments would be appreciated!


texts: partial numerical array - pattern matching

texts:
I have a linear numerical array source and I want to find/match test array as pattern :

source = [39,36,23,21,28,36,30,22,34,37]
test = [36,23,21,28]

we can use brute force or similar method for finding the exact match, by checking test array from index 0 to len(source)-len(test)

but in our problem, we can accept this pattern too ( order is important )

test = [36,24,21,28] // changed 23 to 24

since we have many different ways of solving this problem ( maybe fuzzy!),
I want to know the title of some solutions.


texts: how to train tensorflow chat application on updated dataset everyday

texts:
I have created chatbot on Cornell movie dataset and it's working fine. I have trained chatbot application up to global step 330000. I am using tensorflow library. I am using ||source|| chatbot. I am also receiving output fine.  Then I created this new dataset file by modifying original dataset and i wish to train chatbot application with updated files. Now should I delete previously saved checkpoints and saved data and start training from zero or should i train from 330000 onwards without worrying changes in dataset.
           I wish to modify database daily basis and train on modified database.
Please kindly let me know if you have a suggestion for me as I am stuck on this issue. I will really appreciate if you help me with this issue.


texts: What NLP methods / algorithms depend on the features existing only in some languages? For example, does French has any NLP algorithms that English NLP and Spanish NLP do not have?


texts: Word2Vec Alternative for Smaller Datasets

texts:
I am looking to employ Word2Vec to cluster documents and classify them by topic. However, it seems I need a significant data set  [1,2] to achieve such a task.

I have a data set of thousands (not millions) of documents where my total word count is somewhere in the tens of thousands or maybe low hundreds of thousands.

Is there another ML methodology that allows me to attain my goal?

I see how to use TF/IDF to generate words and phrases from a corpus  but output as expected is a list of common words and phrases along a flat dimension:

What I am looking for is something more along the lines of a high level cluster of vectors in space:
[source]


texts: Evaluating clusters (e.g. built by kmean) using Random Forest

texts: I have made clusters for my data set (1.5 million samples and 800 features) using k-mean. I am aware of internal indices for evaluating clusters. However, I was thinking about training a supervised classification model (e.g. random forest classifier) in which cluster numbers are dependent variable and all other features are independent variable. Is using the accuracy of this model a good evaluation of the kmean cluster and why?


texts: Saving images in a non-retraceable way, but still able to train R-CNN's on them

texts:
For a computer vision project I am working with images that the company only allows me to have on my computer for a maximum of 24 hours due to regulations. Every day a few hundred images come in via the system. I am trying to solve a object detection problem to prioritize workflow. For this I need more than a few hundred images to train RCNN's.

Q: I am wondering if there is a way to save images in such a way that they are NOT re-traceable anymore to the original image, while at the same time it IS possible to train some kind of RCNN on them?

I was thinking about saving saving convolutional features and labeled bounding boxes instead of the images. But that way it might be re-traceable and I want to be compliant with the regulations. Also, this way I can't train the convolutional layers of a network anymore, since I will only be working with it's output.


texts:
I think this is something that experienced programmers do all the time. But, given my limited programming experience, please bear with me.

I have an excel file which has particular cell entries that read

[[{"from": "4", "response": true, "value": 20}, {"from": "8", "response": true, "value": 20}, {"from": "9", "response": true, "value": 20}, {"from": "3", "response": true, "value": 20}], [{"from": "14", "response": false, "value": 20}, {"from": "15", "response": true, "value": 20}, {"from": "17", "response": false, "value": 20}, {"from": "13", "response": true, "value": 20}]]

Now, for each such entry I want to take the information in each of the curly brackets and make a row of data out of it. Each such row would have 3 columns. For example, the row formed from the first entry within curly brackets should have the entries "4" "true" and "20" respectively. The part I posted should give me 6 such rows, and for n such repetitions I should end up with a matrix of 6n rows, and 4 columns ( an identifier, plus the 3 columns mentioned).

What would be most efficient way to do this? By "doing this" I mean learning the trick, and then implementing it. I have access to quite a few software packages(Excel, Stata, Matlab, R) in my laboratory, so that should not be an issue.


texts: Estimating sales volume of new drugs to a therapeutic class

texts:
I work with pharmacy claims data for an insurance company and wish to estimate how many members on existing products in a therapeutic class will shift to the new product in the same class i.e. insulin once the drug has been covered by the insurance company.

New entrants/products are very common but each one has their own nuances, some are blockbusters while others are flops, some drugs have many indications/uses while others are very specific.

I have access to lots of demographic, historical utilisation, diagnosis code data and formulary information that I could incorporate into a model. I would be keen to hear people’s thoughts and suggestions about how one would tackle such a problem.


texts: How can I use a string as input in a neural network?

texts:
I'm making a program that can determine if a user will like a car from different auctions based on the cars that he/she has bought in the past. Therefore, I want to use the make and model (which are represented as strings) as well as the year, mileage etc... but I'm having trouble to decide on how I'll pass them to the neural network

I've been thinking of using a vector of different makes as such:

 car_make = [[10000]  # Nissan
             [01000]  # Toyota
             [00100]  # Ford
             [00010]  # Dodge
             [00001]] # Mazda

I could pass those numerical values as input but making this is limited as there are tons of different makes, since I'm looking for cars in many different auctions. I have no idea how I can represent the models too as they also change in every auction.

You have any ideas on how I can pass the strings onto the neural network?

p.s I'm using python to build the neural network.


texts: Back Propagation in time for tf.nn.dynamic_rnn for sequential input (from batch)

texts:
I have a code like this:

lstm_cell = tf.contrib.rnn.BasicLSTMCell(256, state_is_tuple = True)
c_in = tf.placeholder(tf.float32, [1, lstm_cell.state_size.c], "c_in")
h_in = tf.placeholder(tf.float32, [1, lstm_cell.state_size.h], "h_in")
rnn_state_in = (c_in, h_in)
rnn_in = tf.expand_dims(previous_layer, [0])
sequence_length = #size of my batch
rnn_state_in = tf.contrib.rnn.LSTMStateTuple(c_in, h_in)
lstm_outputs, lstm_state = tf.nn.dynamic_rnn(lstm_cell,
                                                    rnn_in,
                                                    initial_state = rnn_state_in,
                                                    sequence_length = sequence_length,
                                                    time_major = False)
lstm_c, lstm_h = lstm_state
rnn_out = tf.reshape(lstm_outputs, [-1, 256])

Here, I use dynamic_rnn to simulate the time steps from the batch.
While each forward pass, I can get lstm_c, lstm_h which I can store anywhere outside.

So, suppose I have done a forward pass for N items in a sequence in my network and have the final cell state and hidden state provided from the dynamic_rnn. Now, to perform back propagation, what should be my input to the LSTMs?

By default, does backprop happen across time steps in dynamic_rnn?

(say, no. of time steps = batch_size=N)

So is it enough for me to provide the input as:

sess.run(_train_op, feed_dict = {_state: np.vstack(batch_states),
                                            ...
                                            c_in: prev_rnn_state[0],
                                            h_in: prev_rnn_state[1]
})

(where prev_rnn_state is a tuple of cell state, hidden state, which I got from the dynamic_rnn from forward propagation for the previous batch.)

Or do I have unroll the LSTM layer across time series explicitly and train it by providing a vector of the cell states and hidden gathered across the previous time series?


texts: Shuffling the images with DataImageGenerator and flow_from_directory in a Medical application (tensorflow 2/Keras)

texts:
I am facing the following situation:  I have CT images (scans) of patients where 10 images describe each patient.  These images are stored in separate directories based on what is the focus of each image. Each directory has one type of CT image for every patient.  So if say we have 10,000 patients and 10 directories (one per type of CT scan) each directory contains 10,000 images (of the same CT type).

Each image has a name that begins with the ID of the patient and is followed by the type of the CT scan (10 suffixes exist to descibe the 10 types of CT scans).

I want to load images concurrently from all 10 directories corresponding to a single patient and feed them to a Deep Network.  So assuming a batch of 1 --for simplicity-- at timestep t 10 images all relating to patient A are retrieved and fed to the DN.  At timestep t+1 10 images all relating to patient B are retrieved and fed to the DN etc.

Moreover, I would like to use ImageDataGenerator for data augmentation purposes and in particular the method flow_from_directory().

I would like to have the images shuffled but of course I would not like to have them being shuffled in a different manner in each subdirectory which could result in the DN being fed at any timestep t with images from different patients rather than with 10 images from the same patient, which would be the correct way.

I don't know how this can be accomplished and if it can be accomplished at all.

I would appreciate your advice.


texts:
I'm reading a tutorial about ANN. They use the following cost function:

As you can see this equation includes a norm. I'm new to the concept of norm.

My question is what kind of norm they use here (There are Absolute-value norm, Euclidean norm, Euclidean norm of a complex number, Taxicab norm, Manhattan norm, p-norm, Maximum norm, infinity norm, uniform norm, supremum norm, Zero norm, etc.)


texts: What is a visual bag of words and how is it implemented?

texts: I'm currently working on implementing a bag of visual words in Python. I get the general gist of how it works but I can't seem to find any sources that explain it in more detail to a level where I can implement it. I'm guessing scikit learn and scikit image would come in but I can't seem to point myself in the right direction. Any help?


texts: Clustering with custom criterion (minimum cluster weight)

texts:
Edit: following comment from @anony-mousse, I'm changing the question to search for a general clustering approach that matches this criterion (minimum weight per cluster).

I am to use a clustering method on a set of $n$ weighted points:

---------------------------------------------
| id  | weight | feature_1| feature_2 | ... |
---------------------------------------------
| 1   | 4      | 0.2345   | -0.2345   | ... |
| 2   | 2      | 0.675    | 0.7433    | ... |
| 3   | 15     | -0.45    | 0.123     | ... |
| ... | ...    | ...      | ...       | ... |
---------------------------------------------

I have a custom criterion: some algorithms make sure there is a minimum number of points $n_{min}$ per cluster ; here I would like to make sure each cluster has a minimum weight (sum of point weights) $\sum w_i > s_{min}$.

Is there such a clustering method already implemented in Python?


texts: Criteria for saving best model during training neural network?

texts:
I am doing 4-class semantic segmentation with U-net using generalised dice loss as loss function.
General approach to save best model during training is to monitor validation loss at each epoch and save the model if val loss decreases than previous minimum.
But, I am interested in the "model which gives best the average dice score of 4 classes".
During training in my case, using validation loss as criteria doesn't lead to best avg dice score. So what should I consider the best model the one with less validation loss or highest validation dice score?
Below is my validation loss and avg dice score after each epoch. Out of this which epoch gives the best model?
epoch  1/10     validation loss: 0.95     avg dice score: 0.17
epoch  2/10     validation loss: 0.86     avg dice score: 0.23
epoch  3/10     validation loss: 0.77     avg dice score: 0.34
epoch  4/10     validation loss: 0.74     avg dice score: 0.40
epoch  5/10     validation loss: 0.71     avg dice score: 0.45
epoch  6/10     validation loss: 0.69     avg dice score: 0.34
epoch  7/10     validation loss: 0.79     avg dice score: 0.45
epoch  8/10     validation loss: 0.75     avg dice score: 0.51
epoch  9/10     validation loss: 0.76     avg dice score: 0.36
epoch 10/10     validation loss: 0.75     avg dice score: 0.38

If I go by val loss as criteria epoch-6 gives the best model, if i choose avg dice score as critieria epoch-8 gives the best model? how to choose?


texts: Are validation sets necessary for Random Forest Classifier?

texts:
Is it necessary to have train, test and validation sets when using random forest classifier?

I understand it is important with Neural Networks but I am not understanding the importance of it with RF. I understand the idea of having a third unseen set of data to test on is important to know the model isn't overfitting, esp with Neural networks, but with RF it seems like you could almost not even have test or validation data (I know in practise this isn't true) but in theory since each tree of the forest   uses a random sample (with replacement) of the training dataset.

At the moment I am missing out on approx 250 samples by keeping them unseen from the train and test set and I know the model would improve with the extra data, so is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?


texts:
I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, and treat ties by showing both values. When there are no repeating values I want it to return the first-appearing value that is found along with its frequency 1.

"Name         Color
 Drew         Blue
 Drew         Green
 Drew         Red
 Bob          Green
 Bob          Green
 Bob          Green
 Bob          Blue
 Jim          Red
 Jim          Red
 Jim          blue
 Jim          blue

mode of Drew = Blue, 1
mode of Bob = Green, 3
mode of jim = Red, Blue, 2

Here's the function code i have so far, it excludes NAs but does not show both values when there is a tie and does not show frequency. Any help appreciated!

mode <- function(x) {
  if ( anyNA(x) ) x = x[!is.na(x)]
  ux <- unique(x)
  ux[which.max(tabulate(match(x, ux)))]
}


texts: Classifying objects in video without machine learning

texts:
Recently, Nick Bourdakos posted a series of videos demonstrating bottle detection in a video stream using Tensorflow.js. Specifically, he is using SSD-mobilenet.

The problem could be summarised as follows:

Three different drinks bottles appear together or individually in a video stream. Let's assume Coke, Mountain Dew and Pepsi as in the original video
Each bottle must be classified and labelled with its name
A rectangular bounding box should be placed around each bottle

I am interested to know if there are there any established techniques that can achieve similar results without machine learning?

So far I have tried:

Identification based on colour thresholding. This works surprisingly well but is not very robust. For example the red lettering inside the Mountain Dew label can be falsely detected as "Coke". Also, bounding boxes can only be drawn around coloured regions rather than the entire bottle
Template based matching. This didn't work at all. I assume this is intended for static images where an exact match is required.
KCF Tracking. This seems to work well, but I have to manually define a region of interest to track first. That is, objects are tracked but not classified
I also tried using edge detection to classify "bottle" or "not bottle" based on the aspect ratio of edged objects, however because the bottles are "hand held" separating the bottle edge from the user's hand is problematic

Is there some computer vision technique or steps that I'm missing that would come closer to solving the problem?


texts: RNN: Multiple inputs per time step with categorical variables

texts: I am trying to a build RNN model to forecast daily sales for several different cities and different product segments (categorical features and multiple inputs for each day) along with numerical features such as traffic , temperature etc (single input for each day). How do I go about building a model using these features. I tried to label encode (Labelencoder) the categorical features and tried to successively train/update a RNN model feeding it with the time series of each city's product segment data but I ended up with huge prediction errors. Is there a way to one hot encode the categorical features and if I do use one hot encoding how do I go about it (what will be my output dimensions)


texts: Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

texts:
When comparing several regression models in terms of quality, it seems like most have agreed on the MSE.
There are also papers comparing "variance" and "variance accounted for (VAF)".
However, there seems to be a controversial opinion about the variance (R^2). Should it nevertheless be compared in a scientific paper?

And what does VAF say? Is the VAF still a good measure of regression models?


texts:
I am trying to create a sentiment analysis model and I have a question.
After I preprocessed my tweets and created my vocabulary I've noticed that I have words that appear less than 5 times in my dataset (Also there are many of them that appear 1 time). Many of them are real words and not gibberish. My thinking is that if I keep those words then they will get wrong "sentimental" weights and gonna make my model worse.
Is my thinking right or am I missing something?
My vocab size is around 40000 words and those that are "rare" are around 10k.Should I "sacrifice" them?


texts: Self-driving AI model starts overfitting

texts:
I'm trying to make a self-driving AI that can drive around in GTA-San Andreas by following sentdex's videos on making a self-driving AI for GTA-V but my model always starts to overfit after 4 epochs or so.

Even after doubling the dataset that I originally had, the model is still overfitting.

The dataset made up of grayscale images (270x360) and is distributed into 250 .npy files with 1000 frames each totaling 250,000 frames.

I'm using Xception architecture without any pre-trained weights as the model. The code is given below:

def create_model():
    xception_model = Xception(include_top=False, weights=None, input_shape=dimensions)
    x = xception_model.output
    x = (GlobalAveragePooling2D()) (x)
    x = (Dense(1024, activation='relu')) (x)
    predictions = Dense(9, activation='softmax') (x)

    model = Model(inputs=xception_model.input, outputs=predictions)

    opt = adam(lr=1e-3, decay=1e-5)
    model.compile(loss='categorical_crossentropy',
                  optimizer=opt,
                  metrics=['accuracy'])

    return model

model = create_model()
epochs = 30

allhistory = {'val_loss':[], 'val_acc':[], 'loss':[], 'acc':[]}

for e in range(1,epochs+1):
    data_order = [i for i in range(1,total_files+1)]
    shuffle(data_order)

    for count,i in enumerate(data_order):
        try:
            filename = f'/content/drive/My Drive/Dataset/processed_data-{i}.npy'
            data_set = np.load(filename,allow_pickle=True)

            print(f"\n{e+1}:{count+1}:Data-{i}.npy Loaded!")
            X = np.array([np.repeat(data[0]/255.0,3,-1) for data in data_set]).reshape(-1,HEIGHT,WIDTH,3)
            y = np.array([data[1] for data in data_set])
            X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = 5)

            history = model.fit(X_train, y_train, validation_data = (X_test,y_test), epochs=1)

            allhistory['val_loss'].extend(history.history['val_loss'])
            allhistory['val_acc'].extend(history.history['val_acc'])
            allhistory['loss'].extend(history.history['loss'])
            allhistory['acc'].extend(history.history['acc'])

            if (count+1)%10==0:
                print('\nSAVING MODEL!')
                model.save(f'/content/drive/My Drive/models/xceptionmodel-epoch-{e+1}.h5')
                np.save('/content/drive/My Drive/models/allhistory.npy',np.array(allhistory))

        except Exception as ex:
            print(str(ex))

Here is a graph showing how the accuracy and validation accuracy starts diverging after 4 epochs:
Accuracy Graph

Can anyone please help me and point out what mistakes I'm making.


texts:
The input to a convolutional layer of a neural network is an image of size $128\times 128\times 3.$

$ 40$ convolutional filters of size $5\times 5$ are applied to it. Would you get an output? If no then why not? If yes, what will be the output size?


texts: Model to predict coronavirus (covid19) spread

texts:
im new in data sience and machine learning but i have some mathematical and statistics backgroud. I really just want some information about models (like papers or raw models). So if you have any information please share them with me.

thank you


texts: Classification problem: custom minimization measure

texts:
Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods.

I'm trying to develop a classification model, where the desired outcome is the probability, rather than only the output class.

However, whatever variable/model combinations I try, the models are able to distinguish the bad cases quite nicely, but not the good cases. In other words, the distribution function of (smoothed) empirical outcomes vs the model estimated probabilities is not monotone, roughly having the shape of a tilted parabola (like letter $J$).

My question: What are the common  strategies in order to reshape any model to focus the estimation on the good cases? If it is possible at all?

Intuitively, it seems that a possible strategy would be to define a custom metric for the minimizer, that would have asymmetric weights for good vs bad cases. I.e., in the case of penalized linear regression, the variable selection could then be skewed towards distinguishing goods instead of maximizing total AUC. But so far I'm unable to find any realized similar solutions. I guess, implementing something like that is not a trivial task..

Alternatively, is it possible to achieve this by transforming the input variables in a certain way?


texts: Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?

texts:
I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. Pearson’s r gives returns a value 1 if the files are identical, the dtw score and path are 0 if the files are identical.
But I need a solution if the files are as the ones in the figures, similar, with a little time lag. They are readings from two different accelerators who were attached to the same source. The sampling frequency and amplitude is not same. Even the number of readings are not same. Time stamps could be different.
How do I measure the similarity between such files? Is there some metric or measurement I can get using Python? Because dtw score and dtw distance do give some output, but there is no way I can say the files are similar using those values.


texts: Replacing dataloader samples in training pytorch

texts:
Initially, a data loader is created with certain samples. While training I need to replace a sample which is in dataloader. How to replace it in to dataloader.
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

for sample,label in train_dataloader:
        prediction of model
        select misclassified samples and change them in train_dataloader but      how to change sample in train_dataloader

While training, the misclassified samples need to be modified.
So How to replace a sample within train_dataloader?


texts:
I have seen at different places saying that: l1 regularization penalizes weights more than l2.

But the derivative of l1 norm is $\lambda$ and l2 norm is 2$\lambda$w. So l1 regularization subtracts smaller value than l2. Then why is it called that l1 penalizes weights more than l2. Or is it incorrect to say it like this?


texts: How to cluster some text using TensorFlow

texts:
Apologies for any inaccuracies due to the infancy in this field.

I'm trying to learn on how to return a dataset with three classes to be clustered by TensorFlow.

At this stage, I've read a lot and experimented with several Colabs, but none of them points to the clustering technique when it comes to text (just classification).

The whole point is I don't have any clue of how the text could be classified, and I though the recursive process used by TensorFlow would have offered a better approach and results of what I normally get by using K-Means.

I drafted a Colab myself, but it's a real mess of examples now, so I better not to share at this stage.

In a nutshell, I tokenised the keywords, created a dictionary, converted my text into a dense matrix and then started using the models.
However, when it comes to using the fit function, I end up getting a series of error with the most recent one being "ValueError: No gradients provided for any variable".

I would appreciate some support and hints also on materials to read if any.
The release of TensorFlow 2 makes things even more complicated as there are even less examples and trying to adapt the ones for TF1 not always work.

Thanks

UPDATE:

I've cleaned up the project, inferring from another example taken from here.
Clearly the original example was working, but not mine.

In my version I'm trying to train the model to learn three different classes.
My questions at this stage are:

How can I create a suitable sets of labels when the Dense output is greater than 1?

Hot encoding is not an option apparently as the label shape is not inferred from the size of the array as I initially thought, but from the values itself.
How can I predict a new unseen keyword/text?

Here, do I need to train all the time the model to "see" the new keywords. I would expect so, but how?
How can I print/export the output of my prediction?


texts: How to equalize the pairwise affinity perplexities when implementing t-SNE?

texts:
I'm trying to implement the t-SNE algorithm:

I found that to compute the pairwise affinities, I have to follow this:

My problem is computing $\sigma_i$. In the Wikipedia I found:

  The bandwidth of the Gaussian kernels $\sigma_{i}$,
  is set in such a way that the perplexity of the
  conditional distribution equals a predefined perplexity using a binary
  search. As a result, the bandwidth is adapted to the density of the
  data: smaller values of $\sigma_{i}$ are
  used in denser parts of the data space.

I don't understand what this really means. How can I calculate $\sigma_i$?


texts: Remove indiferent respondents in survey data

texts:
I have data for a product rating survey, which requires the respondents to rate a product in five levels: Very Bad, Bad, Regular, Good and Very Good. This survey was applied to several communities of clients.

After taking a look on the data, I observed that a lot of communities have an overwhelming "Regular" preference for the product. Later I found out that "Regular", was the default response for the survey question. Apparently, in some communities user's just didn't bother and left all the questions in the default value.

Is there a proper way of identifying the communities that are indifferent to the survey, so they can be excluded? I was thinking in a simple threshold-based approach (If the community has more than 50% of non-default responses, it is not indifferent), but I was wondering if there is a more principled way.


texts: Text-Classification-Problem, what is the right approach?

texts:
I'm planing to write a classification program that is able to classify unknown text in around 10 different categories and if none of them fits it would be nice to know that. It is also possible that more then one category is right.

My predefined categories are:

c1 = "politics"
c2 = "biology"
c3 = "food"
...

I'm thinking about the right approach in how to represent my training-data or what kind of classification is the right one. The first challenge is about finding the right features. If I only have text (250 words each) what method would you recommend to find the right features? My first approach is to remove all stop-words and use the POS-Tagger (Stanford NLP POS-Tagger) to find nouns, adjective etc. I count them an use all frequently appeared words as features.

e.g. politics, I've around 2.000 text-entities. With the mentioned POS-Tagger I found:

law:           841
capitalism:    412
president:     397
democracy:     1007
executive:     112
...

Would it be right to use only that as features? The trainings-set would then look like:

Training set for politics:
feature law         numeric
feature capitalism  numeric
feature president   numeric
feature democracy   numeric
feature executive   numeric
class politics,all_others

sample data:
politics,5,7,1,9,3
politics,14,4,6,7,9
politics,9,9,9,4,2,1
politics,5,8,0,7,6
...
all_others,0,2,4,1,0
all_others,0,0,1,1,1
all_others,7,4,0,0,0
...

Would this be a right approach for binary-classification? Or how would I define my sets? Or is multi-class classification the right approach? Then it would look like:

Training set for politics:
feature law         numeric
feature capitalism  numeric
feature president   numeric
feature democracy   numeric
feature executive   numeric
feature genetics    numeric
feature muscle      numeric
feature blood       numeric
feature burger      numeric
feature salad       numeric
feature cooking     numeric
class politics,biology,food

sample data:
politics,5,7,1,9,3,0,0,2,1,0,1
politics,14,4,6,7,9,0,0,0,0,0,1
politics,9,9,9,4,2,1,1,1,1,0,3
politics,5,8,0,7,6,2,2,0,1,0,1
...
biology,0,2,4,1,0,4,19,5,0,2,2
biology,0,0,1,1,1,12,9,9,2,1,1
biology,7,4,0,0,0,10,10,3,0,0,7
...

What would you say?


texts: How to convert images (.jpg) to vectors for image classification

texts:
I'm currently working on a project that involves classifying an image as either that of a dog or that of a cat. The twist is that I want to do this without using Convolutional Neural Networks, mainly because I do not quite understand them yet and I don't just want to copy someone's code off of Github.

I know that the classification algorithms on Sci-Kit Learn require the x input data to be in vectorized form for the classifier to fit the data, but I'm unsure of how to do this. It's easy to do with text data (feature_selection.text.CountVectorizer/TfidfVectorizer), but I have no idea how this works with images. How do I convert .jpg image files to vectors/matrices so that the models on SK-Learn understand it?

Thanks in advance, and I'm sorry if this is a stupid question.


texts: Classification problem with 2 level features

texts:
Consider an automated house, where we can collect router data every minute. The objective is to predict router fail/no fail (1/0) in a future time window.
The router sends data at two different levels every minute:

Router properties of the device: Memory, Temperature, Usage (router level)
Data collected from N different IoT devices, received rates, signals (IoT device level)

Since the target is defined at the router level, how could the IoT individual data be incorporated into the classifier?


texts: Rescaling each image Individually with keras

texts:
I am a beginner working on a simple CNN to classify X-ray detector images. Due to source intensity, all images have different max values. I want to use ImageDataGenerator to rescale those images to be in range [0,1], but cant't find a way to do that for each individual image. As far as I understand it, people usually just divide by 255 because that is the RGB max, but in my case the max could be anything between 1 and serveral million. Does anyone have an idea how to do this within ImageDataGenerator? Thank you for your time!

Edit:
I ran over my images and normalized all of them prior to feeding them to the generator. Inspecting them then showed that the generator had scaled them back to rgb values (I assume due to color_mode). I then used rescale with the standard 1./255 and got the expectd max value of 1. However, after adding back samplewise_std_normalization, I again am getting values larger than 1. I understand why this is happening, but I'm not sure which "rule" is more important to follow: normalize to [0,1] or use samplewise_std_normalization


texts: If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?

texts:
I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal operator. Hence, one can apply iterative hard thresholding algorithm to get the sparse solution of the following

$$\min \Vert Y-X\beta\Vert_F + \lambda \vert \beta \vert_0$$

If so, why people are still using $\ell_1$? If you can just get the result by non-convex optimization directly, why are people still using LASSO?

I want to know what's the downside of the proximal gradient approach for $\ell_0$ minimization. Is it because of the non-convexity and randomness associated with? That means the initial estimator is very important.


texts: loss = function(iteration) gets super wobbly once it gets near the bottom

texts:
When fitting neural nets and getting close to the bottom, I consistently get a very distinct pattern in the loss function (and the mse).  See below:

(The lower right plots just give the last 100 iterations, and the x axis is always the iteration.)

Once it gets close to the bottom, the net starts overshooting and then correcting itself -- eventually working its way to the bottom (or a bottom) of the loss surface).  This is annoying -- I'd like it to go faster and not dither.

Features of the net:

N = about 7000, about 2000 input variables, 10 layers of 100 nodes each
The loss is L2-penalized squared error.  This one is only very lightly penalized (and thus will almost certainly end up overfitting given the architecture)
This particular net is using full-batch gradient descent, no dropout, and full-connections.
But I see the same thing when using minibatch, and either dropout or convolutional architectures.
The learning rate gets multiplied by a constant every time the loss goes down, and divided by the same constant squared when the loss goes up.
Activation is the leaky ReLU
No batch normalization
RMSprop for adaptive learning rate
Fit with this package (I'm the author)

I'm not sure how to correct this behavior, and I'd appreciate suggestions.  It would seem like the adaptive stepsize algorithm would need to know when it is going to hit a wall in a given direction, before it hits that wall.  Is that possible?  Would something like adagrad improve over RMSprop?  Alternatively, I haven't yet implemented batch normalization -- not sure how that would help, but, would it for some reason?

Edit: here's an update, a few hundred iterations later.  Same behavior.


texts: Good explanation for why regularisation works

texts:
I am trying to understand regularisation for logistic regression currently, but I am not sure I get it. I understand the issue of overfitting when there are relatively too many features, and I get that you would like to limit the impact of some of these superfluous features, but you are doing in regularisation is impacting all features.

And this brings me to my next point, which is that by putting a limit on weights, you can still construct the same hyperplane. As long as you have the same ratio between the weights $w_i$ for $i \in \{1,2,..,n\}$, the "angle" of the hyperplane would still be the same, and then you can simply "adjust" it's position by the intercept $w_0$.

Many lecturers and authors use the complex polynomial example with additional terms $f(x_i)$ that make the decision boundary non-linear to illustrate that using regularisation will make this boundary more linear, and thus less prone to overfitting, but then again, you could plot $f(x_i)$ on a separate axis, making decision boundary a hyperplane, and then you could fit the exact same decision boundary with smaller weights, as long as the ratios are conserved.

Thus, if you can make the exact same hypothesis with smaller sum of squares of weights $\sum w_i^2$ (that is with smaller regularisation term), what is the point of regularisation? Or in other words, as there obviously exists empirical evidence showing that regularisation works, how does it work? Is there a good proof for it, or some good (and less vague) intuition?


texts: How respective gating functions are ensured in LSTM?

texts:
I'm studying the Hochreiter-Schmidhuber long-short term memory recurrent architecture. The overall idea, information flow and manipulation is clear, and it seemingly works, but what I cannot understand that how could the effect of the respective "gates" are ensured, or indeed, checked one by one?
For example, it's clear that the "forget gate" (the first in the sequence of interactions between cell state, hidden state and new incoming training instance) is responsible for "forgetting" information from cell state, but I cannot see how it is the one that is actually forced to do the "forgetting", and not some other gate? The overall mechanism (unite x_t and h_t-1, squeeze this through some function and inject the result to c_t) is pretty similar in case of the input and output gates as well, so in theory (and in practice) maybe not the forget gate that does the forgetting, but perhaps the input gate in the specific case. How to explain and check this? Maybe catching the cell state immediately after the forget gate, compare it with c_t-1, and see if the vector values are somewhat decreased/nulled is an evidence? Or maybe the names of the gates are not necessary to be taken seriously but just interpret the whole mechanism as a random black box what somehow still works?


texts: Why does BERT embedding increase the number of tokens?

texts:
I am new to DataScience and trying to implement BERT embedding for one of my problems. But I am having one doubt here.
I am trying to embed the following sentence with BERT - "Twinkle twinkle little star".
BERT tokenizer generates the following tokens - ['twin', '##kle', 'twin', '##kle', 'little', 'star']
But the final embedded tensor is having a dimension of [1,8,1024]
Why is the number of tokens 8 instead of 6? For any text, I am observing that number of tokens in the final embedding is getting increased by 2. Can anyone please help me to understand this?
I am giving the code snippet here -
from transformers import BertTokenizer, BertForSequenceClassification, BertModel

PRE_TRAINED_MODEL_PATH = 'BERT\wwm_cased_L-24_H-1024_A-16'
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_PATH)
model = BertModel.from_pretrained(PRE_TRAINED_MODEL_PATH)

encoded_input = tokenizer(texts, return_tensors='pt', padding=True)
emb = model(**encoded_input)


texts:
I'm trying to apply k-means$\|$ clustering in PySpark.

According to this paper, there is an oversampling factor, $l$, that would affect the model's cost.

I couldn't find any parameter regarding oversampling factor in PySpark's k-means function. There is a parameter called initializationSteps but there is no proper definition for that.

Is there a way I could use oversampling factor in k-means function?


texts: what would happen in max_pool layer if backprop would add gradient to all inputs for particular neuron but only if it's positive

texts: in max_pool layer ANN performs this operation max([in1, in2, ... inN]), now if gradient that comes back to this layer is negative sure it should affect only max value, but if gradient is positive shouldn't it affect all values (after all this layer performs max operation so work of gradient shouldn't be symmetrical) because if max value is too low then all other input values for this neuron are also too low


texts: Acceptable variation in accuracy of each k fold when using K-Fold Cross Validation?

texts:
I have a relatively small dataset consisting of 1432 samples.

I have trained a Random Forest Classifier and performed KFold CV. The results of running 10 Fold CV are as follows:

=== 10 Fold Cross Validation Scores ===

CVFold 1 = 90.2%
CVFold 2 = 87.6%
CVFold 3 = 86.7%
CVFold 4 = 86.7%
CVFold 5 = 83.9%
CVFold 6 = 75.8%
CVFold 7 = 87.2%
CVFold 8 = 82.8%
CVFold 9 = 86.1%
CVFold 10 = 89.3%

Mean Cross Validation Score:  85.6%

I am just not sure how to explain why there are such high variances between some of the folds, i.e from 75.8% in fold 6 to 90.2% in fold 1.

My understanding is that it is simply that the classifier found the samples in the 1st fold (90%) easier to classify than it did in fold 6 (75%) but I'm actually not entirely sure if that is the case.

I understand that each case is different but is it common to have such variances? And is it acceptable?

Edit: some details regarding my data

I have 5 classes, which are imbalanced:

class 1 - 5%

class 2 - 10%

class 3 - 15%

class 4 - 60%

class 5 - 10%

I am using SMOTE to balance the classes.


texts:
I am fitting a function $y:R^3\to R$ using the following model in Keras.

from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(50, input_dim=3, activation='tanh'))
model.add(Dense(1, activation='linear', kernel_initializer='normal'))

model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10, batch_size=8)
predictions = model.predict(x_test)

To train, I used a data set consisting of $\{\mathbf{x}_i\}_{i=1}^N$ and $\{y_i\}_{i=1}^N$. This worked as expected.

Now imagine a slightly different situation where for each sample $i$ I have a pair of $\mathbf{x}_{i,1}$ and $\mathbf{x}_{i,2}$ ($\mathbf{x}_{i,\alpha} \in R^3$) and my target data point corresponding to $i$ is modelled as $y_i=z(\mathbf{x}_{i,1})+z(\mathbf{x}_{i,2})$, with $z:R^3 \to R$. Is it possible to define such a model in Keras, and how to do it? If not Keras, any other framework?


texts: Why is it important to have stable activations in the forward pass?

texts:
In many sources on the literature on weight initialization, I find the idea that it's a good idea to keep the activations stable through the layers, that is making sure they stay about the same size/order of magnitude as you go through the layers. Sometimes it's implied that this has something to do with avoiding exploding or vanishing gradients.
For example in this blog article:

During the forward step, the activations (and then the gradients) can quickly get really big or really small — this is due to the fact that we repeat a lot of matrix multiplications. Either of these effects is fatal for training.

But why? To me this seems like an non sequitur, I don't see why unstable activations would imply unstable gradients. The only thing I can think of is that if we have an edge with weight $w$ connecting a node with activation $f(x)$ into a node with pre-activation $y$, we have the following formula:
$$
\frac{\partial C}{\partial w} =
f(x)\frac{\partial C}{\partial y}
$$
Is it just because the activation $f(x)$ appears in that formula? I'm not sure because no source that I've found explicitly says this. But what if the differential $\frac{\partial C}{\partial y}$ cancels out the effect of $f(x)$, for instance if the activations get really big as we go through the layers, but the differentials get really small, and it cancels out?


texts: Generating normally distributed data frame with 3 columns

texts: I am trying to generate a normally distributed data frame in Python with 3 columns. Each column requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate sales data of Unique Coke products, Unique Products and Total Revenue which are normally distributed. Also, how can I find the mean of the entire normally distributed data frame?


texts: Features & Models to compute the probability of certain customer accepting an offer/product from a bank?

texts:
What are the features & models that can be used to compute the probability of a certain customer accepting an offer/product from a bank?

After some research, I came to know of what is called 'Propensity Scoring' and it's definition is very relevant to what I am trying to find.

But I failed to find any technical papers going into any appreciable detail. I tried Google Scholar but reached a dead end because the term 'Propensity modeling' is actually a marketing term not exactly academic.

I also know from my search that a broader term is 'Campaign Response Models' but this turns out to be too broad and I am looking for something very specific. Any references/links that contain some technical details about the modeling process, the features or the choice of models will be very helpful.


texts: ValueError: Found input variables with inconsistent numbers of samples [43,19]

texts:
So, I've been trying to split my dataset into a 70-30 ratio using train_test_split in order to work things out with sklearn's PLS. However, I encountered an error that says:

# Read data
df = pd.read_csv(
    filepath_or_buffer='colontumor.csv',
    header=None,
    sep=',')

df.tail()

X = df.iloc[:, :-1].values

# Split data to train and test on 70-30 ratio
X_train, X_test = train_test_split(
    X, test_size=0.30, random_state=None)

pls = PLSRegression(n_components=2)
pls.fit(X_train, X_test)
Y_pred = pls2.predict(X_train)

and somehow encountered this error:

Line (17): pls.fit(X_train, X_test)
ValueError: Found input variables with inconsistent numbers of samples: [43, 19]

Is there any solution for this? I've been circling around for like hours now.


texts: What are good sources to learn about Bootstrap?

texts:
I think that Bootstrap can be useful in my work, where we have a lot a variables that we don't know the distribution of it. So, simulations could help.
What are good sources to learn about Bootstrap/other useful simulation methods?


texts: How are boosted decision stumps different from a decision tree?

texts:
For example, taking the image from sebastian raschkas post "Machine Learning FAQ":

I would expect a very similar (if not exactly the same) result for a decision tree: Given only two features, it finds the optimal feature (and value for that feature) to split the classes. Then, the decision tree does the same for each child considering only the data which arrives in the child. Of course, boosting considers all the data again, but at least in the given sample it leads to exactly the same decision boundary. Could you make an example where a decision tree would have a different decision boundary on the same training set than boosted decision stumps?

I have the intuition that boosted decision stumps are less likely to overfit because the base classifier is so simple, but I couldn't exactly pin point why.


texts:
I am implementing WGAN-GP using Tensorflow 2.0, but each training iteration of the critic is very slow (about 4 secs on my CPU, and somehow 9 secs on Colab GPU).

Is WGAN-GP usually this slow or there is a flaw in my code?

Here's my code for training the critic:

def train_critic(self, X_real, batch_size, gp_loss_factor, optimizer):
    y_real = np.ones((batch_size, 1))

    # Get batch of generated images
    noise = np.random.normal(0, 1, (batch_size, self.z_dim))
    X_fake = self.gen.predict(noise)
    y_fake = -np.ones((batch_size, 1))

    X = np.vstack((X_real, X_fake))
    y = np.concatenate((y_real, y_fake))

    # Interpolate images
    alpha = np.random.uniform(size=(batch_size, 1, 1, 1))
    X_interpolated = alpha * X_real + (1 - alpha) * X_fake
    X_interpolated = tf.constant(X_interpolated, dtype=tf.float32)

    # Perform weight update
    with tf.GradientTape() as outer_tape:
        # Calculate gradient penalty loss
        with tf.GradientTape() as inner_tape:
            inner_tape.watch(X_interpolated)
            y_interpolated = self.critic(X_interpolated)
        gradients = inner_tape.gradient(y_interpolated, X_interpolated)
        norm = tf.sqrt(
            1e-8 + tf.reduce_sum(tf.square(gradients), axis=[1, 2, 3]))
        gp_loss = gp_loss_factor * tf.reduce_mean((norm - 1.) ** 2)

        # Calculate Wasserstein loss
        y_pred = self.critic(X)
        wasserstein_loss = wasserstein(y, y_pred)

        # Add two losses
        loss = tf.add_n([wasserstein_loss, gp_loss] + self.critic.losses)
    gradients = outer_tape.gradient(loss, self.critic.trainable_variables)

    optimizer.apply_gradients(zip(gradients, self.critic.trainable_variables))

    return wasserstein_loss, gp_loss

def wasserstein(y_true, y_pred):
    return tf.reduce_mean(y_true * y_pred)


texts: Standardization with positive and negatives

texts:
I have a data set that has a few columns such as:
Total cost: mean = 3,000,000
Percent complete: mean = 50
final profit %: mean = 14
I know with such different orders of magnitude before I fit a linear regression I should standardize the data (using python and sklearn). The problem is there are negatives in this data that I need to keep so I don't know which type of standardization I should use? The only two I am familiar with are log transformations and StandardScaler both of which I think get rid of negatives.


texts: Dissecting and understanding the Adam optimization's formula

texts:
Adam's optimization has the fololwing parameter update rule :

$$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space v_t \space \text{is second moment of gradient} $$

I have the following questions with regards to the above formula:

What exactly is first and second moments of gradients ? what's the intuition behind the first and second moment's formula?
I understand SGD with momentum and SGD with RMSprop but here we are making use of both of these. Again I don't understand the intuition behind diving the first moment by the square root of second moment

I looked up online and read various articles before coming here because none of the articles were helpful in providing intuition. I also tried reading the original papers but I found it hard to comprehend it.


texts:
Creating a classifier to do "Sentiment Analysis" can be done with several algorithms like SVM, KNN, Neural networks,Decision tree...

and lately i have read about Decision tree and how it works and i would like to ask if someone can provide me with an explanation and a conceptual representation of how the decision tree will be created to tackle this problem, and what/how attributes will placed in the nodes of the tree
to classify a sentiment of a text as positive|negative|neutral ?

any help would be appreciated, and thanx.


texts: Difference between Time series clustering and Time series Segmentation

texts:
In the context of time series data mining, I have read about time series segmentation and time series clustering, but I couldn't differentiate between both. In case they are different, how these methods are correlated with each other?

Well from my understanding (please correct me if I am wrong), the segmentation is considered as a preprocessing step for the clustering phase. I mean that the segmentation step is used mainly to partition your time series data into segments, let's say into states. After that, a conventional clustering algorithm can be applied to group these segments into clusters (similar segments belong to the same cluster).

As an example, let's say that the segmentation process represents a given time series into the following segments: (S1, S2, S3, S4, S5, S6). Then after applying the segmentation process, a conventional clustering method is applied to cluster the extracted segments. So we might end up with something like this: If k = 3: then K1 {S1, S5}, K2 {S3, S6}, K3 {S2, S4}

Please correct me if I am mistaken, and provide links for more clarification if you have any.


texts: How to calculate coverage in recommender systems?

texts:
I am trying to calculate coverage metrics for a recommender system that I have designed. This  blog post talks about how to do it. I had some difficulties in understanding the same. It says that These metrics will be for user u of U users over n items out of N potential items in a recommendation list L. Each item has a content vector C of length c ... My difficulty arises in understanding N and U. Is U the total number of users? Secondly, what does N potential items mean? Is it the list of all items that are present or is it a subset of all the items present?

I am using the Python LightFM package to generate the recommendations


texts: Create tensor of a sequence with a defined shape using TensorFlow

texts:
Create a tensor Y of shape (2,3) having a sequence of numbers 12,15,18,...., 150

I tried using the below code

tf.constant(np.arange(12,153,3),dtype=tf.dtypes.float32,shape=[2,3])

I got an error saying the tensor with 47 elements cannot be converted into 6 elements.

I added the following third dimension in shape and it worked.

tf.constant(np.arange(12,153,3),dtype=tf.dtypes.float32,shape=[8,2,3])

How to proceed as the given question wants the sequence tensor to be in shape [2,3]?

Please help.


texts: How to run KNN (or other) on nested features? Image metadata

texts:
I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I would like to create categories of images based on relative positions of several items in the image.

Example:

img1 has

logo in top x1, right y1, and is height h1 and width w1

text in top x2, right y2, and is height h2 and width w2

from img1 to n

I thought about running a KNN classifier to find images that are more similar to others but I am having trouble thinking about how to organize the data.

Should I group them by image? In which case: how would KNN or other algorithms work on grouped data? Should I do OHE to know if there is a specific element? Would it be better to use a DecisionTree?

Its my first data science job, and there is nobody with seniority here so I am having trouble organizing this dataset.

Thanks in advance!


texts: What kind of neural network structure is suitable for image to image learning?

texts: There exists a mapping from input image to out image. Say input image is a piece of paper with a square hole in the center, and output image is the shadow of the input image when light shines on the paper, which also looks like a square, but with some blurs and different in size. (The real pattern is far more complicated than this square hole) I'm trying to implement a neural network that can learning this kind of mapping, when given the input image, the network can predict the "shadow" of the input image. I've tried to use a convolutional autoencoder-like structure (like this https://blog.keras.io/building-autoencoders-in-keras.html) by replace the target image from the input image it self to the output image training set. But it didn't looks work well. I would like to know what kind of neural network is suitable for this task? And what kind of loss  function is suitable?


texts: Confused about steps_per_epoch and validation_steps in Keras

texts:
According to the documentation in TF 2:

       steps_per_epoch: Integer or `None`.
            Total number of steps (batches of samples)
            before declaring one epoch finished and starting the
            next epoch. When training with input tensors such as
            TensorFlow data tensors, the default `None` is equal to
            the number of samples in your dataset divided by
            the batch size, or 1 if that cannot be determined. If x is a
            `tf.data` dataset, and 'steps_per_epoch'
            is None, the epoch will run until the input dataset is exhausted.
            This argument is not supported with array inputs.
        validation_steps: Only relevant if `validation_data` is provided and
            is a `tf.data` dataset. Total number of steps (batches of
            samples) to draw before stopping when performing validation
            at the end of every epoch. If validation_data is a `tf.data` dataset
            and 'validation_steps' is None, validation
            will run until the `validation_data` dataset is exhausted.

I don't understand this part:

Total number of steps (batches of samples)
                before declaring one epoch finished and starting the
                next epoch.

If I have 1000 samples, batch size = 100, then an epoch will take 10 steps to reach. Why is another steps_per_epoch is needed? If both are used, they are conflicting. Isn't it? If the batch size is 100, then 10 steps are needed. If also steps_per_epoch = 20, then it means in one epoch it needs 20 batches, which is conflicting with the '10' steps calculated through the batch size parameter 100.

Where am I wrong?


texts:
I am working on a ML model where individual features may have a highly variable number of observed values.
The model will predict a continuous variable so I am planning to use a Regressor.
More precisely, I am dealing with roughly 40 features some of which are categorical and some are continuous.  All the features can have a variable number of observed values for each target value in the training set.  Are there any models that natively handle a variable # of observations per feature or do I need to select one value per feature in some manner (e.g. mean, first observed, etc.)?


texts: When using padding in sequence models, is Keras validation accuracy valid/ reliable?

texts:
I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have the same lengths, I use padding.
An example of padding:
# [0,0,0,0,0,10,3]
# [0,0,0,0,10,3,4]
# [0,0,0,10,3,4,5]
# [10,3,4,5,6,9,8]

In order to evaluate if the model is able to generalize, I use a validation set with 70/30 ratio. In the end of each epoch Keras shows the training and validation accuracy.
My big doubt is whether Keras validation accuracy is reliable when using padding (When you run Keras over several epochs, in the end of each epochs it prints training accuracy and validation accuracy). Because the validation set can simply be sequences of 0's --> [0,0,0]. Since there are a lot of sequences of 0's (because of padding), the model can easily learn and predict the sequences of 0's correctly, and as a result, create a fake high validation accuracy. In other words the model may learn sequences of zeros and not learn the real sequences.
So, does padding influences the validation accuracy in Keras?