texts: Modeling count data with time-dependent rate

texts:
For processes of discrete events occurring in continuous time with time-independent rate, we can use count models like Poisson or Negative Binomial. For discrete events that can occur once per sample in continuous time, with a time-dependent rate, we have survival models like Cox Proportional Hazards.
What can we use for discrete event data in continuous time where there is an explicit time-dependence that we want to learn?
I understand that sometimes people use sequential models where each node is the predicted time to the next event. But this doesn't have the robustness of something like a rate function in survival analysis, where you can express a probability of event over any given time window. Ideally I would like to express a probability distribution over counts for any custom time window, with the model.


texts: Comparing Non-deterministic Binary Classifiers

texts:
I have two classifiers which I am implementing, and they are both non-deterministic in the sense that they can each give different results (FPR and TPR) when you run them multiple times. I would like to compare these two algorithms to evaluate their performance. How do I go about this? Usually what people do most times is to run the classifier till they get the best FPR and TPR values, then they publish the results. But the problem with this approach is that it might not be a good representation of the performance of such a classifier. This is what I planned on doing so far, but don't know if that is correct:

Split my evaluation data into train and test, and after training, predict using the test data to get the FPR and TPR, then repeat this prediction for 99 more times to form 100 FPR and TPR readings, then take an average of this. To get an ROC, use the mean FPR and TPR.  OR
Use k fold cross validation of say k=3 or 10 on the data, and this will return 3 or 10 different values for TPR and FPR, then I will take the mean to get the mean FPR and TPR, and also use this mean for plotting the mean ROC.

Which of the two methods I stated above is ok? And if they are both wrong, what do you suggest I do. Thanks.


texts: Using PCA to cluster multidimensional data (RFM variables)

texts:
So i am performing k-means clustering on RFM variables (Recency, Frequency, Monetary). The RFM variables are in the form of quantiles (1-4). I used PCA and found the PCA components. I then used the elbow method to find the optimal number of clusters and then I use it in the k-means algorithm.
Could anyone guide me if this is a correct method? Further, the clusters I get range on the graph, their axis ranges from -3 to 3 and I am not entirely sure why it ranges from that way.


texts: Is Dynamic Time Warping a good loss function for a time series auto-encoder?

texts:
I've been trying to implement a multivariate time-series auto encoder. I thought DTW could be a good loss function but my implementation is still too slow.
Anyone has some ideas of pros and cons of using DTW?
What other loss functions can be good for training the auto encoder? Will MSE do the job?
Thanks :)


texts: Data driven approach to define a churn user

texts:
I'm trying to define a churn prediction model for an online service (betting/gambling). A lot of papers talk about churn analysis/prediction for telco companies where defining a churn user is straightforward: a churn user is a user who cancels his or her contract. I'm wondering how to define a churn user within the context of an industry where users don't cancel any contract/account?

I can envision a general approach: if a user who plays every day for a month stops playing for 2 days, this is could be flagged, but a user who only plays once a week and then has no activity for 2 days would not. What's the best approach to encoding this logic algorithmically? Is survival analysis and good approach?


texts: Positively skewed target label in regression

texts:
I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-network regression models.

I see the same problem with the Boston Housing prediction dataset, and recommendations to apply a log transformation to the target label. This has given some small improvement but not enough. Additionally I've tried to randomly duplicate values within the tail to shift the mean, although I'm not overly comfortable with method.

Are there any alternative transformations to apply, or any models that can put a higher cost weighting on labels with high residuals?


texts: Classification: Dealing with unknown class balance in the prediction dataset

texts:
I'm working on a classification problem that predicts if a grant application will be accepted. The data I'm training on is from 2005 to 2008. I'de like to predict any time after 2008.

The issue I'm running into is that the ratio of successful grant applications differs over time. For example, the success rate in the training set is 20%. But depending on the period I use after 2008, the success rate could be anywhere from 10 to 30%. This results in under or over predicting.

Since the success rate of grant applications is not constant, how do I account for this in my model?


texts:
I'm working on classification with word embeddings that are compared using cosine similarity.
I wanted to use Naive Bayes classifier, and since I want to compare embedding on the unit circle, I though of Directional Naive Bayes (similar to Gaussian Naive Bayes but using the Von Mises-Fisher distribution).
I found this paper describing such classifiers but I cannot found a Python implementation (there is a source code link in the paper but it's dead).
Does anyone know a good implementation in Python?


texts: Anomaly Detection from available sensor data set?

texts:
I am working on a live sensor data set and looking for abnormal patterns (leading to a machine fault condition) from the available data set.

I am learning and new to the world of data science, but comfortable with Python. I have few questions that I am looking to get suggestions.

What kind of algorithms would be best suited for this case?
What are the basic steps for doing a predictive analysis in python?

Please correct me if my questions are not correctly framed.


texts: How do I determine if variables are correlated? Is it simply a mathematical calculation?

texts:
I'm self learning data science so bear with me as I try and make my question as clear as possible. Lets assume I have a dataset of a dependent variable y and some independent variables x1, x2 and x3 etc.
Is the correlation between these variables simply a mathematical calculation? I mean if i ran a correlation test in R for example and the test came back with no correlation between the variables is that necessarily true? What if I knew or suspected that one of the variables is correlated to another how would I define that in my model or would relying on a correlation test be sufficient.


texts: Model selection: large mean and variance vs small mean and variance

texts: This question was always in my mind. Imagine you are doing 5-10 fold cross validation and one model gives you mean accuracy of 0.8, but with 0.2 standard deviation and the other one gives 0.7 with 0.05 standard deviation. Which one is better?


texts:
There isnt a lot of wisdom here but on a regression data set, I can get a slightly lower RSME with adding in a learning rate schedule for exponential decay. The curve on the plot is also slightly smoother too. Its sort of hard to see but this first plot is with adding the exponential decay.

Exponential decay, Final RSME is 6.5

No exponential decay, final RSME is 7.5

Would anyone have any other tips to try for deep learning tuning/better performance? This is my script with tf.keras below. The model depth is something I already experimented with on the layers & amount of epochs...

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import math
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from scipy.stats import t
from tensorflow.keras import backend
import os

# This function keeps the learning rate at 0.001
# and decreases it exponentially after that.
def scheduler(epoch):
  if epoch < 2:
    return 0.001
  else:
    return 0.001 * tf.math.exp(0.1 * (1 - epoch))

callback = tf.keras.callbacks.LearningRateScheduler(scheduler)

#function to calculate RSME
def rmse(y_true, y_pred):
    return backend.sqrt(backend.mean(backend.square(y_pred - y_true), axis=-1))

# load training/baseline dataset
#df = pd.read_csv('School District Test Data/hourly_summary.csv', index_col='Date', parse_dates=True)
df = pd.read_csv('df_ols_noHvac_all.csv', index_col='Date', parse_dates=True)

dfTrain = df.copy()

# split into input (X) and output (Y) variables
X = dfTrain.drop(['Demand'],1)
Y = dfTrain['Demand']

#define training & testing data set
offset = int(X.shape[0] * 0.7)
X_train, Y_train = X[:offset], Y[:offset]
X_test, Y_test = X[offset:], Y[offset:]

#define model
model = Sequential()
model.add(Dense(60, input_dim=42, kernel_initializer='normal', activation='relu'))
model.add(Dense(55, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, kernel_initializer='normal', activation='relu'))
model.add(Dense(45, kernel_initializer='normal', activation='relu'))
model.add(Dense(30, kernel_initializer='normal', activation='relu'))
model.add(Dense(20, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
model.summary()
model.compile(loss='mse', optimizer='adam', metrics=[rmse])

# train model, test callback option for rlrp or no rlrp
history = model.fit(X_train, Y_train, epochs=85, batch_size=1, verbose=2, callbacks=[callback])
#history = model.fit(X_train, Y_train, epochs=85, batch_size=1, verbose=2)

# plot metrics
plt.plot(history.history['rmse'])
plt.title('kW RSME Vs Epoch')
plt.show()

model.save_weights('./saved_model/kwFinal.h5')
print('[INFO] Saved model to disk')


texts: tsm_system_time extension not available on Azure Postgresql

texts: The tsm_system_time extension is not available on Azure Postgresql. Anyway to use Orange without this extension on the server?


texts: Search Query Sample Size Determination for validation set

texts:
While designing a search system, which searches in N identifiable categories, how many search queries does one need in each category to validate the target metric (DCG) scores accurately (balanced variance and bias)? does this number depend on N or the corpus size or both? Please add any publications possible.
I would also like the understand if effect size and and bayesian effective sample sizes play some role here.
Given a set of search queries Q for retrieving documents from another set D, where there are N possible categories in the corpus with k(n) documents in each set, the objective is to define test minset(Q) such that DCG obtained is reliable.

Example: Searching for Fruits
Search queries: [Red apple, Green apple]
Category: Apple
Documents: [Red Apple, Gala Apple, Green Apple,...]k1 documents
Search queries: [Alphonso, Haden, Keitt]
Category: Mango
Documents: [Keitt, Kent, Haden,....]k2 documents
.
.
.
The search results will be ranked and the accuracy of search is measured with DCG. How many search queries do you need in each set to have a reliable DCG as a function of the categories and the document set?


texts: Which graph will be appropriate for the visualization task?

texts:
I have some terminal charging values for US and CHINA
comes in a pandas DataFrame like the following,

       value country
0      550.0     USA
1      820.0   CHINA
2      835.0   CHINA
3      600.0   USA
4      775.0   CHINA
5      785.0   USA
6      790.0   USA

This is the sample data and I have in total 5K+ entries. The data is cleared for the outliers and needs to be visualized. What kind of visualization can I use to plot my data meaningfully?


texts: How do I feed a FFT data as input to a CNN/DNN?

texts:
As shown in the picture, I have some samples of FFT data, out of which some are from the system which are working good, and some from systems which has faults.

I tried calculating PeakValue, RMS and tried to train the network using DNN as well as Linear methods, but in vain.

My question is, Do we have any mechanishm so that we can feed the raw data (without pre processing) and train the system, if not, what pre-processing steps should be taken care and what model of neural network should I deploy ?


texts: What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely helpful for both beginners and experienced.


texts:
I'm making some RFM Analyses (Customer Segmentation) and, in order to feed the RFM data to K-Means, I need to unskew the data, as K-Means works best when dealing with symmetrical distributions.

One of the best practices I've found so far is to use log transforms in order to unskew the data. However, after a bit of experimenting (and, hopefully, correct intuition), I've also found that PCA is working really well at unskewing the data, without the need of performing a log transform. Is this supposed to work or am I in danger of making incorrect assumptions?

On a (cough) safer bet, I was planning on using PCA after a log transform and the scaling of the data, since it would help to orthogonalize it and, I suppose, improve K-Means's performance. Is this better or worse than simply applying PCA without the log transform?


texts:
everyone. I am a Geophysicist who has recently started learning Data Science. As part of this process I am going over some example source code included in the book "Machine Learning for the Oil and Gas Industry". The problem I have is that the source code has the following lines of code:
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential, load_model
from keras.wrappers.scikit_learn import KerasRegressor, KerasClassifier

The first two lines seem to run correctly, however a ModuleNotFoundError is raised for the third line. I use Anaconda individual distribution and installed Tensorflow using Anaconda Navigator in my base environment.
Is there any way to get this to run properly?


texts: resampling image takes a large amount of time

texts:
I am working on medical image exactly CT scan images, there is a method for reading these type of images, also another method for resampling, the code for two methods shown below:

def load_itk_image(filename):
    # read infos from route
    itkimage = sitk.ReadImage(filename)
    # read imagearray
    numpyImage = sitk.GetArrayFromImage(itkimage)
    # read coords
    # the given coords and spacing comes: x, y, z and we should transfer it to z, y, x
    numpyOrigin = np.array(list(reversed(itkimage.GetOrigin())))
    numpySpacing = np.array(list(reversed(itkimage.GetSpacing())))

    return numpyImage, numpyOrigin, numpySpacing

def resample(image,oldspacing,newspacing):
    start_time = time.time()
    resize_factor = np.array(oldspacing).astype(np.float)/np.array(newspacing).astype(np.float)
    new_real_shape = image.shape * resize_factor
    new_shape = np.round(new_real_shape)
    real_resize_factor = new_shape/image.shape
    image = scipy.ndimage.interpolation.zoom(image, real_resize_factor, mode = "nearest")
    print("%s time takes in seconds" % (time.time() - start_time))
    return np.array(image)

my problem is, resample function takes a large amount of time to resample only one 2d image approximatly 30 seconds, why it takes these amount of time? is there any way to reduce resampling time?


texts:
I have a long data frame having 96 consecutive observation belonging to same variable.
I have to two such variables. One variable is Energy denoted by "param2(kWh)" and another is demand denoted by "param2(kW)". So data frame is like---

first row has variables described as "Interval" and "param2(kWh)" and then 96 observations belonging to Energy variable, followed by 98th row having variables described a "Interval" and "param2(kW)" and then 96 observations belonging to Demand variable.

Again after that there is row described as "Interval" and "param2(kWh)" and then 96 observations belonging to Energy variable. After that again there is row describing variables "Interval" and "param2(kW)" and 96 observations belonging to Demand variable and such intertwining of these two variables i.e. Energy and Demand continues for long.

The objective is to have two separate data frames one for Energy i.e. "param2(kWh)" and one for Demand i.e. "param2(kW)".
How these two data frames be formed from the above single data frame?

I am attaching the internet site of the data frame.

https://1drv.ms/f/s!Apm2LEjQkhz3aNBBoUygHPMz2_E


texts: Modulo as activation function in neural network

texts:
Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our input data.
By modulo function $f(x)$ I mean

\begin{equation}
f(x) = \begin{bmatrix}
\vdots \\
  x+a \ \ \  \if \  -a < x < 0 \\
 x  \  \ \ \ \ \ \  \if\  \ \ \ 0 < x < a\\
 x-a \  \ \  \if  \ \ \ \ a < x < 2a \\
\vdots \\
\end{bmatrix}
\end{equation}
where a is a positive constant number and could be treated as hyperparameter for simplicity.

I want my output to take values between [0,1] and I am sampling the output from a gaussian distribution (, $\sigma^2$) where , $\sigma^2$ are the outputs of neural network. Hence the output may go outside the range [0,1]. I don't want to do clipping because it will create further problems in my network

I am new to Latex, sorry for not using a good formating.


texts: How to define person's gender from the fullname?

texts:
Given person's name, e.g. 'Adjutor Ferguson'.

How to define is it a male or female?

One solution came to my mind:

     I have found Person NLP training dataset here mbejda.github.io.

      And via a machine learning software like Apache Mahout, train it and provide real data.

But I am not sure about the accuracy of the results.

May be another approach exist? (e.g. scikit-learn.org)


texts: Can the same CNN architecture be used for different data sets?

texts: I have a CNN architecture that works well on 32x32x3 images. Can I use that same architecture for a data set made up of 28x28x1 images? (Both data sets have 10 classes). If this is possible, what changes would I need to make to the architecture I have?


texts: Site packages location not shared between Python and Jupyter Lab

texts:
I appreciate the fact that Jupyter runs in an isolated mode. I read several posts about it by now.

What I don't understand is why the JUPYTER_PATH variable is ignored as well as appending manually (as a proof of concept) the path of the current site packages from my brewed Python dir.
I couldn't find any documentation specific for the Lab so I assumed this should have worked out of the box.

Any idea on how to avoid installing all the necessary packages via a !{sys.executable} -m pip install .... from within Jupyter?

Of course modifying the sys.path from within the environment is one option, but this has to be done on a per notebook basis, which is not ideal.

I'd like to install things from command line via pip and be able to use them immediately.

Thanks


texts: What are the cases where it is fine to initialize all weights to zero

texts:
I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, breaking symmetry.

However, there were other cases where I saw people initializing using zero weights. Unfortunately, I can't remember what those were. I think it might have been non-neural-network cases, like a simple linear or logistic regression model (simple weights only on the inputs, leading directly to an output).

Are those cases safe for zero initialization? Alternatively, could we use random initialization in those cases too, just to stay consistent?


texts: What it means for a neural network to memorize data

texts:
I've seen at many places that sometimes neural networks simply memorizes training data. What that means actually ?

Neural network consists of bunch of weights which gets trained and outputs based on input data. It'll output different thing for different input. Where does this memorization comes into play ?


texts: Any free or affordable resources for network training?

texts: I have a really large network that I need to train but I don't have enough computational power. I tried using Colab but colab doesn't allow multiple parallel runtimes. I'd like to use the caiver approach for finetuning my learning rate and other hyperparameters but I can't afford to babysit as it'll take me years. Could you suggest any online resources like Colab, Kaggle and such?


texts: Which features can help to differentiate these two density?

texts: I'm wondering that is there any features that can help in differentiating the following two images. I mean differentiating in related numbers.


texts: Voting combined results from different classifiers gave bad accuracy

texts:
I used following classifiers along with their accuracies:

Random forest - 85 %
SVM - 78 %
Adaboost - 82%
Logistic regression - 80%

When I used voting from above classifiers for final classification, I got lesser accuracy than the case when I used Random forest alone.

How is this possible? All classifiers are giving more or less same accuracies when used individually, then how does Random Forest outperform their combined result ?


texts: Are there any object detection datasets with the same format as PASCAL VOC?

texts: I was wondering if there are any object detection datasets that have the same format as the PASCAL VOC dataset. Furthermore I would be specifically looking for satellite imagery datasets in that regard but anything would be fine.


texts:
I have installed Orange 3 in Ubuntu 18.04 using Anaconda. It runs just fine, but the menus appear as blank.

I obtain the following error when I execute it:

QGtkStyle could not resolve GTK. Make sure you have installed the proper libraries.

I have been trying to sort it for days without success.

Any idea of how to fix this?


texts: What are good practices in reporting RMSE or MAPE estimates for a machine learning model?

texts:
I have built a regression model to estimate prices given some attributes. Prices are expressed in million USD, with average of 4.4 million USD. I need to provide an estimate of the RMSE and MAPE of the model. The average RMSE and MAPE of my model seem too high, with values of 3.4 million USD and 515 % respectively. However, looking at the RMSE and MAPE distribution in relation to the prices, I can see that there are a few, very low prices, that are difficult to predict and cause the very high mean error estimate. The scatterplot below shows price on the x axis and MAPE on the y axis

If I eliminate those 9 'outliers', the RMSE and MAPE go down to more reasonable values, namely 0.95 million USD and 26.7% respectively. My questions are:

should I eliminate those 'outliers' (i.e. very low prices) before training?
or should I train the model to predict all the data, and then exclude the outliers from the final RMSE/MAPE estimation?
or is there any other best practice in providing the most realistic estimation of the error
or, is there any other error metric that is robust to outliers?

Thank you very much


texts: Why do we pick random features in random forest

texts:
I understand that random forest is a stylized version of bagging of trees. We choose randomly data points as well as random features for constructing random forest.

But if we use just plain version of bagging by choosing only data points randomly then we have trees which have trained on more number of features unlike the random forest in the stylized version. Since learning with more features every individual tree has more information about the data points and so are more 'intelligent' in some sense than the individual trees in the random forest.

So why does the random forest using stylized version of bagging performs better than the random forest using plain implementation of bagging?

I understand that the random forest using the stylized version gives a model lower variance but since each of its trees is trained on some of the features shouldn't it make the model a bit  high biased?


texts: How to normalize just one feature by scikit-learn?

texts:
Wanna apply a specific scaler, say StandardScaler, on a specific feature, keeping other features intact.

the dataset format is something like:
[
[1, 0.2, 1000],
[2, 0.1, 2400],
[3, 0.9, 7620]
]

I need to transform only one column, the third in this example.

I don't want to use pandas.


texts: Why infinite sampling is not realisitc assumpition in most real applications

texts:
I came across the below paragraphs, which I believe are the answers to the question Why infinite sampling is not realistic assumption in most real applications. Still i dont get the below explanation ?. When we draw more samples from the environment, MC brings the approximate value function close to the true value function isn't it ? Then why infinite sampling is not considered as a realistic assumption.

We made two unlikely assumptions above in order to easily obtain this guarantee of convergence for the Monte Carlo method. One was that the episodes have exploring starts, and the other was that policy evaluation could be done with an infinite number of episodes. To obtain a practical algorithm we will have to remove both assumptions. We postpone consideration of the first assumption until later in this chapter.
The second approach to avoiding the infinite number of episodes nominally required for policy evaluation is to forgo trying to complete policy evaluation before returning to policy improvement. On each evaluation step we move the value function toward , but we do not expect to actually get close except over many steps. We used this idea when we first introduced the idea of GPI in Section 4.6. One extreme form of the idea is value iteration, in which only one iteration of iterative policy evaluation is performed between each step of policy improvement. The in-place version of value iteration is even more extreme; there we alternate between improvement and evaluation steps for single states.
For now we focus on the assumption that policy evaluation operates on an infinite number of episodes. This assumption is relatively easy to remove. In fact, the same issue arises even in classical DP methods such as iterative policy evaluation, which also converge only asymptotically to the true value function. In both DP and Monte Carlo cases there are two ways to solve the problem. One is to hold firm to the idea of approximating in each policy evaluation. Measurements and assumptions are made to obtain bounds on the magnitude and probability of error in the estimates, and then sufficient steps are taken during each policy evaluation to assure that these bounds are sufficiently small. This approach can probably be made completely satisfactory in the sense of guaranteeing correct convergence up to some level of approximation. However, it is also likely to require far too many episodes to be useful in practice on any but the smallest problems.


texts: Can an R^2, or coefficient of determination be used on non-linear data?

texts: I have used the $R^2$ metric to determine how well my neural network performs a non-linear regression. And it seems to work. The plots look almost identical, and I get an $R^2$ value of 0.93... it seems to work perfectly,  however I have been told that $R^2$ should only work for linear data. Is this true or false?


texts: Is there something like class-based object detection? Or class-based selective search?

texts:
I've been reading a lot about computer vision lately, and while there is a huge amount of info about object classification, and a lot less on object detection, I have not found anything on class-based object detection i.e when I know what I am looking for. For example, looking for cats on a picture. Nowadays you can say if a picture is a cat (object classification), or if the picture has cats among other things.
How would I use this knowledge to improve the performance (in both accuracy and speed)?

Somethings that I wouldn't do (but i'm tempted to do):

Perform a brute-force solution with a binary classifier
Implement a shape-based selective search and then use a binary classifier

What i would really like to do, but haven't found in the literacy:

Implement a trainable selective search to find class-based regions of interest.
Use a binary classifier to check if ROI is in fact the object to be found.


texts:
I understand that so long as i am defining a computation in tf.GradientTape() context, the gradient tape would compute the gradient w.r.t all the variables that the output of the computation depends on. However, i think i am not quite grasping the subtelties of the gradient as the following code does not execute as i expect it to:

import tensorflow as tf
x = tf.Variable(2.)
loss_ = x**2-2*x+1
with tf.GradientTape(persistent=True) as g:
    loss = loss_*1
print(g.gradient(loss,x))
output: None

Why is the gradient wrt x not computed?

I am able to compute only the gradients that are wrt to the variables that are getting explicitly used within the context. for example the following code does not compute gradients as well:

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
x = tf.Variable(2.)
t1 = x**2
t2 = -2*x
t3 = 1.
with tf.GradientTape(persistent=True) as g:
    loss = t1+t2+t3
print(g.gradient(loss,x))


texts:
In the overview page of the pandas documentation the Series data structure is described as 'homogeneously-typed'.

Data Structures
Dimensions  Name    Description
1   Series  1D labeled homogeneously-typed array
2   DataFrame   General 2D labeled, size-mutable tabular structure with potentially heterogeneously-typed column

However it is possible to create Series objects with multiple data-types.

pd.Series(data=[1,2,3,4,5,'x'], index=['a','b','c','d','e','f'])
#=>a    1
#  b    2
#  c    3
#  d    4
#  e    5
#  f    x
#  dtype: object

So what could be the meaning of homogeneously-typedmentioned in pandas documentation?


texts: Training InceptionV3/Resnet on custom data with CoreML/CreateML

texts:
I'm new to the CoreML/CreateML scene. I know that there is an image classifier built by Apple that can be easily trained by drag/dropping data, with pretty good accuracy.

What I am wondering is, is there a way to train InceptionV3 or ResNet or something like that on custom data using CoreML/CreateML? What would the process for that be? Thanks!


texts: Appending the values of multiple columns into one whilst retaining UIDs

texts:
I have a table structured as below:

UID|ColA|ColB|ColC|ColD

I want to restructure so that all columns are combined (appended) into one, retaining the UID of each as below:

UID|ColA
UID|ColB
UID|ColC
UID|ColD

I'm very new to Pandas and was wondering how this could be done? Thank you!


texts: How CNN applies backpropagation to update its weights and biases?

texts:
I understand that the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer.

However, I do not understand how CNN updates its weights and biases using backpropagation.

I understand that backpropagation uses partial derivatives. But I do not see how they are used in CNN.

Any reference is appreciated.


texts: "Trending" feature to predict number of views

texts:
I am working on a problem where I have access to a database with news articles, their publication date and the number of views they got 24hrs they got published.

The objective is to be able to predict the number of views for any new article.

I want to add a feature that, for a given article, gives an indication of the number of views / publications of all articles with a similar theme that happened a short time before the publication of this article. This would capture the "buzz" phenomenon. (If there has been a lot of articles on elections, then it is quite likely that a new article on elections will get lots of views.)

Here is my strategy :

We classify all articles among X themes.

We segment the timeline into Y timesteps.

For each timestep:

For each theme:

we count the number of views of articles of this theme published in this timestep
ditto for the number of publications
we add these two values as features for all articles published in the next timestep

So, I was wondering if this violates the i.i.d. assumption? (Because, if I randomly separate the data into train/test, this injects information from the target values of the test set as a feature of the training set.) Is this "allowed"?

Am I missing something? Do you see another strategy?


texts: Spark DataFrame "Limit" function takes too much time to display result

texts:
import pyspark
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
import findspark
from pyspark.sql.functions import countDistinct
spark = SparkSession.builder \
.master("local[*]") \
.appName("usres mobile related information analysis") \
.config("spark.submit.deployMode", "client") \
.config("spark.executor.memory","3g") \
.config("spark.driver.maxResultSize", "1g") \
.config("spark.executor.pyspark.memory","3g") \
.enableHiveSupport() \
.getOrCreate()

handset_info = ora_tmp.select('some_value','some_value','some_value','some_value','some_value','some_value','some_value')
I configure the spark with 3gb execution memory and 3gb execution pyspark memory. My Database has more than 70 Million row. Show I call the
 handset_info.show()

method it is showing the top 20 row in between 2-5 second. But when i try to run the following code
mobile_info_df = handset_info.limit(30)
mobile_info_df.show()

to show the top 30 rows the it takes too much time (3-4 hour). Is it logical to take that much time. Is there any problem in my configuration.
Configuration of my laptop is:

Core i7 (4 core) laptop with 8gb ram


texts: Why feature crosses may work better than having them as individual features?

texts:
On Google ML Crash Course it is said the following:

  If we build a feature cross from both these features:

  [behavior type X time of day]

  then we'll end up with vastly more
  predictive ability than either feature on its own. For example, if a
  dog cries (happily) at 5:00 pm when the owner returns from work will
  likely be a great positive predictor of owner satisfaction. Crying
  (miserably, perhaps) at 3:00 am when the owner was sleeping soundly
  will likely be a strong negative predictor of owner satisfaction.

Why cross features is better than having them as individual features?


texts: Encoding categorical variables using likelihood estimation

texts:
I am trying to understand how I can encode categorical variables using likelihood estimation, but have had little success so far.

Any suggestions would be greatly appreciated.


texts: Why do people use CrossEntropyLoss and not just a softmax probability as the loss?

texts:
I don't understand why one would add additional complexity to log, probabilities for the loss function of a classification Neural Network. What benefit does that have, as opposed to just using the 0-1.0 values(probabilities of a class) you get from the softmax function at the final layer?

Does this add extra non-linearity that we don't understand why it does good, but just happens to do good a lot of times since we give the Neural Net some more complexity?


texts: Extract title from xml file using python


texts: Is the micro averaged precision/recall/f1 score for multiclass classification always the same?

texts:
I was under the impression from this post that in the micro averaging case for multi-class classification, the precision and recall are the same. This is because the number of false negatives and false positives are equal. Each classification error will cause a false positive for X and a false negative for Y.
However, in both my own experiments and in some from the transformers library, the micro averages for precision and recall are different, even though it is multi-class classification. Here is an example table:
           precision    recall  f1-score   support

 LOCderiv     0.7619    0.6154    0.6809        52
  PERpart     0.8724    0.8997    0.8858      4057
  OTHpart     0.9360    0.9466    0.9413       711
  ORGpart     0.7015    0.6989    0.7002       269
  LOCpart     0.7668    0.8488    0.8057       496
      LOC     0.8745    0.9191    0.8963       235
 ORGderiv     0.7723    0.8571    0.8125        91
 OTHderiv     0.4800    0.6667    0.5581        18
      OTH     0.5789    0.6875    0.6286        16
 PERderiv     0.5385    0.3889    0.4516        18
      PER     0.5000    0.5000    0.5000         2
      ORG     0.0000    0.0000    0.0000         3

micro avg     0.8574    0.8862    0.8715      5968
macro avg     0.8575    0.8862    0.8713      5968

These examples seem to be calculating the precision/recall from this function. Why would the precision, recall, and F1 be different?


texts:
has number of output layer of DNN any effect in speed of find the optimal answer of DNN?
For instance the more episodes is needed to train a DNN when the number of outputs is more? Is it correct?


texts: Theoretical treatment of unlabeled samples

texts:
In a typical supervised learning setting with a few positive and a few negative examples, it is clear that unlabeled data carries some information that can benefit learning and that is not captured in the labeled data. For example one can estimate mean values, bounds and some other geometrical characteristics of the data-set with much higher precision if you do not discard the (massive) unlabeled data.

On the other hand, the most common ML algorithms from Neural Networks to SVM do not take advantage of this information (at least in their standard, most common form). My question:

Is there any theoretical framework where unlabeled data is treated in the supervised setting?

I can think of semi-supervised ways to approach this (first cluster and then label the clusters). Are there any other?


texts: Automatic question categorization when we know important words in each category

texts:
I am currently working on a question categorization problem where I automatically want to assign a category to the question. The question set I have is unlabelled. The categories for the problem are well defined e.g. Business, Technology, Sports etc. I also have a specific set of words which are important for each of the category (which have been taken from the dataset) and which define the respective categories. Using these information, which learning algorithm can I use wherein I can specify these categories and the important words for that category in context of the unlabelled data that I have?

Repsonses are highly valued.
Thanks in advance.


texts: Problem to display decision tree: InvocationException: GraphViz's executables not found in Python scikit-learn?

texts:
I try to display Decision Tree in Jupyter Notebook in Python.

This is my code:

X = data.drop(["Risk"], axis=1)
y = data["Risk"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
from sklearn.tree import DecisionTreeClassifier
klasyfikator = DecisionTreeClassifier(criterion = "gini", random_state=0, max_depth=4, min_samples_leaf=1)
klasyfikator.fit(X = X, y = y)

dot_data = StringIO()
export_graphviz(klasyfikator, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True,feature_names = X.columns,class_names=['0','1'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png('diabetes.png')
Image(graph.create_png())

And by using above code I have Error:

InvocationException: GraphViz's executables not found

What can I do to display decision tree ? Where is a mistake ?


texts: How would you encode missing pixels in image data?

texts:
I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it.  Since the values are always positive, and normalized between 0 and 1, would it make sense just to encode it as -1 or something.
Thanks


texts: Collaborative filtering with human-adjusted latent factors

texts:
Having tried some of movie recommendation engines available on the web I have the feeling they are not satisfactory. I just fail to get movies similar to those I like based on traits interested for me personally.

My guess is that the lack of precision can be overcome with extending the number of latent factors in CF SVD algorithm. Another idea is that latent space can be initially hand-crafted.

E.g. we can take 50 well-studied movies and asses them by 200 traits manually (partially fill item-latent factors matrix). Accept this values as fixed. Then having some of the user ratings we can perform matrix factorization and inherit user-factors and utility matrices as usual, while keeping fixed values.

Any present work on this?


texts: How to ensure diversity in my recommended ranking?

texts:
I have generated a ranked list of items but I want to ensure that the ranking takes care of diversity basis some item metadata. Most of the way I can think of seems computationally expensive. Is there a faster way to do this?
Even if the rankings are approximately diverse, it should be fine.


texts: Square Root Regularization and High Loss

texts:
I am testing out square root regularization (explained ahead) in a pytorch implementation of a neural network. Square root regularization, henceforth l1/2, is just like l2 regularization, but instead of squaring the weights, I take the square root of their absolute value. To implement it I penalize the loss as such in pytorch:

for p in model.parameters():
  loss += lambda * torch.sqrt(p.abs()).sum()

p.abs() is the absolute value of p, i.e the weights, torch.sqrt() is the square root and .sum() is the sum of the result for the individual weights. lambda is the penalization factor.
With no regularization, the loss settles around 0.4. With lambda=100, the loss settles somewhere around 0.45, if I use l2 or l1 regularization. Interestingly enough with lambda=0.001, the final value for the loss is around 0.44. Now if I use l1/2 with the same lambda it settles around 5000! This just does not make sense to me. If the regularization is such an overwhelming factor in the loss, then SGD has got to bring the (absolute value) of the weights down until, the penalty from the regularization is balanced with the actual classification from the Cross Entropy Loss I'm using - I know this is not the case, because the train and validation accuracies, are around the same of the original network (without regularization) at the end of training. I also know that the regularization is indeed happening in all three cases, as the loss in the early epochs differ significantly).
Is this to be expected or is this some error in my code or pytorch's SGD?

One more note If use l1/2 with a small lambda like 0.001, the loss comes down to around 0.5 and then becomes nan around epoch 70. For lambda=0.01, it becomes ~1.0 and then nan around the same epoch. For lambda=0.1, loss becomes 5 but no nan anymore for this lambda or any value higher (in 120 epochs total). For lambda=1.0, loss settles at ~50- as expected: apparently the weights settle down at a point where the sum of their square roots equals ~ 50, regardless of lambda...


texts: How do I know the best pruning criteria for decision trees?

texts: Right now,I am working on decision trees on python,how do I know what would be the best pruning criteria based on my data?


texts: I am working on a problem where I need to identify the most likely used routes for a number of trips. The trips have their own starting and ending locations, along with distances covered and durations of these trips. I am not sure how to use the underlying road maps. Is indirectly modeling the route segments from these attributes a good idea? Any suggestion on how could I implement and test it? Some alternatives?


texts: What did Geoffrey Hinton mean when he said this?

texts:
I was going through some lectures from the Deep Learning course that Geoffrey Hinton taught on Coursera and I came across this statement:

"RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects."

I have no idea what he meant nor seen/figured out any examples of this idea (and I'm sure other people are wondering too.) So if somebody would care to demonstrate, it would be much appreciated.


texts: How to feed data to Machine Learning Model?

texts:
I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression Omnibus.

Link to data set. The data set is related to Thyroid cancer. As you scroll down you can see a table. My problem is how can I feed that data to my model.

I don't have much idea on dealing with the data set. Could you tell me more about the data set from the mentioned website? I didn't understand it properly.

I'd be happy if you help me out with this. I'm open to all kinds of suggestions and ideas. If you know about any another data set repos, let me know.

Have a great day.


texts:
I seem to have a problem modelling my CNN network.

I want to extract from features vector from different sized images.
Whats consistent with the images is the y-axis, and the color dimension, but the x-axis is not constant.

Depending on the length of the x-axis will the length of the feature vector also be altered. I already know how long they should be,
but not the ratio between the length of the x-axis and length of the feature vector, I guess one does exist.

Is it possible to train a CNN network such that it can alter the feature vector length depending on the length of the input of the x-axis of the image?


texts: RNN package and problems with "Predictr"

texts:
I have two questions about how to use R's RNN package, specifically the trainr and predictr functions.

Let's suppose I have a time series of 4000 steps for 5 different variables.

How should this be passed on to the RNN, since the input has to be a 3D array? Should it be an array of dim(1, 4000, 5) or dim(4000,1,5) or something completely different?

Moreover, let's suppose I successfully train my RNN and would like to make some predictions using the predictr function.

Which dimension will the output have and how can it be interpreted?

Until now I have tried to input the data as 3D array (4000, 1, 5) with the following code:

RecNN <- trainr(TrainR, TrainI,
  learningrate = 0.003, momentum = 0.003, hidden_dim = c(100),
  network_type = "rnn", batch_size = 50,
  numepochs = 100, sigmoid = "logistic", use_bias = TRUE)

print("--- DONE LEARNING ---")

And then used the predictr function with the input (250, 1, 5):

results1 <- predictr(RecNN, TestI, hidden = TRUE, real_output = TRUE)

List of 2
 $ : num [1:250, 1, 1:100] 0 0 0 0 0 0 0 0 0 0 ...
 $ : num [1:250, 1, 1:2] 1 1 1 1 1 1 1 1 1 1 ...

Thanks a lot!


texts: VAE: first hidden layer of encoder having more neurons than the input layer

texts:
Many blog posts explain the architecture of variational autoencoders (VAEs) with symmetrical encoder-decoder architecture where the number of neurons in the encoder's hidden layers decreases, starting from the same neurons as the input layer. For instance, [128, 64, 32, 16] where 16 is the latent dimension and 128 equals the number of input features/dimension, and the architecture of decoder is [16, 32, 64, 128].
It is obvious that symmetrical encoder-decoder architecture is the matter of choice, and asymmetrical structure is still valid.
However, I'm puzzled about the number of neurons in the first hidden layer of the encoder: is it a must that the first hidden layer of an encoder to have neurons less than or equal to the size of the (flattened) input layer? E.g., assuming my input layer has 128 neurons, is it "valid" to structure the number of neurons of the encoder as [256, 128, 64, 32, 16]? (I'm using the power of two only for clarity.)
I know, in general, in neural networks it is a valid architecture for the first hidden layer having more neurons than the input, and the number of neurons in subsequent hidden layers not to be in decreasing order. However, my question is particularly in the context of VAE. Also, the motivation of having layers with a decreasing number of neurons for the encoder and increasing for the decoder in VAE is clear.


texts: E-step for EM algorithm for document clustering

texts:
I have a code for the E-step in the EM algorithm for Document Clustering in the version of hard-EM algorithm. I'm trying to implement the E-step for soft-EM algorithm. Here is my code for Hard-EM:

E.step <- function(gamma, model, counts){
  N <- dim(counts)[2] # number of documents
  K <- dim(model$mu)[1]
  for (n in 1:N){
    for (k in 1:K){
       gamma[n,k] <- log(model$rho[k,1]) +  sum(counts[,n] * log(model$mu[k,]))
    }
    logZ = logSum(gamma[n,])
    gamma[n,] = gamma[n,] - logZ
  }
  gamma <- exp(gamma)
  return (gamma)
}

I want to implement a E-step for SOFT - EM algorithm. Any suggestions will be helpful.

Thanks in advance.


texts: Show overall average on plots of single data

texts:
I have data from a survey from multiple locations (1 data set per location per year, each containing about 20-25 numerical scores from 0 to 5). This data is aggregated over topics (e.g. first 5 questions are Topic 1, questions 6-10 are Topic 2) so I can calculate the average score per topic, per location, per year.
I would like to show radar charts showing the values for one location and comparing it with the overall average; I thought about creating a fake data set, call it "Average" and always plot it together with the individual locations; I am new to PowerBi but I believe this is not the way to go and that I am missing an option to implement it that does not require fake data sets.
How do I show overall average when plotting single data sets?
Edit:
I was able to find an answer on the PowerBi forum: https://community.powerbi.com/t5/Desktop/Power-BI-Compare-Individual-vs-Average-data-per-month/td-p/355805, but do not understand how to create the "right" measures for it


texts: What exactly is BatchNormalization() in Keras?

texts:
A month or two straight away into building image classifiers, I just sandwiched the BatchNormalization layer between conv2d. I wonder what it does, but I have seen my model learn faster in presence of these layers.
But I'm worried if there's any catch? I read somewhere that I don't need dropout layer if I'm using batch normalization! Is it true?
And also tell me in which manner should I use this layer. In which kind of problems I should and shouldn't use this layer?
Just write down anything you know about the layer that you think will help me!


texts:
I am working on a DC GAN model using my own data set. How can I visualize (see output) of the GAN generator to see how my network is working (replicating the training data)?

Here is my code:

img_rows = 28
img_cols = 28
channels = 1
latent_dim = 100
img_shape = (img_rows, img_cols, channels)

The following is the discriminator function:

def build_discriminator():

    model = Sequential()

    model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=img_shape, padding="same"))
    model.add(LeakyReLU(alpha=0.2))
    model.add(Dropout(0.25))
    model.add(Conv2D(64, kernel_size=3, strides=2, padding="same"))
    model.add(ZeroPadding2D(padding=((0,1),(0,1))))
    model.add(BatchNormalization(momentum=0.8))
    model.add(LeakyReLU(alpha=0.2))
    model.add(Dropout(0.25))
    model.add(Conv2D(128, kernel_size=3, strides=2, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(LeakyReLU(alpha=0.2))
    model.add(Dropout(0.25))
    model.add(Conv2D(256, kernel_size=3, strides=1, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(LeakyReLU(alpha=0.2))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(1, activation='sigmoid'))

    model.summary()

    img = Input(shape=img_shape)
    validity = model(img)

    return Model(img, validity)

The following is the Generator function:

def build_generator():

    model = Sequential()

    model.add(Dense(128 * 7 * 7, activation="relu", input_dim=latent_dim))
    model.add(Reshape((7, 7, 128)))
    model.add(UpSampling2D())
    model.add(Conv2D(128, kernel_size=3, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Activation("relu"))
    model.add(UpSampling2D())
    model.add(Conv2D(64, kernel_size=3, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Activation("relu"))
    model.add(Conv2D(channels, kernel_size=3, padding="same"))
    model.add(Activation("tanh"))

    model.summary()

    noise = Input(shape=(latent_dim,))
    img = model(noise)
    return Model(noise,img)

The following is the training function:

def train(epochs, batch_size=128):

    # Adversarial ground truths
    valid = np.ones((batch_size, 1)
    fake = np.zeros((batch_size, 1))

    for epoch in range(epochs):

        # ---------------------
        #  Train Discriminator
        # ---------------------

        # Select a random half of images
        idx = np.random.randint(0, images.shape[0], batch_size)
        imgs = images[idx]

        # Sample noise and generate a batch of new images
        noise = np.random.normal(0, 1, (batch_size, latent_dim))
        gen_imgs = generator.predict(noise)

        d_loss_real = discriminator.train_on_batch(imgs,valid)
        d_loss_fake = discriminator.train_on_batch(gen_imgs, fake)
        d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)

        # ---------------------
        #  Train Generator
        # ---------------------

        # Train the generator (wants discriminator to mistake images as real)
        g_loss = combined.train_on_batch(noise, valid)

        # Plot the progress
        print ("%d [D loss: %f, acc.: %.2f%%] [G loss: %f]" % (epoch, d_loss[0], 100*d_loss[1], g_loss))

train(epochs=10, batch_size=32)
```


texts:
I was trying to execute a hive query:
select name, count(*)
from amazon
where review != NULL
group by name ;

Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>

Starting Job = job_1564815666993_0001, Tracking URL = http://aamir-VirtualBox:8088/proxy/application_1564815666993_0001/

Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1564815666993_0001

It works only till here. The table is all right but it is just stuck and doesn't work . Can anyone help me please?


texts: Find the best interpolation method for missing observations

texts:
I have a database  which has measurements of objects every day every hour.
However, some data is missing and I don't have measurements for all the hours.
in order to get over this challenge I have used different interpolations methods in order to create this missing data (with pandas).
So now I have several databases with those interpolations methods, and I need only one.
My question is how can I determine which interpolation is the best interpolation method?
I have researched the internet but mainly found explanation abut how to interpolate data  but not how to choose the best method and how can I visualize it.


texts: Feature Importance based on a Logistic Regression Model

texts:
I was training a Logistic Regression model over a fairly large dataset with ~1000 columns.

I did apply scaling of features using MinMaxScaler.

I was wondering how to interpret the coefficients generated by the model and find something like feature importance in a Tree based model.

Should I re-scale the coefficients back to original scale to interpret the model properly?

It will be great if someone can shed some light on how to interpret the Logistic Regression coefficients correctly.


texts:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
style.use("ggplot")
from sklearn import svm

X=[[1,0,0,0,0],
           [0,1,0,0,0],
           [0,0,1,0,0],
           [1,0,0,1,0],
           [1,0,0,0,1]]

y=[0,1,1,1,0]

model=svm.SVC()
model.fit(X,y)
print(model.predict([1,0,1,0,0]))

I was working on this but Iam getting an error as

"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[1. 0. 1. 0. 0.].
Reshape your data either using array.reshape(-1, 1) if your data has a single fe
ature or array.reshape(1, -1) if it contains a single sample.

Iam new to this can u guys help me out?


texts: How does L1 regularization make low-value features more zero than L2?

texts:
Below formulas, L1 and L2 regularization

Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero value. Could you please explain the reason why L1 has more tendency to make zero value? (It's gonna be great if you let me know this reason by using formula(like above equation)!


texts: Why does "classification+regression" work better than "classification" in my NN?

texts:
I am working on a research problem where I got stuck on something which doesn't really make sense to me. To explain the issue I will give an example of a similar problem (face age detection).

Suppose that I have many face images that I precisely know how many **days** old the person in each image. (I need precision in my problem).

I train my model using CNN in three ways.

In the first training, I create a neural network that gives an output of 75 classes (say the ages are 0-74) which tells me the correct age with 30-40% accuracy. Below is the final code piece after the CNN layers:

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=utils.lrelu, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout1_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=utils.lrelu, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout2_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits( logits=y_, labels=y))

n_classes=75 as the number of ages is 75.

I keep everything the same and change the output layer from classification to a regression which results in significantly lower accuracy that each person approx 37 years old. This tells me that my neural network is trying to minimize the error and doesn't learn anything. I keep everything the same except for the output layer and loss function.

y2_ = slim.fully_connected(net1, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))

Up to now, nothing sounds weird to me (although I am a newbie in DL). The weird thing is happening when I decide to use two heads instead of a single head. I split the network into two right after the last convolutional layer.

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')
net2 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=activation, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=activation, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y_, labels=y))

# Fully-connected regressor
net2 = slim.fully_connected(net2, first_fc, activation_fn=activation, scope='cmi_ff1')
net2 = slim.dropout(net2, prob, scope='dropout_net1')
net2 = slim.fully_connected(net2, second_fc, activation_fn=activation, scope='cmi_ff2_net2')

y2_ = slim.fully_connected(net2, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))

Here, the accuracy of classification goes up to 65-70% and regression remains to the same (converges to the middle value). While regression seems to be useless in the entire process, somehow it gives an amazing push to the classification to improve accuracy that I can't even speculate about the reason.

I am adding a figure that shows the accuracy and loss of the two-headed network. (I was trying to optimize some hyperparameters.)

Explanation of the figure:

subplot 1,1: Training accuracy for classification head
subplot 2,1: Validation accuracy for classification head
subplot 1,2: Loss for classification head
subplot 1,3: Loss for regression head
subplot 2,2: Total loss: 0.2* regression + classification

I wonder what could be the reason for this or is there something that I do wrong? Also, I know the code is not complete but I can't share the entire code. I just need some directions about what to think and where to look.


texts: How does " Sparsity of connections" in CNNs causes the network to have less parameters?

texts:
I am studying Andrew NG's lectures on Convolutional Neural Network and he had provided two reasons for CNNs having less parameters compared to Non-Convolutional networks . They are :

Parameter Sharing
Sparsity of connections .

While I can make sense of the first reason causing CNN to have less parameters . I don't understand why Sparsity of connections , that is , " Each output in a layer comes from a small number of inputs " cause the network to have less parameters .
Isn't the second reason a bit redundant ?
Can someone please explain ?


texts:
Logic often states that by underfitting a model, it's capacity to generalize is increased. That said, clearly at some point underfitting a model cause models to become worse regardless of the complexity of data.

How do you know when your model has struck the right balance and is not underfitting the data it seeks to model?

Note: This is a followup to my question, "Why Is Overfitting Bad?"


texts: Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

texts:
My understanding is that:

Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables.
Missing not at random: When the propensity for a variable's value to be missing depends on the value.

But what about when the variables are correlated, as they often are?
To make things more concrete, let us consider an experiment where we are collecting data on temperature, humidity, and CO2, and let us suppose that there relationship between these is T = H = C.
Say that we are missing all CO2 variables below 50, because the sensor freezes.
In this case, it is
Missing at random: Because the propensity of CO2 to be missing is dependant on the value of temperature and humidity.
Missing not at random: Because all CO2 values below 50 are missing.
Since the variables are interlinked, missing at random => missing not at random.
Or have I made a mistake in my reasoning somewhere?


texts: Is there any use to running Pandas on Spark?

texts:
I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark?

Isn't the whole purpose of running scripts on PySpark to overcome the limitations of packages like Pandas?

Does Pandas performance improve if it run on Spark? Is it compatible with Spark's RDD ?


texts: Variance in cross validation score / model selection

texts:
Between cross-validation runs of a xgboost classification model, I gather different validation scores. This is normal, the Train/validation split and model state are different each time.

flds = self.gsk.Splits(X, cv_folds=cv_folds)
cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=xgb_param['n_estimators'], nfold=cv_folds, flds, metrics='auc', early_stopping_rounds=50, verbose_eval=True)

self.model.set_params(n_estimators=cvresult.shape[0])

To make the parameters selection, I run multiple times this CV and average the results in order to attenuate those differences.

Once my model parameters have been "found", what is the correct way to train the model, which seems to have some inner random states ?

Do I :

train on the full train set and hope for the best?
keep the model with the best validation score in my CV loop (I am concerned this will overfit)?
bag all of them?
bag only the good ones?


texts: Proper way of fighting negative outputs of a regression algorithms where output must be positive all the way

texts: Maybe it is a bit general question. I am trying to solve various regression tasks and I try various algorithms for them. For example, multivariate linear regression or an SVR. I know that the output can't be negative and I never have negative output values in my training set, though I could have 0's in it (for example, I predict 'amount of cars on the road' - it can't be negative but can be 0). Rather often I face a problem that I am able to train relatively good algorithm (maybe fit a good regression line to my data) and I have relatively small average squared error on training set. But when I try to run my regression algorithm against new data I sometimes get a negative output. Obviously, I can't accept negative output since it is not a valid value. The question is - what is the proper way of working with such output? Should I think of negative output as a 0 output? Is there any general advice for such cases?


texts: Streamlit widgets doesn't display in Jupyter Notebook

texts:
I'm trying to create streamlit widgets in Jupyter Notebook but when executing the following code nothing happens.
import numpy as np
import streamlit as st

age = st.selectbox("Choose your age: ", np.arange(18, 66, 1))

According to the following article a SelectBox like this should be created.

But in my code I only get to know that print(age) is 18. What am I doing wrong?


texts: I am doing a project where I will have to detect emotions from reviews gained by users (through comments). My task is to classify the comments in at least 5 types of emotions that is: happiness, satisfy/unsatisfied, disappointment, sadness, joy. Can I find an already existing model trained for emotion analysis where I can analyze the emotions in each comment. My output needs to give whether the comment is happy or sad/angry etc. I am willing to use the RNN technique. Can someone please help me with my project? Thank you.


texts: Yarn timeline recovery not enabled error upgrading via ambari

texts:
Using the automated upgrade, when I try to upgrade I get:

“YARN Timeline state preserving restart should be enabled
Reason: YARN should have state preserving restart enabled for the Timeline server. The yarn-site.xml property yarn.timeline-service.recovery.enabled should be set to true”

However, using Ambari, whenever I go to change the setting in yarn-site.xml it gets me to create a new configuration group and changes it there. I have moved all of the servers into that config group but the check still fails.

How do I modify the default yarn-site.xml?
Alternatively, does anyone know the actual script (or how to find it) that is kicked off when the automated upgrade is invoked?


texts: How much batch effect is too much batch effect?

texts:
Algorithms such as ComBat/SVA are powerful tools for the removal of batch effects. Small batch effects can be confidently removed by these methods. But surely there must exist batch effects which are so large that even these algorithm cannot help. (For example, near perfect PCA clustering into the batches.) The only remedy is to re-measure the data.

Is there a measure which allows me to distinguish between small and large batch effects? Is there a reliable theoretical foundation for normalization and imputation? What should I read/google?


texts:
I am trying to find a way to cluster/group students by their knowledge of different subjects.

Given following as an example:

            Subject 1   Subject 2   Subject 3
Student 1       4           5           1
Student 2       4           5           2
Student 3       5           2           1
Student 4       2           5           5
Student 5       5           5           2
Student 6       4           5           1
Student 7       2           2           5
Student 8       4           4           2
Student 9       1           2           1
Student 10      1           1           3
Student 11      1           2           1
Student 12      3           1           4

Also given that the number of students per group should be between 3-4 (could be anything, I just felt that for this example those boundries make sense)

I would like to group students in such a way, that after grouping, the average value of a subject for a given group would approach the average value for that subject to the best of the ability.

Maybe one way to define this would probably be minimalizing the total variance of each subject between group and total.

How would you suggest approaching this problem? What algorithm would you suggest to use?


texts: What statistical method can be applied in my case?

texts: I want to study the impact of 26 parameters on one variable, and therefore determine the 3 or 4 which have to most influence on it. For that, I have constructed a 10 x 26 matrix: 26 parameters with 10 observations, and a vector of 10 arrays containing the variable I want to study with 10 observations. From this, how can I determine which are the 3 or 4 most influential variables on the parameters. I was considering using PCA, but I'm not sure that's the right approach. Can anyone give me some direction here?


texts: Why are Chunking and IOB tags necessary?

texts:
I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions:

Why and how is Chunking helpful?
Plus aren't POS tags enough?
Why are three tags necessary for the representation? Why not just use 'inside' tag and 'outside' tag?


texts: Anomaly detection on a time-series data in a CSV format using python

texts:
I have a time versus current data for a days work which is as follows:

11/15/2016  5:12:27 10:42:27    26.61
11/15/2016  5:12:28 10:42:28    42.27
11/15/2016  5:12:29 10:42:29    25.48
11/15/2016  5:12:30 10:42:30    24.24
11/15/2016  5:12:31 10:42:31    25.91

The first column being the date
The second column being the time in GMT
The third column being time in IST
The fourth column being the value of the current used by the machine

Can someone suggest me an algorithm for finding the anomalies in the pattern plotted by current versus time in IST?
Any help regarding my approach will be appreciated too.


texts: K-fold crossvalidation: how do MSE average and variance vary with K?

texts:
I'd like to get an intuition about how varying k impacts k-fold validation.
Is the following right?

Average of the OOS MSEs should generally decrease with k
Because, a bigger "k" means the training sets are larger, so we have more data to fit the model (assuming we are using the "right" model).

Variance of the OOS MSEs should generally increase with k.
A bigger "k" means having more validation sets. So we have have more individual MSEs to average out. Since the MSEs of many small folds will be more sparse than MSEs of few large folds, variance will be higher.


texts: Question about scaling in (R) Principal Components Regression

texts:
I'm reding Introduction to Statistical Learning (ISL), Ch. 6 Principle Component Regression. When I look at the Labs, I wonder how scaling is handled in R in this case.
The ISL-Lab "Chapter 6 Lab 3: PCR and PLS Regression" suggests the following approach:
pcr.fit=pcr(Salary~., data=Hitters,subset=train,scale=TRUE, validation="CV")
pcr.pred=predict(pcr.fit,x[test,],ncomp=7)

Here train and test are lists of row indexes, indicating test and training data, like:
> train
  [1]  70  98 150 237  53 232 243

Question:
During training the data is scaled by scale=TRUE. However, scaling appears to be only used for subset=train if I'm not wrong (which seems correct).
However, when predicting on x[test,], no scaling seems to take place. My take would be that the same scaling should be applied to the test data as was used to scale the training data in order to produce correct predictions.
Am I wrong here? Or is there anything going on in the background with pcr() which makes scaling the test set unnecessary?
Unfortunately the DOCS come with no details on scale() with pcr().
What is the correct approach here?


texts: Machine learning algorithm that uses the Pearson or Spearman correlation?

texts:
I've come across linear and multiple regression, SVM, random forests. Does any know of a machine learning algorithm that uses the Pearson correlation or Spearman correlation?
Best,
Dave


texts: How does a Decision Tree handle an unknown category in a split?

texts:
I have in mind a situation where a Decision Tree is trained with a dataset where one category has got just three possible values: A, B and C.

So as I understand the node for this category will have three splits: A, B and C.

What will happen in case an observation with a null for this feature arrives, or if in the testing set appears a value D for this category?

Does Decision Tree leave one of the splits as default to handle this situations?


texts: Drastic increase in accuracy while using pickle file with sklearn

texts:
I trained a xgboost classifier and it gave an accuracy of 49.99 % and i saved that model into a pickle file. When i ran the same data with pickle file (.pkl) it's giving an accuracy of 88.99 percent. I don't know why it's happening. Please help me out from this situation.
bank_dataset = pd.read_csv(r"dataset.csv")

missing_val = pd.DataFrame(bank_dataset.isnull().sum())

bank_dataset[' Balance'] = bank_dataset[' Balance'].fillna(bank_dataset[' Balance'].mean())

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = bank_dataset.select_dtypes(include = "object").columns

for feat in objList:
    bank_dataset[feat] = le.fit_transform(bank_dataset[feat].astype(str))

correlation = bank_dataset.corr()
print(correlation['Outcome'].sort_values(ascending = False), '\n')

k = 10
cols = correlation.nlargest(k, 'Outcome')['Outcome'].index
print(cols)
cm = np.corrcoef(bank_dataset[cols].values.T)
f, ax = plt.subplots(figsize=(14,14))
sns.heatmap(cm, vmax = .8, linewidths = 0.01, square = True, annot = True, cmap = "coolwarm", linecolor = "white",
           annot_kws = {'size':12}, xticklabels = cols.values, yticklabels = cols.values)

X = bank_dataset.iloc[:, [7,8,12,24,11,16,4,18,20]].values
y = bank_dataset.iloc[:, -4].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

from xgboost import XGBClassifier

xg = XGBClassifier()
xg.fit(X_train, y_train)

y_pred = xg.predict(X_test)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred))
cm = confusion_matrix(y_test, y_pred)

import pickle

filename = 'xg.pkl'
pickle.dump(xg, open(filename, 'wb'))

The above code for training and saving the model into .pkl
bank_dataset = pd.read_csv(r"dataset.csv")

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = bank_dataset.select_dtypes(include = "object").columns

for feat in objList:
    bank_dataset[feat] = le.fit_transform(bank_dataset[feat].astype(str))

X = bank_dataset.iloc[:, [7,8,12,24,11,16,4,18,20]].values
y = bank_dataset.iloc[:, -4].values

file = open('xg.pkl', 'rb')

data = pickle.load(file)

X_ = data.predict(X)

The second code is running the code with pkl file with the same data used for training.


texts: Is the attached workflow correct in case of training and testing with different data sets? Is the attached workflow considered "transfer learning" since my images are not related to the images that the network was pre-trained?


texts: CART classification for imbalanced datasets with R

texts:
Hey guys i need your help for a university project. The main Task is to analyze the effects of over/under-smapling on a imbalanced Dataset. But before we can even start with that, our task sheet says, that we 1) have to find/create imbalanced Datasets and 2) fit those with a binary classification model like CART. So my auestions would be, where do i find such imbalanced datasets? And how do i fit those datasets with CART, and what does that help in regard of over/under-sampling?

Thats my whole first try.

 # CART - Datensatz laden
 setwd("C:\\Users\\..\\Dropbox\\Uni\\Präsentation\\Datensätze")
 add <- "data1.csv"
 df <- read.csv(add)
 head(df) # Ersten 6 Zeilen
 nrow(df) # Anzahl der Reihen des Datensatzes

 # CART - Wichtige Daten selektieren
 df <- mutate(df, x= as.numeric(x), y= as.numeric(y), label=factor(label))
 set.seed(123)
 sample = sample.split(df$x, SplitRatio = 0.70)
 train = subset(df, sample==TRUE)
 test = subset(df, sample==FALSE)

 # grow tree (Baum wachsen lassen)
 fit <- rpart(x~., data = train, method = "class")
 printcp(fit)
 plotcp(fit)
 summary(fit)

 # plot tree
 plot(fit, uniform = TRUE, main="Bla Bla Bla")
 # text(fit, use.n=TRUE, all=TRUE, cex=.08)

 # prune the table --> to avoid overfitting the data#
 pfit<- prune(fit, cp=   fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"])
 plot(pfit, uniform=TRUE,
 main="Pruned Classification Tree for Us")

Why do i need to make such a decision tree and how does it help with Over/Under-Sampling?

Help is much appreciated


texts:
Data:
Each corresponds to an event (a person's visit to the hospital, as an example). I have a series of data associated with this event (duration of visit, motive, etc...).

Objective:
Summarize the above information in a per person data set (meaning that the new data set should have only on row per person and capturing as much information about their history as possible).

My initial solutions:
1 - The most obvious, and potentially useful, is to create relevant variables by hand. For instance, if the objective is to predict the average time of next visit, the average time on the past is relevant.
However, this is very problem specific, and I feel there should be other (not in replacement) options.

2 - Recurrent neural networks. As visits have a time sequence, it seams logical to mean to apply a recurrent autoencoder, in order to summarize this data. (If this is by no means correct, can you point out why?)

3 - What if I don't have a time sequence?


texts: Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?

texts:
Where is the difference between one-class, binary-class and multinominal-class classification?

If I like to classify text in lets say four classes and also want the system to be able to tell me that none of these classes matches the unknown/untrained test-data.

Couldn't I just use all the methods that I mentioned above to reach my goal?
e.g. I could describe C1, C2, C3 and C4 as four different trainings-sets for binary-classification and use the trained models to label an unknow data-set ...

Just by saying, Training-Set for C1 contains class 1 (all good samples for C1) and class 0 (mix of all C2, C3 and C4 as bad samples for C1).

Is unlabeled-data C1 -> 1 or 0

Is unlabeled-data C2 -> 1 or 0
... and so on ...

For multinominal classification I could just define a training-set containing all good sample data for C1, C2, C3 and C4 in one training-set and then use the one resulting model for classification ...

But where is the difference between this two methods? (except of that I have to use different algorithms)

And how would I define a training-set for the described problem of categorizing data in those four classes using one-class classfication (is that even possible)?

Excuse me if I'm completely wrong in my thinking. Would appreciate an answer that makes the methodology a little bit clearer to me =)


texts: Imbalanced data identification with Shannon entropy

texts:
I was studying about the imbalanced data. Then I had a question , how would someone know that which data is imbalanced or not by looking at its percentage(20,30 or 40).
Then I read an answer on stack exchange it says:
You could use the Shannon entropy as a measure of balance.
On a data set of n instances, if you have k classes of size ci you can compute entropy as follows:
H=−∑i=1kcinlogcin.
This is equal to:
0 when there is one single class. In other words, it tends to 0 when your data set is very unbalanced
logk when all your classes are balanced of the same size nk
Therefore, you could use the following measure of Balance for a data set:
Balance=Hlogk=−∑ki=1cinlogcin.logk
which is equal to:
0 for a unbalanced data set
1 for a balanced data set
I understand this part and tried as well but I got a value of "Balance" in decimal(0.79, 0.92 etc)
By looking at this decimal values how would i know that the data is balanced or imbalanced?


texts: Is the way to combine weak learners in AdaBoost for regression arbitrary?

texts:
I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression.
In paper Improving Regressors using Boosting Techniques, the final prediction is the weighted median.

For a particular input $x_{i},$ each of the $\mathrm{T}$ machines makes a prediction $h_{t}, t=1, \ldots, T .$ Obtain the cumulative prediction $h_{f}$ using the T predictors: $$h_{f}=\inf\left\{y \in Y: \sum_{t: h_{t} \leq y} \log \left(1 / \beta_{t}\right) \geq \frac{1}{2} \sum_{t} \log \left(1 / \beta_{t}\right)\right\}$$ This is the weighted median. Equivalently, each machine $h_{t}$ has a prediction $y_{i}^{(t)}$ on the $i$'th pattern and an relabeled such that for pattern $i$ we have: $$
y_{i}^{(1)}<y_{i}^{(2)}<, \ldots,<y_{i}^{(T)} $$ (retain the association of the $\beta_{t}$ with its $y_{i}^{(t)}$). Then sum the $\log \left(1 / \beta_{t}\right)$ until we reach the smallest $t$ so that the inequality is satisfied. The prediction from that machine $\mathrm{T}$ we take as the ensemble prediction. If the $\beta_{t}$ were all equal, this would be the median.

An Introduction to Statistical Learning: with Applications in R: The final prediction is the weighted average.

As such, I would like to ask of the way of aggregation is mathematics-based, or because the researcher feels it's reasonable.
Thank you so much!


texts: What are the consequences of not freezing layers in transfer learning?

texts:
I am trying to fine tune some code from a Kaggle kernel. The model uses pretrained VGG16 weights (via 'imagenet') for transfer learning. However, I notice there is no layer freezing of layers as is recommended in a keras blog. One approach would be to freeze the all of the VGG16 layers and use only the last 4 layers in the code during compilation, for example:

for layer in model.layers[:-5]:
    layer.trainable = False

Supposedly, this will use the imagenet weights for the top layers and train only the last 5 layers. What are the consequences of not freezing the VGG16 layers?

from keras.models import Sequential, Model, load_model
from keras import applications
from keras import optimizers
from keras.layers import Dropout, Flatten, Dense

img_rows, img_cols, img_channel = 224, 224, 3

base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows, img_cols, img_channel))

add_model = Sequential()
add_model.add(Flatten(input_shape=base_model.output_shape[1:]))
add_model.add(Dense(256, activation='relu'))
add_model.add(Dense(1, activation='sigmoid'))

model = Model(inputs=base_model.input, outputs=add_model(base_model.output))
model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

model.summary()


texts: How to resolve name 'log_reg' is not defined

texts:
I am working with the cancer data everything is working perfect but just after the window update i am getting this error

this is my python code

df=pd.read_csv('breast-cancer-wisconsin-data.csv')
mg.plots.plot_knn_classification(n_neighbors=3)

x_train,x_test,y_train,y_test=train_test_split(cancer.data,cancer.target,stratify=cancer.target,random_state=42)
log_reg100=LogisticRegression(C=100)
log_reg100.fit(x_train,y_train)
print("Accuracy on training set {}",format(log_reg.score(x_train,y_train)))
print("Accuracy on training set {}",format(log_reg.score(x_test,y_test)))


texts: I read the wavenet paper and I am clear about why they used Dilated Convolution. According to it they used this for give the causal property for the model. Where they wanted to introduced a sequential behavior . But lot of posts say it can use to increase the receptive field. Can some give me simple examples where we can use this in signal processing domain CNN + LSTM) and vision domain.


texts: Application of Machine learning or Neural Networks for automatic Time table scheduling

texts:
I've been trying to come up with an intelligent solution to build a Time table scheduling application with the use of Machine learning or Neural networks. What would be the algorithm or approach to build such application. I'm planing to take data from google calendar API and through the system. The system should purpose best time slots to conduct lectures considering the data taken from lecturers and students.

I found some research documents where they have used some genetic algorithms but can this be done with the help of neural networks?


texts: i create model using sklearn library and i want to run this model in JavaEE application i have been trying Jython, but it's impossible to import some important library like pandas and numpy, so how I can do to call a python script for JavaEE application.


texts: Algorithms inherently supporting multilabel classification

texts:
In the documentation of sklearn, it says that several algorithms inherentrly support multilabel classification, such as RandomForest or MLP : https://scikit-learn.org/stable/modules/multiclass.html

Does it come from the implementation or the structure of the models? Moreover, how does it works for these algorithm? Is is a One-vs-the-Rest strategy or something else?

Thanks.


texts: Why does Transfer Learning works better on smaller datasets than on larger ones?

texts:
This question is not about the utility of Tranfer Learning compared with regular supervised learning.
1. Context
I'm studying Health-Monitoring techniques, and I practice on the C-MAPSS dataset. The goal is to predict the Remaining Useful Life (RUL) of an engine given sensor measurements series. In health-monitoring, a major issue is the low amount of failure examples (one can't afford to perform thousands of run-to-failure tests on aircraft engines). This is why Transfer Learning has been studied to solve this, in Transfer Learning with Deep Recurrent Neural Networks for Remaining Useful Life Estimation, Zhang et al, 2018. My question is about the results presented in this article.
2. Question
C-MAPSS dataset is composed of 4 subdatasets, each of which has different operational modes and failure modes. The article cited above performs transfer learning between these subdatasets. Particularly, when training a model on a target subdataset B using the weights of a trained model on a source dataset A, they don't train on all the B dataset. They conduct an experiment in which they test various sizes for the target dataset B : they try on 5%, 10%, ..., 50% of the total dataset B.
The results are presented in page 11. A few cases excepted, the have better results on smaller target datasets. This seems counter intuitive to me : how could the model learn better on less examples ?
Why does Transfer Learning works better on smaller datasets than on larger ones ?


texts:
I am trying to use the first individual BertSelfAttention layer for the BERT-base model, but the model I am loading from torch.hub seems to be different then the one used in hugginface transformers.models.bert.modeling_bert:
import torch, transformers

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
torch_model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')

inputs = tokenizer.encode_plus("Hello", "World", return_tensors='pt')

output_embedding = torch_model.embeddings(inputs['input_ids'], inputs['token_type_ids'])

output_self_attention = torch_model.encoder.layer[0].attention.self(output_embedding)[0]

# compare output with using the huggingface model directly
bert_self_attn = transformers.models.bert.modeling_bert.BertSelfAttention(torch_model.config)

# transfer all parameters
bert_self_attn.load_state_dict(torch_model.encoder.layer[0].attention.self.state_dict())
# <All keys matched successfully>

output_self_attention2 = bert_self_attn(output_embedding)[0]
output_self_attention != output_self_attention2 # tensors are not equal?

Why is output_self_attention2 different from output_self_attention? I thought they would give the same output given the same input.


texts: Replace all numeric values in a pyspark dataframe by a constant value

texts:
Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the dataframe by a constant numeric value (for example by the value 1)? Thanks in advance!

Example for the pyspark dataframe:
$$
\begin{array}{c|lcr}
& \text{c1} & \text{c2} & \text{c3} \\
\hline
1 & 0.04 & 1 & 1.35 \\
2 & -1 & null & -1.2 \\
3 & null & 1.2 & null
\end{array}
$$

The result should be:

$$
\begin{array}{c|lcr}
& \text{c1} & \text{c2} & \text{c3} \\
\hline
1 & 1 & 1 & 1 \\
2 & 1 & null & 1 \\
3 & null & 1 & null
\end{array}
$$


texts: Can we use ReLU activation function as the output layer's non-linearity?

texts:
I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value.

Can I use ReLU as the activation of the output layer? I am afraid of trying, since it is generally used in hidden layers as a rectifier. I'm using Keras.


texts: Logistic regression from scratch in Python

texts:
Here is my logisticRegression class I developed to do gradient descent. There is this one line I marked as problematic

import numpy as np
class logisticRegression():
    """Logistic Regression classifier

    Parameters
    ----------
    alpha : float
        Learning rate(between 0.0 and 1.0).
    iters : int
        Number of iterations.

    Attributes
    ----------
    w_ : 1d-array
        Weights after fitting.
    """
    def __init__(self, alpha = 0.001, iters = 100000):
        self.alpha = alpha
        self.iters = iters

    def fit(self, X, y):
        """Fit training data
        Parameters
        ----------
        X : array-like, shape = [n_samples, n_features]
            Training vectors
        y : array-like, shape = [n_samples]
            Target values

        Returns
        -------
        self : object"""
        # Initialize weight
        self.w_ = np.zeros(1 + X.shape[1])

        self.errors_ = []
        m = X.shape[0]
        x0 = np.ones(X.shape[0])

        for _ in range(self.iters):
            h = self.hyp(X)
            gradient = (X.T)@(h - y)/m
            self.w_[1:] -= self.alpha*gradient
            self.w_[0] -= self.alpha*x0@(h - y)/m # This line is problematic !!!
        return self

    def sigmoid(self, z):
        """Compute sigmoid"""
        return 1/(1+ np.exp(-z))

    def hyp(self, X):
        """Compute hypothesis (probability)"""
        return self.sigmoid(self.w_[0] + X@self.w_[1:] )

I got wrong result. But if I rewrite this line as:

self.w_[0] -= x0@(h - y)/m # Remove the learning rate term

Then I got correct result. But this doesn't seem right. Did I oversee something here?


texts: is there any package in python similar to tidyrin R? Except Pandas

texts: is there any package in python similar to tidyr in R?  Except Pandas


texts: How to fix setting an array element with a sequence error?

texts:
I am trying to make a keras functional model that solves Sentiment Analysis through the amazon database. I am trying to split the input data into 4 different tensors that can be individual feed in to yield a concatenated output that gets processed (there is a reason behind it). However, when training it, it spits out the error of ValueError: setting an array element with a sequence and directs me to the model.fit() line which I could not find an issue with, so I assume it is the inputs to the network.

Code:
https://pastebin.com/Fc33jEbv

Thank you for your time.


texts:
I have a bunch of users, each of them with about 100 features.
My goal is to create an embedded space to compute the "distance" between users.
Also, I want to be able to visualize it with Tensorboard (look this example with Word2Vec http://projector.tensorflow.org/)

However, I only find example and articles about NLP, whilst my I am not dealing with sparse matrix or "vocabulary sizes".

Since I am new to this problem, could you suggest some approaches and some papers/articles/tutorial I can start with?


texts: Test Loss plateau fast in Convolutional Neural Net

texts:
I have a 10k dataset of 1 channel 100X100pixels images with 31 classes.
I set up a CNN with 3 convolution layers each followed by a batchnorm and a 2d pooling. I tried out several combinations of learning step size and batch size, but my loss on the test set (crossentropy) always plateau at the same level.

Here's my model:

class myCNN(nn.Module):

    def __init__(self):
        super().__init__()
        # conv2d( 1-in channels, 2-out channels, 3-kernel size, padding=0, bias=True)

        self.conv1 = nn.Conv2d(1, 32, 5, padding=2)
        self.bn1 = nn.BatchNorm2d(32)
        self.conv2 = nn.Conv2d(32, 64, 5, padding=2)
        self.bn2 = nn.BatchNorm2d(64)
        self.conv3 = nn.Conv2d(64, 128, 5, padding=2)
        self.bn3 = nn.BatchNorm2d(128)
        self.fc = nn.Linear(128*5*5, 31)

    def forward(self, x_in):

        # x is [bs, 1, 100, 100]
        x = xin + F.relu(self.bn1(self.conv1(x_in)))
        x = F.max_pool2d(x, 2)
        # x is [bs, 32, 50, 50]
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.max_pool2d(x, 5)
        # x is [bs, 32, 10, 10]
        x = F.relu(self.bn3(self.conv3(x)))
        x = F.max_pool2d(x, 2)
        # x is [bs, 64, 5, 5]
        x = x.view(-1, 128*5*5 ) # flatten
        x = F.relu(self.fc(x))
        return x

I'm just looking for general lines to take when this happens: more layers, less layers, transform.normalize on the data_loader, no padding, etc? Or maybe just increase the epochs ?


texts: What are the pros and cons of Keras and TFLearn?

texts: What are the pros and cons of Keras and TFlearn? When is one library preferred over the other?


texts: Tensorflow hub module taking long time for embedding single sentence

texts:
I am using universal sentence encoder from tensorflow hub to encode sentence into embedding.

import tensorflow_hub as hub
import tensorflow as tf

module_url = "https://tfhub.dev/google/universal-sentence-encoder/1?tf-hub-format=compressed"

def get_emb(text):
    embed = hub.Module(module_url)
    similarity_input_placeholder = tf.placeholder(tf.string, shape=(None))
    similarity_message_encodings = embed(similarity_input_placeholder)
    with tf.Session() as session:
            session.run(tf.global_variables_initializer())
            session.run(tf.tables_initializer())
            message_embeddings_ = session.run(similarity_message_encodings, feed_dict={similarity_input_placeholder: text})
    tf.reset_default_graph()
    return message_embeddings_

Now the problem is each time I call the function the time is taking is about 10 seconds which is very high. Is there a way to save the model or something and do it instantly. This will be used in restful service where the response should be near instant


texts: Is there a difference between correlation rules and sequential patterns?

texts: In security I read correlation rules in SIEM systems while in data mining I read sequential patterns. They appear to be something similar, because in both cases the order of itemsets matter. Is there a difference between them?


texts: Correlation between specific columns of a data set

texts:
I have a CSV file which has 150 columns belonging to 7 categories but I want a correlation between 2 categories. The categories are movies and music, 12 and 19 columns respectively.

Is there a way to plot a correlation matrix or a correlation graph between two of the categories and selected columns?

For example, 19 columns on x and 12 columns on y. Or summing 12 and 19 columns and having a correlation between only 31 columns instead of 150.

I'm using Python. Which packages could help me?


texts: SVM is margin determined by nearest datapoint or nearest datapoints?

texts:
I am studying support vector machines and different resources seem to define the margin differently. Some define the margin as 2 times the distance to the nearest datapoint.
Others define the margin based on the nearest datapoint in each direction. Why is there there a difference?
Here is my guess:
Case 1:
If we constrain the hyperplane so that it fulfills $y^{i}(w \cdot x + b) \geq 1$, then the hyperplane can be any hyperplane that separates the data. The distance between the closest positive datapoint and the closest negative datapoint do not have to be the same. Hence we find the closest in any direction and calculate the margin as twice the distance to this point.
Case 2:
If we constrain the hyperplane so that it fulfills $y^{i}(w \cdot x + b) \geq 1$ for all datapoints and $y^{i}(w \cdot x + b = 1)$ for the support vectors (nearest datapoints) then the hyperplane must separate all the data and the nearest datapoints on each side must have the same distance. So the margin is now the sum of the distance to the nearest positive data point and the distance to the nearest negative data point.
Does the difference in margin definitions come from the additional constraint that the nearest datapoints lie on the hyperplane $y^{i}(w \cdot x + b = 1)$?
Additionally is there a difference in the optimization procedure if we choose one margin definition over the other? For example if we define our margin as in case 1, will we have to search for a new nearest datapoint after every gradient descent iteration?