texts:
I have ~7 million rows of customer data (~500 sparse attributes)
A million out of them have opted in to a new service.
How do I use this signal to predict which of the remaining customers are likely to adopt the service? And how do I measure the effectiveness?
Problems face so far -

Unable to treat this as a supervised problem due to lack of definitely negative variable
Unable to apply label propagation because there is only one class

Apart from treating this as an anomaly detection problem (oneclasssvm etc.), I also tried using nearest neighbors based approach.
Looking for other ways to solve the problem if there are some go-to techniques that I am missing.
I know there is an answer here but it only talks about oneclasssvm that I have already tried. Also trying to find ways to measure model effectiveness along with any novel ways to solve.


texts: Confusion matrix and ROC AUC curves are not in sync

texts:
I created a classification model with three target classes and created a confusion matrix to measure the accuracy, here is the matrix code

from sklearn.datasets import load_wine
data = load_wine()
x = pd.DataFrame(data=data.data, columns=data.feature_names )
y = pd.DataFrame(data=data.target, columns=['target'])
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.2,shuffle=True,random_state = 42)
model = DecisionTreeClassifier()

model.fit(x_train, y_train)

y_pred = model.predict(x_test)

mat=confusion_matrix(y_test, y_pred, labels=[0,1,2])

The output of the above matrix is

array([[13,  1,  0],
       [ 0, 14,  0],
       [ 1,  0,  7]])

So, this is a balanced dataset and giving me an accuracy of 94% almost.

The problem is, when I tried to draw ROC AUC curve for class 0 using the below code, the AUC curve is the opposite and I am getting only 0.05 area under the curve.

fpr,tpr, thres =  roc_curve(y_test, y_pred, pos_label=0)
roc_auc = auc(fpr,tpr)

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

I tried to calculate and draw ROC AUC manually and I got TPR as 0.9285 and FPR as 0.04545 and the graph is perfect on the sheet, could you please help me why the above code is giving the graph the other way. I verified the above code for the other two classes and the graph is good. Thanks in advance.


texts: Reduce serving time complexity for real-time recommender systems

texts: I am working on a real-time recommender system predicting a product to a user using deep learning techniques (like wide & deep learning, deep & cross-network etc). Product catalogue can be huge (1000s to 1 million) and for a given user, the model needs to be evaluated against each product in real-time. As scalability is an important concern, is there any way to reduce the serving time complexity by tuning model architecture?


texts: Operands Could not be Broadcast with Shapes (19,)(0,)

texts:
I have googled and read something similar to the problem I have but I do not seem to know how to fix the error I got from this particular code:

import operator
def getNeighbors(movieID, K):
    distances = []
    for movie in movieDict:
        if (movie != movieID):
            dist = ComputeDistance(movieDict[movieID], movieDict[movie])
            distances.append((movie, dist))
    distances.sort(key=operator.itemgetter(1))
    neighbors = []
    for x in range(K):
        neighbors.append(distance[x][0])
    return neighbors

K = 10
avgRating = 0
neighbors = getNeighbors(1, K)

**ValueError:** operands could not be broadcast together with shapes (19,)(0,)

The full code:

import numpy as np
import pandas as pd
r_cols = ['user_id', 'movie_id', 'rating']
ratings = pd.read_csv('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.data', sep='\t', engine='python', names=r_cols, usecols=range(3))  # please enter your file path here. The file is u.data
print(ratings.head())
movieProperties = ratings.groupby('movie_id').agg({'rating': [np.size, np.mean]})
print(movieProperties.head())

movieNumRatings = pd.DataFrame(movieProperties['rating']['size'])
movieNormalizedNumRatings = movieNumRatings.apply(lambda x: (x - np.min(x)) / (np.max(x) - np.min(x)))
print(movieNormalizedNumRatings.head())

movieDict = {}
with open('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.item') as f:     # The file is u.item
    temp = ''
    for line in f:
        fields = line.rstrip('\n').split('|')
        movieID = int(fields[0])
        name = fields[1]
        genres = fields[5:25]
        genres = map(int, genres)
        movieDict[movieID] = (name, genres, movieNormalizedNumRatings.loc[movieID].get('size'), movieProperties.loc[movieID].rating.get('mean'))

print(movieDict[1])

from scipy import spatial
def ComputeDistance(a, b):
    genresA = np.array(list(a[1]))
    genresB = np.array(list(b[1]))
    genreDistance = spatial.distance.cosine(genresA, genresB)
    popularityA = np.array(a[2])
    popularityB = np.array(b[2])
    popularityDistance = abs(popularityA - popularityB)
    return genreDistance + popularityDistance

print(ComputeDistance(movieDict[2], movieDict[4]))

import operator

def getNeighbors(movieID, K):
    distances = []
    for movie in movieDict:
        if (movie != movieID):
            dist = ComputeDistance(movieDict[movieID], movieDict[movie])
            distances.append((movie, dist))
    distances.sort(key=operator.itemgetter(1))
    neighbors = []
    for x in range(K):
        neighbors.append(distance[x][0])
    return neighbors

K = 10
avgRating = 0

neighbors = getNeighbors(1, K)
```


texts: How does one decide on the right image quality filters for a training and testing set?

texts:
What image quality metrics do people use to choose appropriate quality (e.g. focus, orientation, obstruction, etc.) images for important for image classification models?
I would imagine anything that is too low quality will be useless or even lead to overfitting noise, too high in quality could hamper the model's ability to generalize when faced with real world images.
Can someone point me to some papers about this topic? Most quality papers seem to deal with automated measures of image quality rather than its role in data cleaning and robustness.


texts: Is it possible to use Recurrent NN (LSTM) for classification?

texts:
I have a dataset C of 50,000 (binary) samples each of 128 features. The class label is also binary either 1 or -1. For instance, a sample would look like this [1,0,0,0,1,0, .... , 0,1] [-1]. My goal is to classify the samples based on the binary classes( i.e., 1 or -1). I thought to try using Recurrent NN (LSTM) to generate a good model for classification. For that, my question is: Is it possible to use Recurrent NN (LSTM) for such a classification problem?

tr_C, ts_C, tr_r, ts_r = train_test_split(C, r, train_size=.8)
batch_size = 1000

print('>>> Build STATEFUL model... (%s, %s)' % (tr_C.shape[1], tr_C.shape[2]))
model = Sequential()
model.add(LSTM(129, batch_input_shape=(batch_size, C.shape[1], C.shape[2]), return_sequences=False, stateful=True))
model.add(Dense(1, activation='relu'))

print('>>> Training...')
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(tr_C, tr_r,
          batch_size=batch_size, epochs=1, shuffle=True,
          validation_data=(ts_C, ts_r))

Thank you


texts: How can I minimize features of the trainded model?

texts:
I have real technological process, that explained with complex model (xgboost). I.e. current mass of a product (y) depends on current temperature (x1), pressure (x2) and so on. I would like to solve optimization task: which minimal values of the features can be selected, that mass of a product can reach the maximum?
It looks like simple optimization task: ||y-y0||^2 where y - equation of the model process and y0 - maximum or some of the closest to maximum values. But it is impossible to get weighted coefficients of the xgboost, so I can`t use skopt and even if I can get the coefficients, the real equation will be very difficult. Only decision that I have right now is sort out all possible values for all possible features, make predictions for this features and choose optimal, if y will reach maximum or close to it.
Could you give an advice, how can I solve this problem?


texts: How to escape 'na' and '-' value in a dataset (Can only use Numpy and Matplotlib for this project)

texts:
I have imported csv file and this is the link to get the data set CSV file https://data.gov.sg/dataset/median-rent-by-town-and-flat-type
My problem is how to escape the value of "na" and "-" in the dataset so that I can calculate the average value of Median Rent according to the town.
data=np.genfromtxt(fname,skip_header=1,dtype=[('quarter','U10'),('town','U10'),('flat_type','U10'),('median_rent','int64')],delimiter=',',missing_values=['na','-'],filling_values=[0])

the above code seems never replace with value 0, but even replaced, it is not correct to calculate the average.
So, I wrote the following, but I don't know how to extract the value of median_rent by using numpy only.
null_rows=np.isnan(data['median_rent'])
nonnull_values=data[null_rows==False]

Could someone please help urgently?
Thanks.


texts: What is the input space of a neural network (or other supervised learning algorithms)?

texts:
While training the neural network (or any other supervised learning algorithms), we supply input variables and corresponding outputs. The input variables can be continuous or discrete (binary in many cases).

What happens if after training with a binary input data, we supply a continuous value for the same input at the time of evaluation? Does the algorithm internally treat all variables as continuous variables?

For example, suppose one of the inputs is Young/Old encoded in the form of 0/1 in the training dataset. What happens if we supply a value of, say, 0.2 at the prediction stage? Does it/should it make any sense to the network?


texts: How Orange makes decision about declaration of a meta attribute

texts:
I wonder how Orange decides which attribute is declared as a string? I have few discrete attributes in Excel, which I can't use as Orange does not recognize them as discrete. Have tried export excel to csv format as well, no difference. Where in Orange I can enforce change from meta to discrete attribute?

Many thanks in advance,
Marko


texts: Scaling monotonically increasing features between 0 and 1

texts:
To keep the test set blind to the neural network algorithm it is generally better to build a scaler based on the training set and then scale the test set on this scaler.

I am building an LSTM for predicting a somewhat monotonically increasing price chart. I am not shuffling the data before splitting into training and testing since the sequence is important.

This will mean that the test set can have values higher than 1 (say 1.2, 1.01 etc...)

Would this be a problem to the neural network? Is it important that every input to the neural network be less than zero if that is what it was trained with?


texts: Are Denoising Variational Autoencoders deterministic?

texts:
I have a pretty good understanding of regular autoencoders and, to a certain extent, of variational autoencoders, where the latent representation is forced to follow specific probabilistic distributions. From what I understand variational autoencoders are used as generative models by randomly sampling the learned latent space distributions to obtain a new output.

But in the case of denoising variational autoencoders, I guess that they are being fed with a noisy input whose missing parts will be generated on the output layer. In this case, is there a sampling operation during inference ? Is this generative process deterministic ? If I try to denoise twice a noisy input, will I get the same output ? Or will I get different good denoised possibilities according to the learned distributions ?

Thanks


texts: Concatenation of CNN and LSTM to model time of a series of images

texts:
I have collected a dataset consisting of around 30'000 heat maps of 80 users. The heat maps represent typing behavior on a keyboard and are just images with a resolution of 39 x 39 x 3 (3 color channels). Each user had to fill-in self-reports about the actual pleasure (emotion). This is the ground truth. For each user I have between 30 and 700 such labels. Each label (self-report) has an associated heat map (which I have created over 30 minutes before the self-report).
I have now built a CNN which I have used to predcit the ground truth (emotion) based on the heat maps. This works well but does not incorporate the time series. I see two possibilities to incorporate time:

Instead of creating one heat map over 30 minutes, I could create 3 heat maps where each heat maps takes data from 10 minutes. These 3 heat maps could then be stacked (39 x 39 x 3 x 3) and used in a CNN with 3D convolutions.

The time series of self-reports could also be exploited by a combination of CNN and LSTM. I could process each heat map by a CNN and feed the flattened output then into a LSTM to predict the self-reports of each user. Here, also ConvLSTMs could be used but this has the disadvantage that I cannot incorporate additional context data (such as the daytime). When concatenating a CNN and a LSTM, I can append context data to the flattened output of the CNN.

Are there any other (better) possibilities to exploit time?
Regarding the second option, I also see the problem that the series of self-reports ranges from 30 to 700 for the users. How can I handle this in a LSTM? Should I just pad with zeros if a sequence is smaller than 700? But this would result in a lot of zero padding if a user only had 30 self-reports...
Last but not least, the timings of the self-reports are not equally spaced. How can I incorporate the time information (either absolute time or relative time to previous self-report) in such a model? As mentioned above, I think one possibility is to append this information to the flattened output of the CNN but I'm not sure if that is the best way to go.


texts: Running Neural Network experiments in loop

texts:
I need to run neural network training with different hyper parameters settings like this:

for seed in range(2):
        NN_inst = FeedForwardNN(seed)
        NN_inst.load_data()
        NN_inst.classification_model()
        NN_inst.run_model()

First run everything works. On the second run I get tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value training/Adam/Variable_39
     [[{{node training/Adam/Variable_39/read}} = IdentityT=DT_FLOAT, _class=["loc:@training/Adam/Variable_39"], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

It looks like I need to clear tensorflow memory, but

K.clear_session()
tf.reset_default_graph()

do not help. I am using Keras, error is on fit().
What should I do?


texts: Custom Decision Function for Custom Outlier Detection Algorithm

texts:
I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier:

X1  X2  X3  Probability Threshold
1   2   3   0.01         1
4   5   6   0.015        1
7   8   9   0.04         1
10  11  12  0.06         0
13  14  15  0.1          0

Now using such data, how can I build the decision function/score similar to the one in scikit-learn?

My ultimate objective is to able to produce precision/recall plot, ROC plot for this algorithm and compare its performance to other scikit-learn and PyOD library algorithms.

The only thing that I was able to come up with so far is to loop through different thresholds of probability and at each iteration set accordingly whether a point is outlier or not and compute precision/recall/tpr/fpr metrics. However, I understand that this is not the proper way to compute the decision function and, thus, it cannot be compared to scikit-learn algorithms.


texts: How YOLO training and prediction works for an object fall in multiple grid?

texts:
So far what I understood about YOLO, it expects training image should be divided in to fixed grid, where each grid has Label like P(object present or not), object bounding box, object classes. Similarly it will return the same output for each image prediction.

If it's correct, I'm not able to map those images for both cases training and prediction where some objects  are the part of multiple grids.
During training we provide bounding box information corresponding to particular(single) grid only, how it clubbed the bounding box info of multiple grids?

Note: Non-max suppression is again confusing, if it is related with it.


texts: EfficientNet: Compound scaling method intuition

texts:
I was reading the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks and couldn't get my head around this sentence:

  Intuitively, the compound scaling method makes sense because if the input image is bigger, then the network needs more layers to increase the receptive field and more channels to capture more fine-grained patterns on the bigger image.

In the case of a big image, why the network needs more layers to increase the receptive field ? What does increasing the receptive field mean? Increasing its width/height ? If so, we can do it directly without increasing the number of layer in the network no ?
is "fine-grained patterns" referring to noisy shape we can see after visualizing convolution output ?

I feel like I am missing / misunderstanding something evident.


texts: How to scatter plot with each dimension having its own color

texts:
I want to get insight for some values. I have two vectors of 300 dimensions, and want to compare them coordinate by coordinate. So I thought to plot each point with a different color, but the color for a dimension to be the same so that I would know which coordinates are lying where.

I have the below code, which I modeled on some results at stackoverflow.

colors = itertools.cycle(["r", "g", "b"])

for d_in in inp_samples:
    clr = next(colors)
    plt.scatter(len(inp_samples), d_in, c=clr)

colors = itertools.cycle(["r", "g", "b"])

for d_out in out_samples:
    clr = next(colors)
    plt.scatter(len(out_samples), d_out, c=clr)

But the plot I am getting is very weird. I was expecting it would be scattered but it is something like this:

I also tried this:

colors = cm.jet(np.linspace(0, 1, 300))

for d_in, d_out, c in zip(inp_samples, out_samples, colors):
    plt.scatter(len(inp_samples), d_in, c=c)
    plt.scatter(len(out_samples), d_out, c=c)

Can anyone help in understanding what I am doing wrong?


texts: Label Encoder encoding the same class as two integers

texts:
First I have defined classes of a label encoder with the keys of a dictionary. Then I have used that label encoder to encode some strings. But for the same string, its giving different integers. Why is this happening?

grid_dict = {'8861892541fffff': 588,'886189254dfffff': 65, '8861892543fffff': 103, '8861892547fffff': 83, '8861892545fffff': 401,
 '8861892569fffff': 131, '8861892727fffff': 39, '886189255dfffff': 13, '8861893597fffff': 2, '8861892555fffff': 4, '886189209bfffff': 13, '8861892549fffff': 3}

le = LabelEncoder()
le.classes_ = list(grid_dict.keys())

Now

le.transform(['8861892547fffff']) #4th class

gives me array([1])
but le.transform(['8861892545fffff']) #5th class
also gives me the same array([1]) which should not happen. Also when I do le.inverse_transform(0) its giving '8861892541fffff' which is the second class.
Also when I do le.transform(['8861892549fffff']) #last class its giving array([5]) and for le.transform(['8861892569fffff']) #class6 its giving array([5]).

I am not fitting the label encoder anywhere. I have some stored the keys in a dictionary and declaring the label encoder classes with the dict keys. Why is this happening?


texts: Treatment and Control selection in A/B Testing

texts:
I'm hoping to get a better understanding of A/B Testing design. In particular, I'm interested in understanding how treatment and control units are selected. I read that these 2 groups are selected randomly (for example, here), but then there are also approaches where after picking the treatment (either randomly or not) the control is selected based on "similarity" to the treatment group. Are both approaches valid and what's the rationale for picking one or the other?

For example, Alteryx has specific
Treatment and Control Tools for this purspose, and they are not random (they use nearest neighbor methods).


texts: Which algorithm or tool to use to classify as good or bad?

texts:
I have a feature vector with different data types, Considering all the data in that feature vector. I have to classify as Good or Bad.

Which algorithm should be used to just get the output Good or bad based on different data types in a feature vector?

The feature vectors are as follows:

[Application_Name(string) , Uptime (Integer) , Criticality factor (0-1 float value) and few integer type ]


texts: Is it true more CPU core is better for deep learning?

texts:
I just started to learn the deep learning in my free time. I was hoping to buy a laptop where I want to implement some small(alexnet) to medium(GoogleNet) networks maybe something bigger. I searched for the GPU. Everyone is suggesting minimum RTX 2060. However, it's vague about the CPU. I heard people saying that it's better to have more number of core in CPU. All I understand CPU only do the data pre-processing. What I want to know is it feasible to have a 4 core CPU(e.g. i5 9300H) with RTX 2060 having a tight budget? Will this configuration serve my purpose?
Another things is new 10th Gen Intel Core i7-10750H processor with up to 5.0 GHz3 have a 6 cores. While AMD Ryzen 7 4800HS have 8 cores. Which will be best for deep learning. Is the number of cores really matters?


texts: Use text similarity (cosine) instead of machine learning to classify companies into industries

texts:
I am building an industry classifier. I.e. classifying companies into industries based on a company's description. Each company can only have one industry.

I took 2000 companies and assigned them into 15 industries. Then I trained a few models based on the company's description. All of them performed extremely bad.

So I wanted to get more data (indicated by the learning curves), i.e. more companies. But getting the classification right is difficult. Thus, I decided to try a simple similarity calculation.

For each industry I select words that fit best to this industry. Then I calculate the similarity between each industry and a new company and select the one that fits the best.

For the three industries I tested it worked great. But I am unsure if this will makes sense down the road. I think that the similarity approach is great because:

I can clearly define (from a business perspective) what is what
It scales nicely

Can anybody tell me if this approach makes sense? All other classification system are build using some model, so I am unsure.

EDIT:
I want to build an industry classifier that can classifier "all" existing companies. By all I mean companies that fit the NACE industries (NACE is the industry standard classification system used in the European Union)

The 2000 companies that I used are arbitrary. They do not represent all the industries from NACE. The 15 industries I ended up with are mostly based on NACE. However, I grouped some of them together. Also I made up some industries based on the most frequent keywords across certain companies (sometimes that is very obvious. Example: "dating companies" like Tinder).

The company descriptions come directly from the company (mostly parsed from their website).

I have tried to cluster the companies but I didn't find any combination of clusters that make sense based on NACE.

Yes, I have filtered stopwords (and applied other normalization such as only keeping text characters and stemming).

Regarding objectivity (thanks to @skiddles): I think this is actually another pro-argument for similarity. If I extract all the keywords that are used to describe the industries in the NACE specification I end up (it seems) with at least one recognized standard.

So, I believe, my question should be:
I have 20 industries. Each industry is described with 100 distinct words. I get a new company with a description. The words of this company description can be found in the descriptions for the industries. Should I classify the companies into industries based on similarity or use a classification model?


texts:
Lately I have been working on image classification problems but just recently I tried a new type of Problem on Kaggle.

I have normalized image data(X) by dividing with 255 and also the keypoints coordinates (y) by dividing with 96 as maximum height and width is 96.

So lastly model architecture in keras, I have tried smallest vanilla network spanning to Huge ones similar to VGG models and with epochs it doesn't seem to learn instead underfits or overfits.

And one other problem is dataset size is small by my best efforts I doubled the datset by flipping image and shifting values accordingly.

Even after that it seems to get nowhere.
I think I'm done improving data but can you help me by giving me a model architecture that works.

Link to Kaggle Competition: https://www.kaggle.com/c/facial-keypoints-detection


texts:
I am trying to build a RNN model to classify time series.
My time series data consist of numbers with length 200.

I am using tensorflow, and when i create the placeholder for the data it is like this:

inputs = tf.placeholder(tf.float32, [None, 200, 1], name='inputs')

200 is the sequence length and 1 is the features, which in my case is 1 feature per time step.

And when i create the RNN using GRU cell i use this:

tf.contrib.rnn.GRUCell(400)

I use 400 as cell size of GRU cell because as far as i know the internal state + the inputs will be used as the new input for the next GRU cell, so i suppose the cell must be input length x 2.

Is this logic of mine correct?
Is the initialization of inputs correct?

BATCH SIZE = None

SEQUENCE LENGTH = 200

FEATURES = 1

And is the GRU cell parameters correct?

CELL SIZE = 2 x SEQUENCE LENGTH?

Or have i understood it incorrectly?


texts: Historical exchange rate data via World Bank API?

texts:
I can see a massive list of indicators available via the World Bank API here (may take 1-2 minutes to load)
I found exchange rates for various currencies into USD, but cannot see how to get exchange rates between currencies which aren't the USD.
Is this possible via the World Bank API?


texts: Decision tree classifier prediction changes from one run of the model to the next

texts:
I'm running a very basic gender ['male', 'female'] classifier using the sklearn DecisionTreeClassifier based on [height, weight, shoe size] in a Jupyter notebook.

The prediction changes from male to female for the same input as I keep running the model.

I don't understand how that's possible. Shouldn't the build of the model be completely deterministic and therefore output the same prediction each time for a specific input?

Here's my code:

X = [[190, 90, 43], [165, 65, 38], [170, 70, 39], [160, 56, 36], [190, 88, 45],
[164, 63, 37]]
Y = ['male', 'female', 'male', 'female', 'male', 'female']

clf = tree.DecisionTreeClassifier()
clf.fit(X, Y)

print(clf.predict([[200, 70, 37]]))


texts:
I want to to fine tune the BERT base model but the only accelerated HW I have access to is a couple of Quadro 600 GPUs which only pack 1GB RAM and 96 CUDA cores each.

My question: is it even possible to fine tune BERT over this HW? If so, would it be worth with so few CUDA cores available? How can I know whether GPU memory will fit all the data and parameters?


texts: Compare the variances of two categorical distributions in a repeated measure design

texts:
I ran two model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds in my nested-cross validation). This gives me a contingency table, with the two models as rows and the absolute frequencies for each of the possible hyperparameter categories as columns. For the sake of simplicity, let's call the rows day 1, and day 2, let the columns be different ice cream sorts and let the 100 folds be the same sample of people buying ice cream:

As you can 'see', on day 1 the distribution of my categorical variable is way more spread out than on day 2. On day two, the hundred people only bought vanilla and choco ice cream.

I would like to measure this difference in variance. My first idea was to use a Chi2-Goodness of Fit test, where I test the second row against the first row and treat this row as 'expected frequencies'. But I am not sure, if this is the right approach, and if the Chi2-Test does the right thing here. Can anyone recommend a test that handles this problem?


texts: model.predict() accuracy extremely low on training dataset

texts:
This question is similar to this.
I'm new to ML, and I am trying to classify breast cancer histology images using EfficientNets with Transfer Learning. The dataset is small (400 images in total - there are 4 classes and all classes are equally balanced) and I am using ImageNet weights, and fine-tuning the model by freezing the first two blocks.
I've implemented a model with Keras that reaches a training accuracy of ~90% after 30 epochs.
When trying to use model.predict on the training dataset (to understand the results of the predict), I expect the results to be good since the prediction is being done on data that the model has already seen but the results I get are extremely low. The prediction accuracy in the report created by sklearn.metrics.classification_report is 27.81%.
After trying various methods to change this, I still have not been able to improve or understand the result.
The labels I use:
{0: 'Benign', 1: 'InSitu', 2: 'Invasive', 3: 'Normal'}
After playing around with code, I found that the expected labels and the results were as follows:
expected:
[0, 3, 1, 0, 0, 3, 2, 0, 2, 2, 1, 0, 3, 1, 2, 3, 2, 1, 3, 2, 3, 3, 1, 0, 3, 1, 3, 2, 2, 1, 1, 3, 0, 1, 3, 0, 1, 2, 0, 3, 2, 3, 2, 3, 1, 0, 1, 0, 1, 1, 2, 1, 2, 2, 2, 1, 1, 0, 3, 3, 0, 3, 1, 1, 3, 1, 1, 0, 2, 1, 0, 0, 1, 1, 1, 2, 2, 1, 3, 1, 1, 3, 3, 3, 0, 1, 1, 0, 2, 3, 0, 3, 1, 2, 2, 3, 0, 3, 3, 0, 0, 0, 2, 0, 1, 0, 2, 1, 3, 1, 0, 2, 2, 0, 1, 3, 1, 1, 3, 2, 3, 0, 2, 1, 2, 3, 3, 0, 0, 1, 3, 0, 0, 3, 2, 3, 1, 2, 0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 3, 0, 0, 3, 2, 1, 3, 2, 1, 1, 2, 2, 3, 3, 1, 2, 1, 0, 3, 3, 0, 1, 2, 2, 1, 3, 1, 3, 0, 1, 3, 2, 3, 3, 0, 2, 1, 3, 2, 0, 2, 0, 3, 3, 2, 3, 0, 2, 2, 0, 3, 2, 2, 1, 1, 3, 2, 3, 0, 2, 0, 3, 1, 0, 1, 2, 0, 2, 0, 0, 2, 2, 2, 1, 1, 0, 3, 0, 1, 1, 2, 2, 0, 0, 3, 0, 1, 0, 1, 1, 1, 3, 2, 2, 3, 2, 1, 2, 0, 1, 2, 1, 1, 1, 3, 1, 2, 3, 1, 2, 0, 2, 3, 1, 2, 1, 2, 1, 2, 2, 2, 0, 3, 3, 1, 3, 3, 3, 0, 1, 1, 2, 2, 3, 3, 2, 0, 0, 2, 1, 0, 1, 3, 0, 2, 0, 0, 3, 2, 1, 2, 2, 1, 0, 0, 0, 2, 3, 3, 0, 2, 3, 1, 0, 0, 0, 0, 3, 0, 3, 2, 2]
result:
[3 0 3 0 3 3 3 1 0 2 3 3 2 1 3 2 1 1 3 1 2 3 2 1 0 3 3 0 2 1 2 2 2 1 3 3 0 2 0 2 1 1 2 0 0 0 0 0 2 3 0 1 3 0 1 1 2 0 0 2 3 3 0 2 2 1 2 3 2 2 1 2 1 3 3 3 1 0 1 3 0 1 2 3 2 0 1 3 0 3 3 0 0 3 3 3 1 3 0 2 3 3 3 0 3 0 0 1 3 1 1 3 0 0 0 3 1 1 0 1 3 2 0 3 2 3 0 1 0 2 1 1 2 3 2 2 1 2 1 2 1 1 0 1 1 1 3 2 2 1 3 1 1 0 2 1 1 1 0 1 0 0 0 0 2 2 0 0 3 2 3 3 2 3 3 3 2 1 2 3 2 3 2 2 1 1 0 1 0 2 3 2 1 3 0 1 0 1 2 0 3 1 2 0 1 1 2 0 3 1 3 2 1 2 2 1 3 0 1 2 3 3 1 3 2 1 2 3 0 3 0 2 2 3 0 1 3 1 0 2 2 1 0 2 3 1 0 1 3 3 2 2 0 0 2 1 0 0 3 2 2 3 1 0 2 1 0 0 0 2 0 2 1 3 1 0 3 1 1 1 1 1 2 0 3 3 0 3 0 0 2 0 3 1 1 3 1 0 2 2 3 2 2 2 2 2 1 0 3 3 2 0 0 0 1 2 3 2 1 2]
The code I use to predict:
training_data.reset()
predicts = model.predict(training_data, steps=20, verbose=1)
predict_class = np.argmax(predicts, axis = 1) # could this be the problem?
errors = np.where(predict_class != truth)[0]
print("No of errors = {}/{}".format(len(errors),training_data.samples))

Where training_data is an image generator defined as follows:
training_gen = ImageDataGenerator(
    rescale=1./255,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.15,
    horizontal_flip=True,
    vertical_flip=True,
    preprocessing_function = preprocess_img
    )

training_data = training_gen.flow_from_dataframe(
    dataframe=df_train,
    directory = train_dir,
    x_col="ID",
    y_col="Truth",
    target_size=(height,width),
    batch_size=batch_size,
    # shuffle=True,
    class_mode="categorical",
    seed=seed_augment
)

Am I misunderstanding when I assume that the predict function will give me good accuracy on training data? Does this imply that my model is actually really bad?
Any direction would be appreciated.


texts: I want to do a quick computation in R that involves estimating the ideal gas constant from experimental data.  Ideally I'd have data from a monatomic gas at different pressures, volumes, temperatures and moles (or just pressure and volume, keeping the others fixed).  Anyone have an idea of where I might find a data set like this?  An R data frame would be nice, but I'll take excel, tab delimited, or whatever.


texts: Keras save model FailedPreconditionError

texts:
Model works and fits. After adding model.save('model.h5'). I am receiving
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dense_11/bias[[Node: _retval_dense_11/bias_0_0 = _RetvalT=DT_FLOAT, index=0, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I tried model.save_weights('model.h5')  saves the model but after trying to load the model I get:

KeyError: "Unable to open object (object 'dense_12' doesn't exist)"

After saving model to json and then trying to load it i get json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0).

What should I do?
Thanks for help!


texts: Clamping Q function to it's theoretical maximum, yes or no?

texts:
I'm implementing DQN algorithm from scratch on MountainCar simulation.
I'm using a setup of $reward = 1.0$ when car hits the flag, and $0$ otherwise.
Reward decay factor is set to $\gamma=0.99$. Algorithm starts with exploration factor of $\epsilon = 1.0$ and decreases that over time to $\epsilon = 0.1$.

If I understood correctly, $Q$ function for some state and action pair is defined as:

$Q(s_t, a_t) = r_t + \gamma \times \operatorname{arg\,max}_a Q(s_{t+1}, a_{t+1})$

So, $Q_{max}$ would satisfy the condition:

$Q_{max} = r_{max} + \gamma \times Q_{max}$

Which means:

$Q_{max} = \frac{r_{max}}{1 - \gamma}$

However, since my network only approximates the $Q$ function it is possible for it to sometimes produce value greater than $Q_{max}$. When that happens, further training causes the values to start growing exponentially, and the entire thing blows up.

When I clamp the error of expected value vs. current predicted value to some small number, it still causes the blowup just a bit slower.

The only solution I can think of is to clamp the predicted value $Q(s_{t+1}, a{t+1})$ to $Q_{max}$ and forcing it to never go above that.
I have done that, and got OK results with it.

Does this make sense? Is this a situation that happens in DQN? Or maybe I missed something and my implementation is a bit buggy?


texts: Trying to understand neural network's performance

texts:
I am trying to build a RNN for classification and below is the layout of the network

import tensorflow as tf
model  = tf.keras.Sequential()
model.add(tf.keras.layers.Masking(mask_value=-100.0, input_shape=X.shape[1:]))
model.add(tf.keras.layers.LSTM(units = X.shape[1]))
model.add(tf.keras.layers.Dense(units=10,activation='relu'))
model.add(tf.keras.layers.Dense(units=1,activation=tf.nn.sigmoid))
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy',tf.keras.metrics.AUC(name='auc')])

and adding class weights

from sklearn.utils.class_weight import compute_class_weight
class_weight = compute_class_weight(class_weight='balanced',classes =np.unique(y),y=y)
class_weight = {i : class_weight[i] for i in range(len(class_weight))}

and fitting using

model.fit(X,y,epochs=10,validation_split=0.4,class_weight=class_weight)

My X has structure :

array([[[  0.29615385,   0.9375    ,   0.52268041, ...,   0.        ,
           0.13050997,   0.15842043],
        [  0.29615385,   0.9375    ,   0.52268041, ...,   0.        ,
           0.13050997,   0.15842043],
        [  0.26538462,   0.9875    ,   0.52268041, ...,   0.        ,
           0.09115686,   0.12374666],
        ...,
        [-100.        , -100.        , -100.        , ..., -100.        ,
         -100.        , -100.        ],
        [-100.        , -100.        , -100.        , ..., -100.        ,
         -100.        , -100.        ],
        [-100.        , -100.        , -100.        , ..., -100.        ,
         -100.        , -100.        ]],

           :
           :
           :

The -100 is the padding done to make those inner 2d arrays of same shape

However the model is not learning as I expect it to be i.e increasing accuracy

The model's accuracy does not increase more than this, it decreases on more epochs.
Is there anything wrong with my Neural Network structure or I need to experiment with more number of hidden layers (which I already tried)

757/757 [=============>] - ETA: 10:08 - loss: 0.6800 - accuracy: 0.5290 - auc: 0.6303


texts:
Is it possible to train a GAN model to inpaint an image taken from a specific setting(e.g. an office, woods, beach ...) after we have cropped out people of it?

For example, I used this repo's pretrained GAN model on Places2

I used a MaskRCNN model to segment people, and then cropped them out :

But when I fed the image to the mentioned GAN model, results weren't so great :

If it's possible, what is the best GAN architecture to use?


texts: Sample Importance (Training Weights) in Keras

texts:
How do you add more importance to some samples than others (sample weights) in Keras?

I'm not looking for class_weightwhich is a fix for unbalanced datasets.

What I currently have is:

trainingWeights which is the desired importance I want to give to each sample.

epochs = 30
batchSize = 512

# Fit model with selected data
model.fit(trainingMatrix, trainingTargets,
          batch_size=batchSize, epochs=epochs,
          sample_weight=trainingWeights)

However the training error is much lower than before, and according to Keras' documentation:

  sample_weight: Optional Numpy array of weights for the training samples, used for weighting the loss function (during training only).

As I understand it, this option only calculates the loss function differently without training the model with weights (sample importance) so how do I train a Keras model with different importance (weights) for different samples.

PD. This is a similar question xgboost: give more importance to recent samples but I would like an applicable answer to Keras.


texts: Random Forest application with 40+ Predictor Variables

texts:
I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals based on historical data. The final variable that is being predicted is a categorical "1, 2, 3, 4, or 5" variable.

I need to incorporate 30-40 variables--their importance will also be analyzed-- to reach a decision for each individual.

I have no problem training a model, initially. Below is a sample data set being trained with R's randomForest package. I am running up-to-date versions of both the package and RStudio.

set.seed(101)
train <- sample(1:nrow(Boston), 300)
Boston.rf <- randomForest(medv~., data = Boston, subset = train)
Boston.rf

My question is: Once I have a model trained, tested, and cross-validated, how do I actually apply new data to this model? Is it viable to use that many variables--all categorical, some binary--for this?


texts: How can I calculate a rolling window sum in pandas across this MultiIndex dataframe?

texts:
I can't work out how to get the moving annual sum from this data:

>                          revenue
> txdate   2014-01-31     2014-02-28      2014-03-31     2014-04-30     ....
> user_id
> 1            0              10             165             0
> 2          265             265             200           250
> 3          770             985            1235           900
> ....

Previously I would have tried something like this and adjusted until it worked:

df.groupby(level='practice_id').apply(lambda x: pd.rolling_sum(x, 12))

but it's deprecated and I'm not getting my head around the 0.18 changes to rolling despite reading the docs, and I'm not sure that the shape of the data is helpful (it's close to what needs to be inserted in a db table). The original data format is as follows:

> txdate          user_id        tx_amount
> 2014-01-01         2               5
> 2014-01-02         2               5
> 2014-01-02         3              30
> 2014-01-03         3              15
> 2014-01-02         2              10

I reshaped with the following cmd:

> df.set_index('txdate').groupby('user_id').resample('M').agg({'revenue': np.sum})

I'm thinking I might need to reverse the order of operations.


texts: How to scale or standardize data that is mostly 0 (ranges from 0-1)?

texts:
I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each column are $0$. The data represents certain attributes about a customer, including the proportion of a certain customer's purchases in a particular category (so that the sum of each of these proportions is $1$), in addition to other data like the number of visits, the time between visits, etc. A set of sample rows could look like:

columns = [other_stuff,'C1','C2','C3','C4','C5','C6']
row = [1945,0.45, 0, 0, 0, 0.3, 0.25]
another_row = [438,0, 0.24, 0, 0.01, 0.5, 0.25]

A sample histogram of one of the "proportional" variables looks like:

As of now, I am struggling to find ways to scale the data to make it usable for clustering with other data with much different units/orders of magnitude. Should I:

Scale all the other variables to a 0-1 range using a Min-Max method and keep the "proportional" variables the same
Scale all variables using the variables' means and standard deviations, even though the "proportional variables" clearly do not follow a normal distribution (also cannot apply a log transformation since most of the values are indeed 0)
Keep everything as is, but perform dimensionality reduction on all variables and perform clustering based off the principal components (if Principal Component Analysis is used)
None of the above; use a completely different set of algorithms/methods

I am currently using option 3 using sckit-learn and Python 3.7. If there are packages in R that could also help, please throw them my way. Thank you.


texts: Is it possible to get worse model after optimization?

texts:
I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.
in order to optimize my model I define param grid and than fit with the train data and then according to the results run again with nre parameters, e.g-
#ROUND 1
param_grid={
    'max_depth': [3,4,5],
    'learning_rate':[0.1,0.01,0.05],
    'gamma': [0,0.25,1.0],
    'reg_lambda':[0,1.0,10.0],
    'scale_pos_weight':[1,3,5]
}

grid_search = GridSearchCV(estimator = clf_xgb, param_grid = param_grid,
                          cv = 3, n_jobs = -1, verbose = 2)
grid_search.fit(X_train,y_train)
grid_search.best_params_

>>>.....

(and now based on the result changing the params...)
after this step I choose the best hyperparameters and run the model;
clf_xgb=xgb.XGBClassifier(seed=42,
                         objective='binary:logistic',
                         gamma=0,
                         learn_rate=0.7,
                         max_depth=6,
                         reg_lambda=0.8,
                         scale_pos_weight=1,
                         subsample=0.9,
                         cilsample_bytree=0.5)

clf_xgb.fit(X_train,
           y_train,
           verbose=True,
           early_stopping_rounds=10,
           eval_metric='aucpr',
           eval_set=[(X_test,y_test)])

The problem is that when I check the model score
clf_xgb.score(X_test,y_test)

I always get lower score than what I got before the optimization which makes me suspect that I'm missing something in the way doing it/basic principle in this process.
Is it possible that after running the optimization my score won't get better (and even worse?) ? Where is my mistake? Are there other parameters that could influence or improve my model?


texts:
I am rather new to this and can't say I have a complete understanding of the theoretical concepts behind this. I am trying to calculate the KL Divergence between several lists of points in Python. I am using this to try and do this. The problem that I'm running into is that the value returned is the same for any 2 lists of numbers (its 1.3862943611198906). I have a feeling that I'm making some sort of theoretical mistake here but can't spot it.
values1 = [1.346112,1.337432,1.246655]
values2 = [1.033836,1.082015,1.117323]
metrics.mutual_info_score(values1,values2)

That is an example of what I'm running - just that I'm getting the same output for any 2 input. Any advice/help would be appreciated!


texts: what are the top level subsets/domains of ML?

texts:
I'm not really happy with the mind maps I've been able to find on Google, most of them are algorithm based.  I want to make a good one that is problem/solution domain based.  Do I have this right for my top level nodes?  Here is the general direction I am headed: https://imgur.com/gallery/CugcS

My questions/doubts about what I have so far are:

Is my starting point below generally correct?  e.g. no high level subclass is missing, and everything presented as a subclass deserves to be here?
is Hybrid learning always just a combination of supervised and unsupervised?  Or, are there real examples of other hybrid models (e.g. 'reinforcement' and 'supervised', etc.).  I know theoretically we can combine any methods...I'm looking for what's real/applied/demonstrable today.
does Reinforcement learning belong at this high level, or is it actually a subset of one of the others (or one I've omitted)?

Machine Learning

1.1 Supervised (uses labelled data to train and validate)

1.2 Unsupervised (uses unlabeled data, or ignores labels if they are present)

1.3 Semi-supervised (uses partially labelled (mostly unlabeled) data)

1.4 Hybrid (combines a supervised method and an unsupervised method)

1.5 Reinforcement Learning (uses data from the environment)

Thank you!


texts:
I am currently testing some approaches for density estimation, and I think the basic approach of histograms may not be the best option to me and KDE is certainly a good alternative to go. While ago I found a very interesting tutorial by Jake VanderPlas which explains KDE in a nice way.
In his tutorial, Jake optimized KDE bandwidth selection using grid search maximizing the log-likelihood of the density estimation given some samples, but that is built-in in sklearn and it's kind of hard to know how to compute the same metric for the histogram because I am not sure the formula they use (I search in the docs).
My question is, do you know a simple and usual way of comparing two density estimations from a set of points for example in 2D? My purpose is to evaluate how good is a particular KDE against a histogram.


texts: What's The Difference Between The Terms Predictor And Feature

texts:
For the term 'predictor', I found the following definition:

  Predictor Variable: One or more variables that are used to
  determine or predict the target variable.

Whereas Wikipedia contains the following definition of the word 'feature':

  Feature is an individual measurable property or characteristic of a
  phenomenon being observed.

What is the difference between 'predictor' and 'feature' in machine learning?


texts: I'm going to start my degree thesis and I want to do a fault detector system using machine learning techniques. I need datasets for my thesis but I don't know where I can get that data. I'm looking for historical operation/maintenance/fault datasets of any kind of machine in the oil & gas industry (drills, steam injectors etc) or electrical companies (transformators, generators etc).


texts: Seaborn Pairplot not showin colors of the labels

texts:
My dataframe looks similar to this

I am using the following line to graph the pairplot

sns.pairplot(iris, hue = 'States', height=7, aspect=0.8 )

The graph is good but I am confused why do the labels on the side have no color corresponding to the graph?


texts: How to perform Permutation Feature importance?

texts:
I am trying to perform feature selection. Currently with Tree based classifiers, even randomly generated column is ranking above some of my real columns. So I was reading about PFI. Can someone help me understand the steps to be followed?

1) Have my data in csv file

2) Fill in missing values/clean/prepare data

3) Split the data into train and test

4) Standardize train and test data

5) Select a model (let's say logistic regression)

6) Fit train data to logreg model

7) Predict using input test data

I have given the code below for my model

logreg=LogisticRegression() # step 5
logreg_cv.fit(X_train_std,y_train) # step 6
y_pred = logreg_cv.predict(X_test_std) # step 7

Would really be helpful if you could kind of explain in a way a noob can understand. I am new to ML, so it would really be helpful

Where does PFI fit in here? I see that we have eli5 package but couldn't find an example yet.

update - my code after suggested solution

logreg=LogisticRegression()
pi=PermutationImportance(logreg, cv=2).fit(X_train_std,y_train) # error is in this line.
rfe=RFECV(pi, step=1, n_jobs=-1, cv=5, scoring="auc").fit(X_train_std,y)
y_pred = rfe.predict(X_test_std)


texts: Approximating t-SNE embeddings for out-of-sample data

texts:
I have a large amount of data which has been reduced to two dimensions using t-SNE. Additional data points keep arriving, which I would like two-dimensional embeddings for, but this cannot be achieved without re-running the t-SNE algorithm which is not feasible for every new data point.

Are there any ways to approximate the t-SNE embedding for new data points? I've tried taking $k$-nearest neighbours in the full data space, getting their corresponding t-SNE embeddings, and averaging them to produce an embedding for the new data point. This works okay, but I'm wondering if anyone knows more rigorous methods to approximate this.


texts: Encoder-Decoder LSTM for Trajectory Prediction

texts:
I need to use encoder-decoder structure to predict 2D trajectories. As almost all available tutorials are related to NLP -with sparse vectors-, I couldn't be sure about how to adapt the solutions to a continuous data.
In addition to my ignorance in seqence-to-sequence models, embedding process for words confused me more. I have a dataset that consists of 3,000,000 samples each having x-y coordinates (-1, 1) with 125 observations, which means the shape of each sample is (125, 2). I thought I could think of this as 125 words with 2 dimensional already embedded words, but the encoder and the decoder in this Keras Tutorial expect 3D arrays as (num_pairs, max_english_sentence_length, num_english_characters).
I doubt I need to train each sample (125, 2) separately with this model, as the way Google's search bar does with only one word written.
As far as I understood, an encoder is many-to-one type model and a decoder is one-to-many type model. I need to get a memory state c and a hiddenstate h as vectors(?). Then I should use those vectors as input to decoder and extract predictions in the shape of (x,y) as many as I determine as encoder output.
I'd be so thankful if someone could give an example of an encoder-decoder LSTM architecture over the shape of my dataset, especially in terms of dimensions required for encoder-decoder inputs and outputs, particulary on Keras model if possible.


texts:
I'm trying to understand the difference between target-values and action-values in Deep Q Networks.

From what I understand, action-value tries to approximate the reward of a given action (at some state). The target-value is also an approximate of the reward. How are they different?

Reference
- https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf


texts: Time Series Forecasting for Multiple Customers using one RNN

texts:
I have a product which has univariate and also multivariate time series data from multiple customers. I have variable amount of data available. Ranging between couple of years to couple of months.
What I want to achieve is a model which takes as input training data from all my customers and forecast for 1 at a time. Thus utilizing patterns from all of them and then forecast for one customer at a time.

E.g.

Train:
Date        Customer  Variable
2018-01-01  A         10
2018-01-01  B         5
2018-01-01  C         13
2018-01-02  A         9
2018-01-02  B         7
2018-01-02  C         15

Predict:
2018-01-03  B         ?

Any help on how to formulate this problem and what would be the best model for this?


texts: Gaussian process regressor returns almost identical std for all datapoints

texts:
I am using a Gaussian process regressor as the regressor for active learning and I use its standard deviation to choose the next training inctance (the one with the highest std is chosen). However, the std values returned by the regressor are almost identical as shown below, that doesn't seem right, especially given that the algorithm's performance doesnt improve after having been taught with 20 new instances that it has queried. I use this data-set. the way I go about that is I divide the data into training, pool and test sets and I use the pool dataset to draw the next most informative instance. the pool data-set contains about 40,000 data instances which makes it more strange why they all returned almost identical std's.
following is the function that is used to determine which instance to query:

def GP_regression_std(regressor, X_pool):
    _, std = regressor.predict(X_pool, return_std=True)
    query_idx = np.argmax(std)
    print(max(std))
    print(min(std))
    print(query_idx)
    return query_idx, X_pool[query_idx]

and the following is the code that calls the above function

# In this section a Gausian process regressor is used with the ActiveLearner
kernel = RBF(length_scale=1.0, length_scale_bounds=(1e-2, 1e3)) \
         + WhiteKernel(noise_level=1, noise_level_bounds=(1e-10, 1e+1))

regressor = ActiveLearner(
    estimator=GaussianProcessRegressor(kernel=kernel),
    query_strategy=GP_regression_std,
    X_training=X_train, y_training=y_train
)

y_pred = regressor.predict(X_test)
print(mean_squared_error(y_test, y_pred))

n_queries = 10
for idx in range(n_queries):
    query_idx, query_instance = regressor.query(X_pool)
    query_instance = query_instance.reshape(1,-1)
    query_label = y_pool[query_idx].reshape(1,-1)
    regressor.teach(query_instance, query_label)
    X_pool = np.delete(X_pool, query_idx, 0)

y_pred_final = regressor.predict(X_test)
print(mean_squared_error(y_test, y_pred_final))

and the following is the output:

39.80552437273547
3.1885401799262976
3.188521246966187
2807
3.1881109991791265
3.1880928244904516
21227
3.1876946235906867
3.1876781696380587
16901
3.18729221237594
3.187276617863699
15687
3.1869024420386807
3.186887562308426
17904
3.186524013419068
3.186510422046729
2204
3.186157844820368
3.1861446681566896
22230
3.1858019593432925
3.185789801369966
17653
3.1854569923377287
3.1854453279443624
37200
3.185121554225284
3.185110784757463
27299
39.89055846241666


texts: "RuntimeWarning: overflow[...]" in TensorFlow in juypter notebook

texts:
I'm implementing a captcha solver in tensorflow (also using ipython/jupyter notebook). After adding more layers to my model it now prints this message in a red box which I don't understand. What does it mean? I'm thinking it has something to do with the loss since I'm using tf.reduce_sum in the loss term (softmax-cross-entropy) which is also mentioned in the message.

/home/user/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: RuntimeWarning: overflow encountered in exp
  This is separate from the ipykernel package so we can avoid doing imports until
/home/user/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: RuntimeWarning: invalid value encountered in true_divide
  This is separate from the ipykernel package so we can avoid doing imports until
/home/user/anaconda3/lib/python3.6/site-packages/numpy/core/_methods.py:32: RuntimeWarning: overflow encountered in reduce
  return umr_sum(a, axis, dtype, out, keepdims)

What does this mean?


texts:
How would you evaluate unsupervised sentimental analysis?
I am reading on evaluating sentimental analysis and learning that much of the classification models that are being used, the data has target/label variables. How would you evaluate a survey that is only text with no target/label variables? Right now, I am just giving the responses a sentimental score (positive/negative/neutral) but I am having trouble approaching validation with 1000+ responses. Any ideas/suggestions?


texts: Set value for column based on two other columns in pandas dataframe

texts:
I have a dataframe that has contracts with different order dates and I need to create a new column that assign a number to each contract if it has more than one order date. For example my sample dataframe looks something like this:
df = pd.DataFrame({'contract': ['123A','123A','123A','123A','123B','123B','123C'],'prod': ['X1','M1','V1','D1','A1','B1','C1'],'date':['2019-04-17','2019-07-02','2019-04-17','2019-07-02','2019-04-17','2019-09-01','2019-08-02'],'revenue': [5688,113932,5688,49157,5002,892,9000]})

I need my final table to have another column with a unique contract id for each date. My final table from above should look something like this:

contract
date
header_contract

123A
2019-04-17
123A_0

123A
2019-07-02
123A_1

123A
2019-04-17
123A_0

123A
2019-08-02
123A_2

I have the following code that does what I need on a smaller dataset:
contracts_num = df['contract'].unique()
for cm in contracts_num:
    for idx,val in enumerate(df[df['contract'] == cm]['contract'].dt.date.unique()):
        df.loc[((df['contract'] == cm) & (df['contract'] == str(val))),'contract'] = df['contract'] + '_' + str(idx)

I'm trying to do it on a much larger dataset (around 50,000 contracts) and it's taking a really long time. Is there anyway to make it more efficient?


texts: Cold start recommender system with features

texts:
I have to develop a recommender system where most of the users only buy 1 item, so I have a cold-start problem. For this reason, I'm discarding matrix factorization techniques and content-based systems in general.
Luckily, I have lots of features for each user. I have the location, age, gender, the device they use to log in, the web browser, mobile carrier, answers to a small login form, among others.
What approaches can I follow given that I have this cold start problem with features?
I understand I can treat it as a multi-class classification problem, where I predict the item that the user is most likely to buy. This has two inconveniences I can think of:

What happens when a new item is added? Do I need to retrain the whole model?
How do I prevent the system from recommending only the most common item? The multi-class problem might give higher probabilities to very common items and my recommendations might get dummy (like recommending bread on a supermarket, doesn't add any value since the user is going to buy it anyway).

Can these issues be solved with a classification approach? Are the other approaches to address these?


texts: Which functions neural net can't approximate

texts:
I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii. Maybe this is an incorrect example or wrong information. Help me to understand.

What about the sum or multiplication of any arbitrary numbers? Are there any other specific functions neural networks can't approximate or not?


texts: Including user-item pairs without interactions in implicit feedback dataset for recommender system

texts:
I have a dataset which contains information about how many times a particular user viewed certain item. So, I don't have rows for all combinations (where the value will be zero ofc because the user has not viewed it), and it would be costly to include this kind of info.

Do I need to include such information if use lightfm, implicit or spotlight library, or they will take care of that in a way that they include them in a calculation (those combinations are important by popular paper http://yifanhu.net/PUB/cf.pdf, because even combinations without interactions have some default value/confidence).


texts: Comparing feature importance in LightGBM + Scikit

texts:
I have a model trained using LightGBM (LGBMRegressor), in Python, with scikit-learn.

On a weekly basis the model in re-trained, and an updated set of chosen features and associated feature_importances_ are plotted. I want to compare these magnitudes along different weeks, to detect (abrupt) changes in the set of chosen variables and the importance of each of them. But I fear the raw importances are not directly comparable (I am using default split option, that gives importance to a feature based on the number of times it appears in the candidate models).

My question is: assuming these different raw importances along weeks are not directly comparable, is a normalization enough to allow the comparison? If yes, what is the best way to do such normalization? (maybe just a division by the highest importance of the week).

Thank you very much


texts: Artificial Neural Networks and Efficient Parameter Optimization

texts:
I have a bunch of test measurements data and a semi-empirical model that has 18 parameters which I have to find so that the model fits my data well. So far I've managed to find and optimise the parameters using Optimisation and Global Optimisation algorithms in MATLAB.

Now I would like to explore different approaches for the parameter estimation. I have read some papers where the approach with NNs is described. I am new to NNs and have no idea if this is even possible.

I would create a two layer network with 18 input and output neurons. I am not sure what kind of transfer function would be appropriate for the problem.

The formula I have to fit and find the parameters look like this:

$ y = D \sin(C \arctan(Bx - E(Bx - \arctan(Bx))))$

where $B, C, D, E$ are macro-coefficients and the micro-coefficients are used to express the variation of each of the primary coefficients with respect to some other data.

This is how my data looks like.
How would you create network in MATLAB for this problem? Can you give me some hints and a push in the right direction to tackle this problem?

Thanks in advance.


texts: How does an individual feature affect predictions in neural network classification problem?

texts:
In the literature, I've come across statements like People with higher income and with long working hours are more likely to be diagnosed with chronic diseases such as stroke. The above-mentioned study (Page:8), explores the association between Behavioral Habits and Chronic Diseases using ANN.

As I'm new to ML,

I am unable to figure out how to make such conclusions with feature
study in neural networks or other machine learning techniques.
Is there a way to quantify the likelihood in ANN similar to logistic
regression wherein regression coefficients give the change in the
log odds of the outcome for a one unit increase in the predictor
variable?

Currently using Azure ML studio


texts: What does the KFold error mean and how to get confusion matrix from Kfold random forest implementation?

texts:
from sklearn.model_selection import KFold

num_folds = 10

seed = 77

kf = KFold(n_splits=num_folds,random_state=77,shuffle=False)

rfc=RandomForestClassifier(n_jobs=2,n_estimators=500,criterion="entropy",random_state=9999)

kf.get_n_splits(X)

for train_index, test_index in kf.split(X):

    X_train, X_test = X[train_index], X[test_index]

    y_train, y_test = y[train_index], y[test_index]

    rfc.fit(X_train, y_train)

    print(confusion_matrix(y_test, rfc.predict(X_test)))

print(10* '#')

I get error which I dont understand.
KeyError: "None of [Int64Index([100, 101, 102, 103, 104, 105, 106, 107, 108, 109,\n            ...\n            990, 991, 992, 993, 994, 995, 996, 997, 998, 999],\n           dtype='int64', length=900)] are in the [columns]"


texts: How to use ndcg metric for binary relevance

texts:
I am working on a ranking problem to predict the right single document based on the user query and use the NDCG metric to measure the model.

Given the details :

Queries ( Q ), Result Document ( D ), Relevance score.
But the relevance score is a binary ( 0 or 1 ) i.e out of document lists, only one document is marked as relevance score =1.

Data set example:

 query, docs,relevance
{
[1, doc2,0],[1, doc3,0],[1, doc4,0 ],[1, doc6,1],[1, doc9,0]
[2, doc3,0],[2, doc5,1],[2, doc10,0],[2, doc11,0],[2, doc1,0]
}

My questions:
1. Is it possible to use the NDCG metric for binary relevance problems?
2. If so, please share some reading notes or suggestions.

Thanks


texts: Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?

texts:
Consider a simple 1-D function $y = x^2$ to find a maximum with the gradient ascent method.
If we start in point 3 on x-axis:
$$ \frac{\partial f}{\partial x} \biggr\rvert_{x=3}   = 2x \biggr\rvert_{x=3} = 6 $$
This means that a direction in which we should move is a $6$.
Gradient ascent gives rule to update:
x = old_x + learning_rate * gradient
What I can't understand why we need to multiply a learing_rate with gradient. Why we can't just use x = old_x + learning_rate * sign(gradient).
Because if we made a  learning_rate step in a positive direction it is already a maximum switch of x we can make.
I know the reasoning behind finding maximum direction in this equation:
$$grad(())⋅⃗=|grad(())||⃗|cos() $$
But I can't undestand why just to accept a sign of gradient (plus or minus) is not enough for ascending.


texts: Association Rules - Data Mining - Train and Test approach?

texts: Does it make sense to use the train, test, and validation model using the Association Rules Technique?


texts: Creating data model out of .csv file using Python

texts: I want to create a data model out of a .csv file using Python. I mean to create dependencies, for example the primary key and stuff such that I can check if the new .csv complies with the given data model. I would appreciate some suggestions regarding how to do that, the libraries, frameworks etc


texts: Understanding ConfusionMatrix for Google Prediction API

texts:
I'm trying to analyze my training model, Google Prediction API provides analyze method to get insights for the model.
Currently I want to improve confidence levels, for my predictions, I haven't found any documentation in how to read this ConfusionMatrix, any insights will be great:

{
 "kind": "prediction#analyze",
 "id": "modelX",
 "selfLink": "https://www.googleapis.com/prediction/v1.6/projects/projectX/trainedmodels/modelX/analyze",
 "dataDescription": {
  "outputFeature": {
   "text": [
    {
     "value": "labelA",
     "count": "681"
    },
    {
     "value": "labelB",
     "count": "127"
    },
    {
     "value": "labelC",
     "count": "814"
    },
    {
     "value": "labelD",
     "count": "427"
    }
   ]
  },
  "features": [
   {
    "index": "0",
    "text": {
     "count": "2049"
    }
   }
  ]
 },
 "modelDescription": {
  "modelinfo": {
   "kind": "prediction#training"
  },
  "confusionMatrix": {
   "labelA": {
    "labelA": "14.17",
    "labelB": "0.17",
    "labelC": "3.83",
    "labelD": "0.67"
   },
   "labelB": {
    "labelA": "0.50",
    "labelB": "2.00",
    "labelC": "1.33",
    "labelD": "0.00"
   },
   "labelC": {
    "labelA": "1.17",
    "labelB": "0.00",
    "labelC": "70.00",
    "labelD": "3.50"
   },
   "labelD": {
    "labelA": "1.17",
    "labelB": "0.17",
    "labelC": "4.17",
    "labelD": "12.17"
   }
  },
  "confusionMatrixRowTotals": {
   "labelA": "18.83",
   "labelB": "3.83",
   "labelC": "74.67",
   "labelD": "17.67"
  }
 }
}


texts: how to extract the Top contributing labels/words in universal-sentence-encoder-large - TransformerModel?

texts:
I'm using the universal-sentence-encoder-large (Transformer Model) encoding process for embedding and then using the embedding for Clustering - Basically for unsupervised learning.

I want to get the top features / words which are contributing to the resultant clusters, in order to define the cluster or name the cluster in the business aspect with the help of the words contributing effectively for the cluster.
Is there any way to do this using the universal-sentence-encoder-large?

where can I actually find the usage of universal-sentence-encoder-large's functions, like,

Fucntions:

embed.export()
embed.get_attached_message()
embed.get_input_info_dict()
embed.variable_map()
embed.variables()
embed.get_signature_names()**

Any help is hugely appreciated!

Thanks
Arav


texts: Fractions or probabilities as training labels

texts:
This it a problem that has come on my path a few times now and I don't have a satisfying solution yet. The goal is to predict probabilities or fractions based on some $x$ where our training $y$ has these probabilties or fractions and thus is in the domain $[0,1]$ as opposed to $\{0,1\}$. My question is with regards to my loss function. In the case of fractions, if the error between 0.4 and 0.5 and the error between 0.89 and 0.99 is the same I can just use MSE if I want to predict the expected value.

In case of probabilities where we want to approach it similarly as classification problems, where the difference between 0.89 and 0.99 is much bigger than that of 0.4 and 0.5, we want to put this in our loss function. Does cross entropy still work properly if I feed it fractions in $y$?

$\mathcal{L}(y,\hat{y})=-y\log(\hat{y}) - (1-y)\log(1-\hat{y})$

Let's say our $y=0.5$ and our current prediction is $\hat{y}=0.6$ we would get:

$\mathcal{L}(0.5,0.6)=-0.5\log(0.6) - 0.5\log(0.4)$

I don't really see why this would go wrong? The function is still convex. Everywhere it says that the target should be in $\{0, 1\}$ however. Maybe my math is lacking or I'm missing something obvious, why is this a bad idea?


texts:
I am trying to replicate a network for facial key point detection like in the following link Daniel Nouri's Blog on KFKD. The blog uses Lasagne but i am trying to do using Tensorflow. I am unable to follow the part of performing data augmentation in Tensorflow.

All the methods I saw either did augmentation before training or were applied only on X values not on Y.

I want to find the replacement of FlipBatchIterator class used in the blog

from nolearn.lasagne import BatchIterator

class FlipBatchIterator(BatchIterator):
flip_indices = [
    (0, 2), (1, 3),
    (4, 8), (5, 9), (6, 10), (7, 11),
    (12, 16), (13, 17), (14, 18), (15, 19),
    (22, 24), (23, 25),
    ]

def transform(self, Xb, yb):
    Xb, yb = super(FlipBatchIterator, self).transform(Xb, yb)

    # Flip half of the images in this batch at random:
    bs = Xb.shape[0]
    indices = np.random.choice(bs, bs / 2, replace=False)
    Xb[indices] = Xb[indices, :, :, ::-1]

    if yb is not None:
        # Horizontal flip of all x coordinates:
        yb[indices, ::2] = yb[indices, ::2] * -1

        # Swap places, e.g. left_eye_center_x -> right_eye_center_x
        for a, b in self.flip_indices:
            yb[indices, a], yb[indices, b] = (
                yb[indices, b], yb[indices, a])

    return Xb, yb

net3 = NeuralNet(
# ...
regression=True,
batch_iterator_train=FlipBatchIterator(batch_size=128),
max_epochs=3000,
verbose=1,
)


texts: Prevent model from over-focusing on strong features

texts:
I have a classification model (DNN/Linear layers with some transformers and other things later). The input to the model are several different modalities of different lengths and different amounts of information. I am trying to mitigate the dimensionality difference by projecting different modalities into the same dimensional space and then combining them into the same space. However, the model seems to be focusing completely on the strongest input features and almost completely ignores the abundance of weaker ones. All of the input features are normalized (bools to 0/1 and continuous to mean=0, stddev=1), so the issue is not the scale of feature values but the predictive power of those few features which end up choking others.
Are there any methods out there for addressing this?


texts: What is the difference between SVM and logistic regression?

texts:
While reading the book by Aurelien Geron, I noticed that both logistic regression and SVM predict classes in exactly the same way, so I suspect there must be something that I am missing. In the Logistic regression chapter we can read:

  $σ(t) < 0.5$ when $t < 0$, and $σ(t) ≥ 0.5$ when $t ≥ 0$, so a Logistic Regression model predicts $1$ if $θ^T · x$ is positive, and $0$ if it is negative.

Similarly, in the SVM chapter:

  The linear SVM classifier model predicts the class of a new instance x by simply computing the decision function $w^T · x + b = w_1 x_1 + ⋯ + w_n x_n + b$: if the result is positive, the predicted class $ŷ$ is the positive class ($1$), or else it is the negative class ($0$).

I know that one way they could be different is because of the loss function they use: while log loss is used in logistic regression, SVM uses hinge loss to optimize the cost function. However, I would like to get this thing completely clear. How are the two models actually different?


texts: How to annotate labels in a 3D matplotlib scatter plot?

texts:
I have made a 3x3 PCA matrix with sklearn.decomposition PCA and plotted it to a matplotlib 3D scatter plot.

How can I annotate labels near the points/marker? Here is my code:

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
%matplotlib notebook
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.scatter(
       existing_df_3dx['PC0'], existing_df_3dx['PC1'], existing_df_3dx['PC2'],  # data
       s=60                                       # marker size
       )

plt.show()

Also, if you know of a better way to plot a 3D PCA, please post your code

Edit:

A solution provided from Stack Overflow:

for i in range(len(data_df_3dx)):
    x, y, z = data_df_3dx.iloc[i]['PC1'], data_df_3dx.iloc[i]['PC2'], data_df_3dx.iloc[i]['PC3']
    ax.scatter(x, y, z)
    #now that you have the coordinates you can apply whatever text you need. I'm
    #assuming you want the index, but you could also pass a column name if needed
    ax.text(x, y, z, '{0}'.format(data_df_3dx.index[i]), size=5)


texts: Prediticting global solar irradiance using satellite images

texts:
I have the aim to build a model to predict global horizontal irradiance (ghi) using satellite images and other features namely the day of the year and time of the day.
For extracting the satellite images features, I aim at using one of the available network in keras (e.g. Xception). Then I can concatenate the features output from the CNN to the other features (I already made the modifications to keras to handle those requirements.
Now, essentially, ghi is also a time-series problem so I assumed using time-series techniques could help. But I don't know where to look for those architectures (CNN + Time series + other features) so :

given my problem and desires, do you have ressources to help me tackle this issue?

one thing that is not yet clear to me is if I have don't freeze Xception internal layers, are the fiters parameters automatically learned by the model? I read here and there that filters are optimized also but I never found in keras a clear statement about that.


texts: Is it possible to decode which neuron represent which feature and why does it represent it?

texts:
In a neural network, Each neuron in the network represents some part of non-linear feature of the input. Ex: Like in mnist data, Consider the stem of number 9 is cut into multiple pieces and different part is represented by different neurons in the first hidden layer(Just an example from 3B1B neural networks video). My questions are:

What determines which neuron get to represent which part of the stem??

Is it possible that if we pass in the same input multiple times, each neuron can represent different part of the stem?? OR Is it that its all the magic of chain rule (i.e At the beginning, all neuron represent some trash feature and as updation of weights occur and then particular features have become synonymous to a particular neuron.) If so, How does this happen?

Thanks in advance


texts: How do I implement the sigmoid function in Octave?

texts: so given that the sigmoid function is defined as hθ(x) = g(θ^(T)x), how can I implement this funcion in Octave given that g = zeros(size(z)) ?


texts: Is web analytics similar to data science?

texts:
I just finished PhD and initially wished to work on data science and deep learning. However, after some rounds of interviews, I have been offered a job of web analytics and business intelligence at a medium size company. Is there any similarity with data science, and is there a future in it?

Because of some precarious situation, I have to accept this job, but should I keep looking for another job meanwhile, or will the experience be helpful to rise up the ladder or join a big brand?


texts:
I know that Autoencoders can be regarded as non-linear generalisations of PCA, but I struggle to understand in depth the analogy between the two.

Once PCA has been performed on a function $F(\vec{\theta})$ of the parameters $\vec{\theta}$, that function can be re-written as

$F(\vec{\theta}) = \sum_{i=1}^{N_{\mathrm{PCA}}} \alpha_i(\vec{\theta}) \, b_i$,

where $b_i$ are the PCA basis functions and $\alpha_i (\vec{\theta})$ the $N_{\rm{PCA}}$ PCA coefficients.

For Autoencoders, one can also build basis functions, by decoding the basis vectors $(1, 0, \dots 0), (0, 1, \dots 0), \dots (0, 0, \dots, 1)$ through the decoder, once the Autoencoder has been trained. Can a function $F(\vec{\theta})$ still be rewritten as $F(\vec{\theta}) = \sum_{i=1}^{N_{\mathrm{AE}}} \alpha_i(\vec{\theta}) \, b_i$, where $N_{\mathrm{AE}}$ is the number of encoded features in the central layer of the Autoencoder?


texts: Is applying simultaneous K Fold Cross Validation and Drop out possible?

texts:
Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem impossible to me. But I couldn't find anything similar. Could you please enlighten me?

Here's the code that I tried to run k-fold cross validation with Drop out in layers and failed with the following error

#Using kfold cross validation in order to evaluate the ANN
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score

#Adding grid search for parameter tuning
from sklearn.model_selection import GridSearchCV

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

def build_classifier():
    classifier = Sequential()
    classifier.add(Dense(output_dim=6, init= 'uniform', activation='relu', input_dim=11))
    classifier.add(Dropout(p=0.1))
    classifier.add(Dense(output_dim=6, init= 'uniform', activation='relu'))
    classifier.add(Dropout(p=0.1))
    classifier.add(Dense(output_dim=1, init='uniform', activation="sigmoid"))
    classifier.add(Dropout(p=0.1))
    classifier.compile(optimizer='adam', loss="binary_crossentropy", metrics=['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size=10, nb_epoch = 100)
accuracies = cross_val_score(estimator = classifier, X=x_train, y = y_train, cv=10, n_jobs = -1) #cv is the number of folds

error:

UnboundLocalError: local variable 'a' referenced before assignment

While there is no "a" there!


texts: Quasi-categorical variables - any ideas?

texts:
Let's say I'm trying to predict a person's electricity consumption, using the time of day as a predictor (hours 00-23), and further assume I have a hefty but finite amount of historical measurements.

Now, I'm trying to set up a linear model akin to

$power.used = \alpha* hr.of.day + \beta * temperature$

Problem: using the $hr.of.day$ as a numerical value is a very bad idea for many reasons, the fact that 23 and 0 are actually quite close values is one problem that can be solved with a simple transformation [1]. The fact that electrical consumption is often bi-modal is another problem which isn't solved by a simple transformation.

A possible solution that works rather well is to treat the time of day as a categorical variable. That does the trick, but it suffers from a significant drawback in that there's no information sharing between neighbouring hours.

So what I'm asking is this: does anyone know of a "soft" version of categorical values? I'm suggesting something quite loosely defined: Ideally I would have some parameter alpha that reduces the regression to numerical regression where $\alpha = 1$ and reduces to categorical regression where $\alpha = 0$, and behaves "in between" if it's some other number.

Right now the only answer I can think of is to alter the weights in the regression in such a way that they tend towards zero the further away the quasi-categorical value is from the desired value. Surely there are other approaches?

[1]
introduce the hour variable as two new variables: $cos(time.of.day/24)$ and $sin(time.of.day/24)$


texts: Best way to model a problem where multiple rows contribute to the result

texts:
This is my first question here and I am very computer savvy but a ML novice. I have done lots of learning algo's where the question fits on a single row so I can train the model with a lot of rows and then give it a value and a clear result is found.
However I have come up with a problem that doesn't fit this pattern and I am stuck on where to start.
I have a formula (simplified representative is)
Z/X = Y
Where:

X can be one of 6 number (1, 2, 3, 4, 5, 6)
Y is an integer between 0 and 20 (inclusive)
Z is the result

Y is the number I am trying to find
The problem is X is invisible to me, the only number I can see is Z. I have multiple rows in the dataset ranging from a single row to 100 rows.
What I want to do is take the list of Z and back calculate the certainty of working out Y. As an example:
Z=

30
10
10
15
5

If you look at these numbers above and apply the 6 possible version of X you get:

30 / 1 = 30 (outside range), 30 / 2 = 15(valid), 30 / 3 = 10(valid), 30 / 4 = 7.5(invalid), 30 / 5 = 6(valid), 30 / 6 = 5(valid)
10 / 1 = 10(valid), 10 / 2 = 5(valid), 10 / 3 = 3 1/3(invalid), 10 / 4 = 5/2(invalid), 10 / 5 = 2(valid)  10 / 6 = 5/3(invalid)
10 / 1 = 10(valid), 10 / 2 = 5(valid), 10 / 3 = 3 1/3(invalid), 10 / 4 = 5/2(invalid), 10 / 5 = 2(valid)  10 / 6 = 5/3(invalid)
15 / 1 = 15(valid), 15 / 2 = 7.5(invalid), 15/ 3 = 5(valid), 15 / 4 = 3.75(invalid), 15 / 5 = 3(valid)  15 / 6 = 2.5(invalid)
5 / 1 = 5(valid), 5 / 2 = 2.5(invalid), 5 / 3 = 5/3(invalid), 5 / 4 = 5/4(invalid), 5 / 5 = 1(valid)  5 / 6 = 5/6(invalid)

What all of them have in common is 5 is a possible value of Y and therefore there is a good chance that Y is 5
So far it looks like a simultaneous equation but the problem exists that the Y can change over time (but very slowly), so it isn't always an exact science. Also the formula isn't exact so the observed Z has a wander error in it.
If I had one row the certainty would be lower, if I have a mix (as above) then there really is only one so I would expect certainty to be higher.
My question is:
Does anyone have any advice on how to tackle this problem, algorithm's to google etc?
Thanks
Dave


texts: Neural Network learning in datasets that lie in small intervals

texts: I am trying to use Keras to learn a Neural Network that predicts the function $f(x)=\frac{x^2}{2}$. The network offers good performance when I try to learn the function over the intervals $[-10,10]$ and other similarly big intervals, but when I try to learn the function over $[-0.001,0.001]$, the neural network simply outputs the same value every time, or something extremely close to it when making a prediction. Does anyone have any idea why that is? I am currently trying out 3 hidden layers with the relu activation function, a dataset of size 10000, and 10 epochs.


texts:
I've gotten to love using the terminal for processing data. At work, I'm using Windows Subsystem for Linux (WSL). For school and personal, I'm using Linux terminal on a Google Cloud instance.

I love how quick I can move files and code around in the terminal and using Vim editor. I love the minimalism of text directories, etc. The major downside is I haven't been able to use charts, like matplot or seaborne, because I'm in a terminal.

Is there a solution, other than copying files and code over to something like Jupyter? I can write modules to plot basic bar charts in the terminal, but I obviously need more than that.


texts:
Can someone give me a tip on how I could incorporate MSE & loss plots? I have been following some machinelearningmastery posts to plot this but the application is classification and I am attempting regression. Also what is different in my script is I am defining the model thru calling a function, so I am curious if my script could be re-written without the function def wider_model() that defines the model.

This script below works except what is commented out on the bottom for the plt plots. In the machinelearningmastery post, someone does ask this question how to do for regression, and supposedly if you print print(history.history.keys()) two values are returned for dict_keys([‘mean_absolute_error’, ‘loss’])...

Any tips help, there isn't a lot of wisdom here... Thank you

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import math

# load dataset
dataset = pandas.read_csv("prepdSPdata.csv", index_col='Date', parse_dates=True)

# shuffle dataset
dataset = dataset.sample(frac=1.0)

# split into input (X) and output (Y) variables
X = numpy.array(dataset.drop(['Demand'],1))
Y = numpy.array(dataset['Demand'])

print(dataset.shape)
print(dataset.dtypes)
print(dataset.columns)

def wider_model():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=11, kernel_initializer='normal', activation='relu'))
    model.add(Dense(10, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=1, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print("RMSE", math.sqrt(results.std()))

# list all data in history
#print(wider_model.wider_model.keys())

# summarize history for MSE
#plt.plot(history.history['acc'])
#plt.plot(history.history['val_acc'])
#plt.title('model MSE')
#plt.ylabel('MSE')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()

# summarize history for loss
#plt.plot(history.history['loss'])
#plt.plot(history.history['val_loss'])
#plt.title('model loss')
#plt.ylabel('loss')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()


texts: Consensus clustering: how to choose the final cluster based on the consensus matrix?

texts:
I've been reading about consensus clustering and the consensus matrix in this article. I understand how the consensus matrix is made after re-sampling and clustering parts of your data H times. I understand that the consensus matrix is used to determine the optimal amount of clusters (k) and allows for making a nice heatmap. What I don't understand is how this consensus matrix results in your final clustering.
Say I have a consensus matrix of 4*4 (so we have 4 items to be clustered) where each value between 0 and 1 in the matrix represents the number of times items i and j are assigned to the same cluster, divided by the total numer of times both items are selected for clustering. We could have the following consensus matrix after 4 iterations of 80% sub-samples (taken from here). We kept track of all 4 clusters that were made in the process of obtaining this consensus matrix. How do we select the final clustering based on this consensus matrix?


texts: How to improve accuracy further for forest cover prediction

texts:
I am doing Forest-Cover-Type-prediction on Kaggle

this is the train and test data

> dim(train)
[1] 15120    56
> dim(test)
[1] 565892     56

So far I have done this :

combined this data comb <- rbind(train,test)

then mapped 40 mutually exclusive Soil_Type columns into one and 4 mutually exclusive Wilderness_Area column into one.

comb$Soil <- apply(comb[grep("Soil_Ty+",colnames(comb))], 1 , function(x){which(x == 1)})
comb$Wilderness <- apply(comb[grep("Wilderness_Are+",colnames(comb))], 1 , function(x){which(x == 1)})

after deleting these 44 columns I got it down to 14 columns.

> dim(comb)
[1] 581012     14
> colnames(comb)
 [1] "Id"                                 "Elevation"                          "Aspect"
 [4] "Slope"                              "Horizontal_Distance_To_Hydrology"   "Vertical_Distance_To_Hydrology"
 [7] "Horizontal_Distance_To_Roadways"    "Hillshade_9am"                      "Hillshade_Noon"
[10] "Hillshade_3pm"                      "Horizontal_Distance_To_Fire_Points" "Cover_Type"
[13] "Soil"                               "Wilderness"

then I separated the data

train <- comb[1:15120,]
test <- comb[15121:581012,]

and ran randomForest on this data

set.seed(415)
fit <- randomForest(Cover_Type ~ .,data=train[-1], importance=TRUE, ntree=2000, na.action = na.omit)
varImpPlot(fit)
predi <- predict(fit,test)

and got an accuracy of 0.70372 on Kaggle. Now I have hit the wall. I tried plotting various variables against the Cover_Type(variable to be predicted) but couldn't do figure out what to do with those.

How do I improve accuracy from here? what is the general approach in such cases?


texts:
I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns.

First, I started with regular 3 column grayscale pages without images and tried to locate paragraphs. For that I separate page into 3 columns:

Calculating average of color (grayscale) along page width
Threshold application to get equal almost whites to whites
Peak detection
Cutting page based on peaks

As it comes to paragraphs I have done a similar process in vertical direction on single columns. Which gives more or less satisfactory results.

All of this fails for any page with images. Many of my searches on the internet lead to OCR solutions, which 1. Do not address my problem directly 2. Are trained models for Latin alphabet based languages, which is not my case.

Are there any standard approaches to paragraph extraction from pages with images?


texts: Is it possible that shap feature importance result will be more accurate than gain?

texts:
XGboost build the boosted tree in the following way:

Each level of each tree (the phase of selecting the next feature with conditional value)  selected according to the max gain (less impurity).

If we check the feature importance (based on gain) we sould see that the more the features have higher gain (same feature may contribute several times at each tree level) the more they will be important.

When building XGboost tree, there is no way that feature with less gain will be selected before feature with higher gain.

When using SHap:

The Shap method runs multiples instances of XGboost with different subset of features and calcuated the difference between the accuracy of results with or without the selected/ unselected features.

So is it possible that shap feature importance result will be more accurate than gain ?
If we run feature selection (wrapper model) and used the selected features to get the importance, is it possible that shap feature importance result will be more accurate than gain ?


texts: How to create a feature vector given final set of feature maps?

texts:
I've got a faster-rcnn (resnet-101 backbone) for object detection, and am extracting feature tensors for each detected object, which is a 7x7x2048 tensor (basically 2048 feature maps, each 7x7). For object tracking, I want to turn this into a Nx1 vector. What is the standard way to do this? I have a few ideas that all seem reasonable:

Flatten each feature map, and then concatenate all these together (so each feature vector would be 49*2048 x 1)?
Do the same after doing some max pooling operation to decrease dimensionality.
Take the mean or max of each feature map, and end up with a 2048x1 feature vector.


texts: Imbalanced dataset in MLP classifier in python

texts:
I am dealing with imbalanced dataset and I try to make a predictive model using MLP classifier. Unfortunately the algorithm classifies all the observations from test set to class "1" and hence the f1 score and recall values in classification report are 0. Does anyone know how to deal with it?

model= MLPClassifier(solver='lbfgs', activation='tanh')
model.fit(X_train, y_train)
score=accuracy_score(y_test, model.predict(X_test), )
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])
cr=classification_report(y_test, model.predict(X_test))


texts: Machine Learning alternative for hashing

texts: Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out there that can do this as well or close to it.


texts: How to tell my neural network that I care much more about precision than recall?

texts:
I am training a neural network for a multilabel classification problem, so my last layer consist of n_classes sigmoid neurons.

Now, I know that it is impossible to nail the learning task both because of noise and lack of predictable power in my features. However, I am assuming that the predictable power should be enough to make good predictions at least for some data points. Even if these cases are just a few, I would be happy if the neural network gets at least those with certainty (high precision) and misses the grand majority because of their difficulty (low recall).

Of course, I could train the model normally and then set that threshold that matches my goals, but this doesn't convince me. I would like to introduce this goal in the training process. If some points are hard to classify I want my neural network to discover that during training, forget about them and focus on the other more doable cases.

So far my only idea is to make training robust towards outliers via label smoothing.

Any other ideas to influence the model during training like this?


texts: Models systematically underestimate values on the test set, why?

texts: I used different models to train on the test set and to predict on the test set. The commonality is that all models underestimate the true values on the test set. Which steps should I take to understand why?