texts:
sorry if this question is out of place. I'm a begginer to machine learning, and I have use for a technique, and I don't even know where to look. The problem is:

I have 5 features which are real valued (Parameters in a deterministic simulation).
This features determine two aspects of the instance (model solution). Its feasibility (binary) and some measure of likelihood given certain experimental data (only for the instances that achieved feasibility).

Since I want to avoid generating "infeasible" combination of features, what I devised was an algorithm that iteratively does the following:

Generate Nc candidate feature vectors
Evaluate Feasibility and Likelihood for each
Find linear combination of features that involves a compromise between least amount of features / holds largest cluster of feasibility. Add this as constraints to the feature vector generation.

In short, it detects and iteratively refines "simple" constraints that once added to the feature vector generation "guarantee" its feasibility to save computational time evaluating combination of parameters that lead to infeasible models. Afterwards, they could be tested by inverting them and looking for other "regions" (if any) of the feature vector where the model is feasible.

Any name of techniques and references I might look for ?

Thanks!


texts: Use single Convolution layer for the classification of two kinds of matrix?

texts:
I have two kinds of matrix with 10*10 size, and each number of matrix is 24000, all matrix forms like this:

0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 0 0 1 1 1 1 1 0 0
1 1 1 1 0 0 0 1 1 1
1 1 0 0 0 0 0 0 1 0
1 0 1 1 1 1 1 1 1 0
0 1 0 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0

and I must classify them with a single convolution layer.
the convolution kernel size is 5*5, the strides are 5*6, the final output is 2*1,the Neural network structure below:

X_input = Input(shape=input_shape)
X=Conv2D(1,kernel_size=(5,5),strides=(5,6),use_bias=None)(X_input)
X=Activation('relu')(X)
X=Flatten()(X)
Y=Activation('softmax')(X)
model=Model(inputs=X_input,outputs=Y,name="JSmodel")
return model

and the final accuarcy is 85.6%. However, I'm confused whether I design the neural network is accurate?


texts:
I am using silhouette_score to find the optimal k value. So I am running a for loop with a range of possible k values. I have added my code below. this program takes a very long time to run. Could you suggest some improvements for a more efficient run time?

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from sklearn.cluster import KMeans
from sklearn import metrics

data=np.load(filename)

coeffs=[]

for i in range(2,8):

    clusters=KMeans(n_clusters=i)
    clusters.fit(data)
    labels = clusters.labels_
    sil_coeff = metrics.silhouette_score(data, labels,metric='euclidean')
    coeffs.append(sil_coeff)

coeffs=np.array(coeffs)
k=np.argmax(coeffs)+2


texts:
I am trying to develeop an algorithm with sklearn and Tensorflow to predict which car can be offer to each customer.

To do that I have a database with the answers of one survey to 1000 customers.

An example of questions/[Answers] are:

Color/[Green,Red,Blue]
NumberOfPax/[2,4,5,6,7]
HorsePower/[Integer]
InsuranceIncluded[yes/no/Don't know]

As you can see all questions are answer previously tipified, and in case the answer can be open I validate the value to be an integer or a radio button.

The purpose of that beahivour is that despite the categorical variables I can easily use sklearn to clustering the data.

Will be a good approach to translate this categories to numerical value as an intern procedure an then cluster with this references?

For example: yes --> 0; No --> 1; Don't know --> 2

Then sklearn will cluster with all variables as numerical values.

I have thought this possibility beacuse I believe that sklearn can not cluster nominal data.

What do you think about this approach?


texts: Why will the accuracy of a highly unbalanced dataset reduce after oversampling?

texts: I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while after balancing (i.e., 100 samples in both the classes)  it is 60%. What are the possible reasons for this?


texts: How may I may fit the cluster number while it is going to the outside of the X-axis and when visualizing clusters in two dimensions?

texts:
I am writing code for DBSCAN clustering. I find the eps value which is 0.12 with the help of the Elbow/Knee method. But in the plot, the cluster number is going to the out of the X-axis. It about 177 clusters and only 13 clusters fitted within the figure box. But the other 164 clusters generated the outer side of the X-axis. So how may I fix this problem?

My code:

import seaborn as sns
dbscan = DBSCAN(eps=distances[knee.knee], min_samples=10).fit(Y)
labels = dbscan.labels_

#fig = plt.figure(figsize=(5, 5))
#fig = plt.figure(figsize=(36,12))

sns.scatterplot(Y[:,0], Y[:,1], hue= ["cluster-{}".format(x) for x in labels])

plt.title("DBSCAN Clustering Method")
plt.xlabel('Number of clusters', fontsize = 10)
plt.ylabel('Silhouette Score', fontsize = 10)
plt.title('DBSCAN Cluster with Silhouette Score Method ')

plt.show()


texts:
I'm trying to pass all my images (71 for each class) from folder 'train' to model.fit.
The method ReadImages get these images and resize them (because are too big 4288x2848)....

But when i run my code throws this error:

ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (142, 1)

This is my code:

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

    # Get all subdirectories
    FolderList = os.listdir(Path)

    # Loop over each directory
    for File in FolderList:
        if(os.path.isdir(os.path.join(Path, File))):
           for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
                # Convert the path into a file
                ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
                ImageCV[index] = cv2.resize(ImageCV[index], (700, 600))

                # Add a label for each image and remove the file extension
                LabelList.append(classes.index(os.path.splitext(File)[0]))
        else:
            ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
            # Add a label for each image and remove the file extension
            LabelList.append(classes.index(os.path.splitext(File)[0]))
    return ImageCV, LabelList

model = Sequential()
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu", input_shape=(605,700,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(4,4), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='RMSprop', loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)
print(data[0])
model.fit(np.array(data), np.array(labels), epochs=10, batch_size=20)

model.save('model.h5')

detail: when I print (data[0]) it return:

  [0 0 0]
  [0 0 0]
  ...
  [1 0 0]
  [2 2 2]
  [0 0 0]]

 [[3 0 0]
  [3 0 0]
  [3 0 0]
  ...
  [2 0 0]
  [2 2 2]
  [0 0 0]]

 [[8 0 0]
  [8 0 0]
  [8 0 0]
  ...
  [2 0 0]
  [0 0 0]
  [0 0 0]]

 ...

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ...
  [0 0 0]
  [0 0 0]
  [0 0 0]]]

What should I do to fix it? I appreciate any help


texts: Graphically Speaking - how weight vector is perpendicular to hyperplane

texts:
why-is-weight-vector-orthogonal-to-decision-plane-in-neural-networks

vector-orthogonal-to-hyperplane

I wanted to visualize this by drawing a 3d plane where and the weight vector. By definition of the plane, all the points on the plane have the same dot product with the weight vector.

Question - Why is the weight vector not perpendicular to the plane? Am I mathematically/code wise wrong here or is it that Matlab viewing angle is not appropriate to see the orthogonality of the weight vector (bold arrow) to the plane? Thanks.

Matlab code below:

w = [0.8 0.3 0.24]  ;%weight vector
%get locus of point whose w'x is same (CONTOUR); x is a vector - (x1 x2 x3)
%randomly choose points from grid and get x3 VALUE

[x1 x2] = meshgrid(-5:0.25:5, -5:0.25:5);
coords = [x1(:) x2(:)];
%say we want to get contour for w'x = 3
coords(:,3) = (3 - w(1)*coords(:,1) - w(2)*coords(:,2))/w(3)
%plot the plane
scatter3(coords(:,1), coords(:,2),coords(:,3),'MarkerEdgeColor','k','MarkerFaceColor',[0 .75 .75])
view(-10,35);
xlabel('coords(:,1)')
ylabel('coords(:,2)')
zlabel('coords(:,3)')
mArrow3([0 0 0],[8 3 2.4])   %overlay weight vector
p1 = [0 0 0]
p2 = [8 3 2.4]
text(p1(1),p1(2), p1(3), sprintf('(%.0f,%.0f,%.0f)',p1)) %show the origin
%show the weight vector direction
text(p2(1),p2(2), p2(3), sprintf('(%.0f,%.0f, %.0f)',p2))

Another view


texts: Multiclass classification with large number of classes but for each user the set of target classes is known

texts:
Hi Data Science Stack Exchange! I'm new here but I'm familiar with some machine learning theory (took some courses in school) and my question is more about how to apply ML in a practical setting.

I have this project where I'm trying to design a system to predict which "store" a user is going to buy a given item from. However, the set of stores a user can potentially buy from is already known for each user (because this promotion only works at limited stores they signed up for). On average, the set of stores the average user can buy from is around 3 but the # of distinct stores across all users is about 10000.

In theory, for a single user this seems like a simple classification problem. We have historical information such as the time/day/month the user bought item Y from store X along with other features such as location (postal code) where the user lives and features related to the type of item they are buying (cost, weight, brand).

However, the issue is that there's currently around 6000 users so going with this approach seems like I would need a separate model for each user but that doesn't seem like an efficient solution to me or at least not how I've generally seen ML algorithms used. Unfortunately, I don't see any other way I could take into account the fact that the set of stores a user can already buy from is known for each user already. I suppose I could have a categorical variable for each buy-able store as a feature but then that would be equivalent to adding 10000 features and I'm not sure if this would scale as the number of distinct stores increases.

It would be really helpful if anyone has any insight on how to apply machine learning techniques to this kind of problem in general as this is sort my first time working on a "real" problem. Thanks!


texts: What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?

texts:
After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard is hugely appreciated.

when we say hidden states are transferred from one segment to another, what exactly is included in these hidden states? Are the weights of the networks implementing the attention mechanism (i.e. calculating the Q, K and V) included? Are the weights involved in calculating the input word embedding included in the hidden state?
When the hidden states are transferred during recurrence, is this transfer from the encoder of one segment to the encoder of the next segment? Or is it from the decoder of the current segment to the encoder of the next segment? Is the decoder involved at all in the hidden state transfer?
I see images like the following in the following in the papers and blogs. what do the dots represent? encoders? decoders? or an entire unit? I guess the answer to my second question will shed a light on this one too.

Thank you


texts: How would I train an HMM using the EM algorithm so the transition matrix is upper diagonal?


texts: Binary classification of similar images with small region of interest

texts:
I have a dataset of microscope images and I want to train a ML/DL algorithm to perform binary classification. The positive class is when there is only one cell in the image, and the negative class is everything else (i.e. when there are either more than 1 cells, or no cells at all).

Below is one of the original images. (there is a cell in the curved site on the center of the image)

Due to the big size of the images (2048x2048) and the excess of information (the cells can only be in the tube-system), I decided to preprocess them. So, I set everything outside the tube-system to 0 (black) and I crop all the images to the boundaries I got by averaging the images of the whole dataset. Below you can see the end result. (there are 2 cells in the tube, one in the center and one at the upper left part)

Then I tried to train a CNN (using python and TensorFlow). I played a few times by changing its hyperparameters, but I had no luck. I think the problem is that the cells (region of interest) are occupying a very small portion of the image, which makes it hard for the algorithm to focus on. To make things worse, there are random dust particles around which make the image noisy.

Do you have any ideas of how I could perform a robust binary classification in such a dataset?


texts:
Our weapons:

I am experimenting with k-means and Hadoop, where I am chained to these options for various reasons (e.g. Help me win this war!).

The battlefield:

I have articles, which belong to c categories, where c is fixed. I am vectorizing the contents of the articles to TF-IDF features. Now I am running a naive k-means algorithm, which takes c centroids to begin with and starts, iteratively, grouping articles (i.e. rows of the TF-IDF matrix, where you can see here how I built it), until converenge occurs.

Special notes:

Initial centroids: Tried with random from within each category or
with the mean of all the articles from each category.
Distance function: Euclidean.

Question(s):

The accuracy is poor, as expected, can I do any better, by making another choice for the initial centroids, or/and pick another distance function?

print "Hello Data Science site!" :)


texts: How to perform Learning to Rank for a small dataset

texts:
I am very interested in applying Learning to rank to my problem doamin. When I read through the literature of Learning to rank I noted that the data they have used for training include thousands of queries.

However, in my problem domain I only have 6 use-cases (similar to 6 queries) where I would like to obtain a ranking function using machine learning. I know the data I have is very very small.

So, my question is; Can apply learning to rank in my problem? If not what are the other methods that I can use to make a ranking of my documents in each query.

I am happy to provide more details if needed.


texts: I am currently using caret which optimises for accuracy. Is it possible to optimise for sensitivity. I see documentation mentioning that metric = 'roc' has been used, but sensitivity does not appear to be one of the options.


texts: When to stop showing content on recommendation engines?

texts:
Let's take an example. I log into my Netflix account and see that it's suggesting the show Friends to me. But I have no interest in watching Friends. So I ignore it. The next time I login, it suggests Friends again and I ignore again. This goes on for quite a few number of logins until Netflix finds content that is probably more relevant to me than Friends.

My question is this: Is there a way to use the information that I ignored Friends for say 5 consecutive logins and inject some bias into the recommendation algorithm such that after 5 impressions and no clicks, it would stop recommending that content to me?

Note. This is a general question about recommendation engines and the use of Netflix and Friends is just to put things into perspective.


texts:
I want to introduce Batch Normalization in my C++/CUDNN implementation of CNN. The implementation is currently performing well (without BN) on the MNIST dataset.

I am using the CUDNN implementation of Batch Norm, but after having read the Batch Norm paper and the CUDNN documentation carefully, still there are some points that are not clear to me.

From what I understood, batch norm layers are placed between the output of convolutional/dense layers and the non-linearity, like:

u --->[conv/dense (weights W)]--->[BN]--->[ReLU]---> y = ReLU(BN(W*u))

In  some literature I read that placing the batch norm layer after the non-linearity is a better option. Is this true?

The CUDNN documentation says to use the BATCHNORM_MODE_SPATIAL for convolutional layers, and BATCHNORM_MODE_PER_ACTIVATION for dense layers. However, in another implementation (YOLO / Darknet), I only see BATCHNORM_MODE_SPATIAL being used. So I ask myself whether SPATIAL is also OK for dense layers?
As mentioned, from what I understood, BATCHNORM_MODE_SPATIAL should be used for conv layers: but on their input (i.e. before the convolution), or on their output (i.e. after the convolution) ?

From what I understood, it is to be used on the output, like:

u ---->[ conv (weights W) ] ---->[BN with mode SPATIAL]------>[ReLU]---> y

is this correct?

I see that in Darknet / YOLO they do not update the parameters explicitly with the deltas obtained from the cudnnBatchNormalizationBackward(). Does this method also perform the parameters update directly? I don't find any info on this, and I am actually applying a Gradient Descent update after having obtained the Gamma and Beta gradients from the cudnnBatchNormalizationBackward method.
I see that in the CUDNN implementation, a Forward method for Inference is available. Is it OK to completely skip the BN units during the inference phase? In this scenario, I would train with the BN units, and remove them from the network during inference.

I really thank you in advance for your patience and your kind help.
I tried to get rid of these questions alone for quite long, but I am now blocked... your help is really appreciated.


texts:
I have a number of large datasets (10GBs) each with data fetched from a NoSQL database that I have remotely downloaded on my desktop. I would like to write a Python program to run some custom data analysis (plots - preferably interactive) and export custom reports in html or pdf.

I was wondering how people do the following:

1) Store the data. For the moment I have plain text files (each file has rows of a fixed number of columns - most of the data are categorical). Would it make sense to save those in some database (SQL) or hdf5? Any hints on which is preferrable?

2) Which plotting library would you propose for the graphs? I have seen about bookeh and matplotlib supports interactive widgets but I don't know what people normally use.

3) Could I export the analysis results in an IPython notebook and then in html programmatically?


texts: Monthly trend with fb prophet-Interpreting the graph

texts:
I have monthly data with month/year in one column and price on another. I would like to get a yearly trend with fb prophet library in python (how to use monthly data with the library is explained at the end of this page ).

This is my code:

import fbprophet
import pandas as pd
import matplotlib.pyplot as plt

data = pd.read_csv('data.csv', sep=';')
data_prophet = fbprophet.Prophet(interval_width=0.95, changepoint_range=0.9, changepoint_prior_scale=0.15, seasonality_mode='multiplicative', n_changepoints=100)
data_prophet.fit(data)

# Make a future dataframe for 5 years
data_forecast = data_prophet.make_future_dataframe(periods=12 * 5, freq='M')
# Make predictions
data_forecast = data_prophet.predict(data_forecast)

#Plot
data_prophet.plot(data_forecast, xlabel = 'Date', ylabel = 'Price')
plt.show()

data_prophet.plot_components(data_forecast)
plt.show()

This is what I got:

At the first glance, seems that minimums/maximums at the yearly trend correspond to mid-month, but they are not at the same point each month. The picks are shifted a little bit.

How to draw a correct yearly plot? And does this make sense for monthly data?


texts: ROC analysis widget has an option to change prior target class probability. I'd like to know, when and how it should be used. Playing with it changes the slope of iso-performance line. Each classifier should be able learn priors from the data, so I don't understand what this is for.


texts: Number and size of dense layers in a CNN

texts:
Most networks I've seen have one or two dense layers before the final softmax layer.

Is there any principled way of choosing the number and size of the dense layers?
Are two dense layers more representative than one, for the same number of parameters?
Should dropout be applied before each dense layer, or just once?


texts: What is the reward function in the 10 armed test bed?

texts:
The Sutton & Barto book on reinforcement learning mentions the 10 armed test bed in chapter 2, Bandit Problems:

  To roughly assess the relative effectiveness of the greedy and
  ε-greedy methods, we compared them numerically on a suite of test
  problems. This was a set of 2000 randomly generated n-armed bandit
  tasks with n = 10. For each bandit, the action values, $q_∗(a), a = 1, . . . , 10,$ were selected according to a normal (Gaussian) distribution with mean 0 and variance 1. On tth time step with a given bandit, the actual reward $R_t$ was the $q_∗(A_t)$ for the bandit (where $A_t$ was the action selected) plus a normally distributed noise term that was mean 0 and variance 1 [. . . .] We call this suite of test tasks the 10-armed testbed.

What is the reward function in the 10 armed test bed ?
I interpreted it as something like q*(a) + some normally random value.
where q*(a) is the true value of action a.

Why is the reward function chosen this way in the test bed and how does the reward function affect the value estimations and the graphs ?

I guess the reason i'm asking this question is because i'm not completely clear how a reward looks like in the real world where i don't know anything about q*(a).


texts: Machine learning model to predict performance degradation over period of time

texts:
I am new in ML space and working on the task to build a model that would predict performance degradation of integration bus in real time or "almost real" time. The idea is fairly straightforward, I have logs from the production and filtered required information in pandas dataframe: "Provider\consumer system" (object), "Request Time" (dataframe64), "Response Time" (dataframe64), and "Time Performance" (float). The last column is the difference between the response and request times in ms, it is our target column to predict performance degradation. So, I have a list of providers and consumers sending messages through the integration system. I am thinking about a model that will predict that if some provider\consumer breaches the threshold for some time it would lead to an increase in performance time and will warn about this. So, we will get a warning in seconds\minutes after potential degradation starts.
When I researched similar problems, I found out that it may fall under the predictive maintenance category, but not much more about this area of ML. Would appreciate any thoughts about the approach or model that may help.


texts: Why do we need dot product as part of the Transformer's training process?

texts:
I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) each token in a sequence should put into other tokens. So the question is why is that important to have scalar value from the dot product when we could just learn a corresponding(a bigger for example) weight during training.
One of the reasons behind my confusion comes from a common example/explanation of how attention/self-attention works: model is able to 'understand' which other word token it is connected to.

it(animal) is too tired, or it(road) is too wide

In that scenario  dot product between it and animal/road should be almost the same( given reasonable initial words' embeddings). And we are still able to learn proper weights.
Questions:

If and in what way dot product helps to learn attention weights?
maybe there are other benefits that dot product brings us?


texts: Scipy.optimize getting back x0 for optimization of input to recurrent neural network

texts:
I need help with this optimization problem which is either not getting solved at all or is taking a copious amount of time.

I am trying to find optimized input to an RNN (GRU) model of a process plant, which will minimize the difference between the output and desired output. I am using obj = (model_output(x) - desired_output(x))**2

If I use
 1) sol = minimize(objective,x0, method = 'SLSQP', bounds = bnds, options = {'disp' : True}) It just gives back x0

2) sol = differential_evolution(objective,bnds) is taking extremely large amount of time (Several minutes for 1 x. If I use it iteratively, it takes several hours)

P.S. I mentioned recursively since x is an input over 20 timesteps, then I need to move from 1:20 indices to 2:21 and so on till 381:400 timesteps for my problem. So I am running this operation recursively using for loop.

Also bounds are simple bounds to input like x should be between (60,80).


texts: Cross validation with GridSearchCV or train-val-test split

texts:
I have a question regarding the CV in GridSearchCV.
To test my model should I split my data into 3: training, validation, test? For easy understanding let's say my data is split into training with 60% of the data, validation data 20%, and test data is the remaining 20%. I want to use my training data to train the model to tune my parameters with the validation data.
However, if I use GridSearchCV, I already use Cross Fold (=5 for example) which already splits the data. Does this mean that by using GridSearchCV I only need to split data into training and test?
Should I divide my data into 80% training and 20% test and use GridSearchCV on my training data (80%) with GridSearchCV to find parameters and then evaluate my model with test data?


texts: Which of the NIPS 2014 papers are most significant, and why?

texts:
As a newcomer to the field, I find many of the NIPS 2014 papers fascinating, but it is difficult for me to evaluate which ones represent real progress over current approaches.

Which papers do you think are most significant and are likely to have a major impact?


texts: Groupby date and find number of occurrences of a value a in another column using pandas

texts:
My Dataset is looking like this

I would like to get the output something like this.

I need to group by date and find the occurrences if each feedback. In this case, i have 5 feedback types (namely 5,4,3,2,1)


texts: how to improve baseline logistic regression in high dimensional binary classification problem?

texts:
info about dataset:

df.shape = (10000, 100)
all feature are numerical values.
few outliers in each column, column with most outlier have 0.7% of data as outlier.

I am trying to improve on my baseline logistic regression however I'm stuck.
baseline = LogisticRegression(solver='lbfgs', max_iter=100, penalty='l2')
Here are some approaches I've took and relative results

Standard scaler - Logistic regression (similar)
Robust scaler - Logistic regression (simliar)
remove outlier(IQR method) - standard scaler - Logistic regression (worse)
Standard scaler - PCA(n_component=n_comp that explain 83% variance) - Logistic regression (more worse)

All approaches seem to perform worse than baseline.
How can I improve my baseline logistic regression model or do I need to resort to nonlinear models like random forest(I've already tried it however it overfits)?
Thanks in advance!


texts:
I came across an error while invoking the Tensorflow model:

tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder_42' with dtype float and shape [?]

Code:

"""
This file is the main entry point for the service. It's responsible for loading
the model and implementing the "invoke" function.
"""

import tensorflow as tf
import numpy as np
import json
import logging

global tf_graph

import time

# load the vocabulary
with open('/opt/ml/model/vocab_dict.json', 'r') as f:
    vocab_dict = json.load(f)

def encode(text_tensor, label):
  encoded_text = [vocab_dict[t] for t in text_tensor.numpy().decode('UTF-8').split(' ')]
  return encoded_text, label

def encode_map_fn(text, label):
    """
    Required to convert the text to tensorflow data type.
    :param text and label
    :return: encoded text on tensorflow datatype.
    """
    encoded_text, label = tf.py_function(encode,
                                       inp=[text, label],
                                       Tout=(tf.int64, tf.int64))

    encoded_text.set_shape([None])
    label.set_shape([])

    return encoded_text, label

def labeler(example, index):
  return example, tf.cast(index, tf.int64)

def convert_request_to_model_input(request):
    """
    Given a request, returns a encoded value
    :param request: Input request
    :return: encoded value
    """
    notif = request['notificationLabel'].lower()
    notifCleaned = ' '.join([t if t in vocab_dict else 'UNK' for t in notif.split() ])
    lines_syntest = tf.data.Dataset.from_tensor_slices(np.array([notifCleaned]))
    labeled_syntest = lines_syntest.map(lambda ex: labeler(ex, 0))

    all_encoded_syntest = labeled_syntest.map(encode_map_fn)
    syntest_data = all_encoded_syntest.padded_batch(7, padded_shapes=([None],[]))
    return syntest_data

start_load = time.time()
# Store the session so that we can execute the model prediction in the same session.
# This is needed in a multi-threaded environment
session = tf.Session(graph=tf.Graph())
with session.graph.as_default():
    tf.keras.backend.set_session(session)
    model_britney = tf.keras.models.load_model('model.h5')
print(f"Time for load- {time.time()-start_load}")

start = time.time()
    with session.graph.as_default():
        start_session = time.time()
        tf.compat.v1.keras.backend.set_session(session)
        print(f"Time for session setting - {time.time()-start_session}")
        start_predict = time.time()
        model_input = convert_request_to_model_input(request)
        proba_syntest_britney = model_britney.predict(model_input)
        print(f"Time for model prediction - {time.time()-start_predict}")

```


texts: Restricting a weight vector (optimization parameter) to be in a certain domain using python ML library linear regression model

texts: Sorry if the title is a bit long, but basically I'm trying to predict values $$ \hat{y}_i \in [-1,1]$$ using a simple model i.e. something like OLS or ridge regression, I'd like to know if anyone knows of a library that has models with features that all you restrict your weight vector to be in a certain domain.


texts:
I want to integrate simple python code with in the tensorflow graph.
I'm not sure if it's feasible. If feasible, please suggest how to integrate it.
Use case is, I want feed the output of intermediate tensor as an input to my python code and then python code output to another tensor.
Does @tf.function can be use here?
Thanks.


texts: Keras models break when I add batch normalization

texts:
I'm creating the model for a DDPG agent (keras-rl version) but i'm having some trouble with errors whenever I try adding in batch normalization in the first of two networks.

Here is the creation function as i'd like it to be:

def buildDDPGNets(actNum, obsSpace):
    actorObsInput = Input(shape = (1,) + obsSpace, name = "actor_obs_input")
    a = Flatten()(actorObsInput)
    a = Dense(600, use_bias = False)(a)
    a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(300, use_bias = False)(a)
    a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(actNum)(a)
    a = Activation("tanh")(a)   # Bipdeal walker applies torque (-1, 1).
    actor = Model(inputs = [actorObsInput], outputs = a)
    criticActInput = Input(shape = (actNum,), name = "critic_action_input")
    criticObsInput = Input(shape = (1,) + obsSpace, name = "critic_obs_input")
    c = Flatten()(criticObsInput)
    c = Dense(600, use_bias = False)(c)
    c = BatchNormalization()(c)
    c = Activation("relu")(c)
    c = Concatenate()([c, criticActInput])
    c = Dense(300, use_bias = False)(c)
    c = BatchNormalization()(c)
    c = Activation("relu")(c)
    c = Dense(1)(c)
    c = Activation("linear")(c)
    critic = Model(inputs = [criticActInput, criticObsInput], outputs = c)
    return (criticActInput, actor, critic)

This causes me to get the following error:

    InvalidArgumentError: You must feed a value for placeholder tensor 'actor_obs_input' with dtype float and shape [?,1,24]
     [[{{node actor_obs_input}}]]

However, If I remove the batch normalization from the first network (a) and do not change the second network (c) at all, It runs as it should.

    a = Flatten()(actorObsInput)
    a = Dense(600, use_bias = False)(a)
    #a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(300, use_bias = False)(a)
    #a = BatchNormalization()(a)
    a = Activation("relu")(a)
    a = Dense(actNum)(a)

Its also notable that if I do it the other way around (remove bn from c and leave it in a) the error still occurs.
Any idea as to why that's happening?
Its odd because it runs fine with batch norm in the critic, just not the actor.
The models are being used by keras-rl DDPG agent btw.

Update:
I've tried rewriting it with the sequential object instead of the functional api. Didn't help. Still got the error with no change. I'm beginning to think this is some sort of problem with keras's batch normalize class when being applied to systems of multiple models.


texts: Classification for 'not something' Neural Networks

texts:
This is a very basic question about neural networks in general.

How do you have a classification for 'not something'?

Example: Say that you have to determine the accuracy of a certain simulation by comparing it to the actual scenario (in this case, comparing simulated and actual galaxy images). Theoretically, I can make a CNN and classify these images as 'galaxies' and 'not galaxies'. But what kind of images do I need to use for 'not galaxies'? I only have images for true galaxies.


texts: Trying to predict extreme values corresponding to rare events

texts:
I need some advice on methodology.

I need to predict a numeric value (claim amount) being as good as possible on high values corresponding to rare events (corporal damage, technological disaster...).

Any suggestion on methodology would be appreciated. Thanks.


texts: What is the time complexity of learninig phase of Hopfield neural network?

texts:
It really intrigues me what is the time complexity of learning phase of Hopfield neural network, how it depends of the number of training examples and number of attributes?
Source code of Hopfield neural network
Can we say that time complexity of Hopfield neural network quadratically depend on the number of training examples or quadratically depend on the number of attributes?
Thank you!


texts: Minimize correlation between input and output of black box system

texts:
I am not sure if "minimize correlation" is the right title for this issue but I could not find a better sentence to describe what I would like to achieve.

Let's say that I have a black box with multiple inputs and a single output. I know one of the inputs and the output and I have multiple example recordings of both. This known input modifies the output in a way that it is not desired, therefore, I would like to get rid of this "noise" caused by the known input. The transfer function for this input can be safely assumed as linear.

What I am doing right now, it is to loop through the example recordings, creating a linear regression model to predict the unwanted outcome and subtracting it from the real measured output signal, for each example. Afterwards, I compute the average of all the fixed output signals to reveal meaningful data beyond noise.

This strategy seems to work according to the following plot:

X axis is the known input signal, Y axis is the output signal, blue and green dots represent the averaged data before and after applying the linear regression algorithm, respectively. Lines are the best fit for each data set.

You can see that the green line ("cleaned" dataset) has the smallest slope, meaning that the output variable is considerably less linearly correlated with the input than it was previously. Therefore, I assume that the regression technique explained before is working as expected.

My question, looking at the plot, is there any mathematical procedure to directly "project" the original dataset in a way that the correlation between the input and output variables is minimized? Is there any math trick to avoid the use of the regression technique on all the example datasets to obtain a similar result?

My written expression is not the best so please feel free to comment the question if you need further explanations.

Any code is welcomed but python (pandas, numpy, etc.) and Matlab are preferred. Theoretical explanations are also very welcomed.


texts: Parking Prediction based on Mobile application

texts:
I am quite new to predictive modelling but have knowledge of GIS, R, python, SQL, etc.

I am currently doing a project in work trying to predict when parking spaces will be available based on data received from a mobile phone application.

I have 2 sql tables

ParkingTickets

ParkingTicketId  (Integer)
ParkingAreaId    (Integer)
Latitude (Integer)
Longitude    (Integer)
Location (Text)
ParkingDate  (DateTime)
DurationInMinutes    (Integer)
ExpiryDate   (DateTime)
Day (Text)

ParkingAreas:

ParkingAreaId (Integer)
MaxSpaces (Integer)

Main assumption is that only app users can park in the 18 parking lots (both on-street and off-street lots (no multi store or underground)). I do not take in to account user preference, weather, events, etc., etc. This is purely presence and absence based on data at hand. I have researched techniques and was looking into using a birth/death model but struggling to find examples of it in use.

Any help or pointers on models to use would be brilliant!

sample data:

 ParkingTicketId    ParkingAreaId   Latitude    Longitude   Date    DurationInMinutes   ExpiryDate  Day
60465   302 42.56246869 -70.91313754    2014/03/07 16:36    5   2014/03/07 16:41    Friday
60466   302 42.57139883 -70.91906364    2014/03/07 16:36    23  2014/03/07 16:59    Friday
60467   302 42.54419925 -70.9417496 2014/03/07 16:36    24  2014/03/07 17:00    Friday
60472   302 42.57576595 -70.92876607    2014/03/07 16:36    16  2014/03/07 16:52    Friday
60477   302 42.55573294 -70.92912634    2014/03/07 16:36    9   2014/03/07 16:45    Friday
60479   302 42.55711998 -70.91200458    2014/03/07 16:36    19  2014/03/07 16:55    Friday
60480   302 42.58008043 -70.91559081    2014/03/07 16:37    5   2014/03/07 16:42    Friday
60485   302 42.55161223 -70.9240808 2014/03/07 16:37    21  2014/03/07 16:58    Friday
60492   302 42.58437849 -70.92764527    2014/03/07 16:37    6   2014/03/07 16:43    Friday

ParkingAreaId   MaxSpaces
302 8
304 50
306 95
308 30


texts:
I tried the sketch engine (no ad!) and I wonder what might be the underlying algorithms to do such a good keyword extraction. I have a document consitsing of rows of sentences and from each I want the proper NERs or keywords. I tried a lot. I need them to write my own NER based on CRF.

Is this just statistics used to extract the keywords or what else do you need for that? I ask in general and not how they did it, but it would be nice to find out if someone knows...


texts: k-means: Only one-dimensional cluster predictions in two-dimensional space

texts:
For this dataset, it seems that the predictions of my k-means model only consider the horizontal axis, although the cluster centers seem reasonable.

Is something wrong with this classification? Please note the color of the grid in the background.

I use scikit-learn, here is the code fragment of classification and visualization.

model = KMeans(n_clusters = 5)
model.fit(df_stuff[['Stuff','Other Stuff']])

fig = plt.figure()
ax = fig.add_axes([0,0,1,1])

ax.scatter(df_stuff['Stuff'], df_stuff['Other Stuff'],c=model.labels_,s=80,cmap='rainbow')
ax.set_xlabel('Stuff')
ax.set_ylabel('Other Stuff')
ax.set_title('Strange Clusters')

# Draw Cluster Centers
for center in model.cluster_centers_:
    ax.scatter(center[0],center[1],c='black',s=5120,alpha=0.2)

# Draw Cluster Grid
cluster_grid = {'x': [], 'y': [], 'cluster': []}
for x in np.linspace(df_stuff['Stuff'].min(),df_stuff['Stuff'].max(),25):
    for y in np.linspace(0.35,0.6,25):
        cluster_grid['x'].append(x)
        cluster_grid['y'].append(y)
        cluster_grid['cluster'].append(model.predict([[x,y]])[0])

ax.scatter(cluster_grid['x'],cluster_grid['y'],c=cluster_grid['cluster'],cmap='rainbow',alpha=0.4,s=10)


texts: How to optimize AUPRC for imbalanced data given a precision or recall bias?

texts:
My general understanding is that when optimizing a model in an imbalanced class case with a small preferred target class one should optimize first for a model with the best AUPRC (assuming one doesn't have a defined cost matrix and only knows that they care about precision and recall). Then one can do a threshold search on that model to tune to a desired balance of precision OR recall (depending on need).
However...
Say I know that I have a slight bias towards precision. I then worry because I believe I could get two equivalent AUPRC scores that represent very different PR curves. Consider the case of the two theoretical PR curves below (red and blue):

Perhaps this specific example doesn't exist but I would think something analogous does. The point being, it seems to me that both would yield the similar AUC, but they behave quite differently. In fact if I really really cared about precision, blue might be a bad choice.
Is my worry here valid, and if so, what is the best way to handle this if I do have a preferential precision or recall bias?
Of course I could then threshold search for an f-beta score, but that can be a huge amount of time when trying out many models and hyperparameters. I could of course actually graph the plots also, but I have similar reservations.


texts:
I have few key words say - RAM, Speaker, Brand, Display etc. I have made a dictionary of all possible values these keys can have. Like - RAM can have 12, 12gb, 12GB; Display can have 12inch, 12", 12 inch.

I am given a title say - "Apple iPhone 5s (Space Grey, 16GB)"

I have to extract tokens from these titles and map to the dictionary values and assign appropriate keyword to each token.

Like for this - Brand - Apple, Color - Grey, Storage - 16GB

How should i tokenize these sentences? Doing it just by space will not be enough- like in this title - "Samsung Guru Music 2 SM-B310E (White)" 2 will be one token and can map to RAM,Display anything.

Will any NLP library help with this? I am using python to code and new to NLP.


texts: Excel error may be caused by pandas writing or big data? advise needed

texts:
I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do:

open an xml file
extract some elements
create a pandas dataframe with the extracted elements
append the results in the excel file named "output.xlsx"(using the code below in python)

My steps are repeated for all xml files that i have (15gb of initial data that usually have 100MB of valuable text data)
This is my python code for appending data frames in the output excel file:
book = load_workbook('output.xlsx')
writer = pd.ExcelWriter('output.xlsx', engine='openpyxl')
writer.book = book
writer.sheets = {ws.title: ws for ws in book.worksheets}
startrow = writer.sheets['Sheet1'].max_row
output.to_excel(writer, startrow=startrow,index = False, header = False)
writer.save()

When I open the "output.xlsx" in Excel, I receive a prompt message saying "We found a problem with some content in "output.xlsx". Do you want us to try to recover as much as we can?" with a yes or no answer
This is the log file that excel generates:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<recoveryLog xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
<logFileName>error001280_01.xml</logFileName><summary>Errors were detected in
 file 'D:\JUPYWORKDIR\2009Results\output.xlsx'</summary><repairedRecords>
<repairedRecord>Repaired Records: String properties from /xl/worksheets/sheet1.xml part
</repairedRecord></repairedRecords></recoveryLog>

I am worried that saving my results to excel format is corrupting my data, i will read "output.xlsx" with pandas in future in order to do some data analysis, does this problem effect my future analysis? I wanted to know why this problem is generated and should I save my data in CSV? any suggestions?
Ps. Checking the last row of "output.xlsx" using python code it is the same number of rows when i import the excel file in a pandas Dataframe, lastly checking the last row of the "recovered file" of Microsoft excel i still find the same number of rows so i think its a generic error of Microsoft excel because of large data but i am not sure


texts: Understanding scipy.signal.convolve2d full convolution and backpropagation between convolutional layers

texts:
I'm learning about convolutional neural networks. The convolution operation in order to extract features that is described in literature and posts used for this is quite intuitive and easy to understand (shown by the next gif), and even trivial to implement in a naive way:

I'm using scipy.signal.convolve2d instead of my own implementation for performance reasons. Checking the documentation, it mentions three different modes: full, valid and same. I've figured out, just by comparing results and shapes, that the valid mode corresponds to the operation the previous gif shows and that every place refers as "convolution". Also, that same is just the same, but with zero padding, i.e.:
In [1]: import numpy as np
In [2]: from scipy.signal import convolve2d

In [3]: image = np.random.rand(8, 8)
In [4]: padded_image = np.pad(image, (2, 2))
In [5]: kernel = np.random.rand(5, 5)
In [6]: assert (convolve2d(image, kernel, mode="same") == convolve2d(padded_image, kernel, mode="valid")).all()

The following gif would explain the operation in the same mode:

What I cannot find even an intuition for is the full mode (which the documentation refers as The output is the full discrete linear convolution of the inputs.). Using the same image and kernel, the operation gives:
In [7]: convolve2d(image, kernel).shape
Out[7]: (12, 12)

It's hard to get an understanding or juts an intuition by the result, and just by the description of the mode parameter and looking for literature about convolution operation. In the field of CNNs, the convolution is always explained as an operation to "reduce" the dimensions of an input image in order to extract its features.
This question comes from the need of having to propagate back the error delta between two convolutional layers. I need to calculate the error delta in layer l, so for that I'm performing a convolution between the kernel and the error delta of the l+1 layer (having then to multiply the gradients of layer l), and performing a valid convolution gives obviously an error delta whose dimensions don't match with the l layer activation, but it does doing a full convolution (figured out just by chance). I want to know if this is the correct way of propagating the error back.


texts: Given a machine learning algorithm, what is the minimum size of the training set for it?

texts:
I understand that the more data we have, the more reliable is our model trained on that data. I also understand that the more parameters a machine learning model has, the more training data it requires. For example, deep neural networks need more data than linear regression or support vector machines.

Successful application of machine learning to texts and images requires millions of data points. I read this attempt (link) with a data set of 5000 quotes of thinkers to generate new quotes. The author had several failed attempts until he understood that his data set is too small for the task. He succeeded only by using transfer learning employing an AWD-LSTM model developed by Salesforce trained on 100 million tokens from Wikipedia.

On the other hand, I often see scientific articles that train machine learning models on just a few hundreds of data (or even less) and boast about their "great discoveries". For example, this paper (link) uses a data set of only 46 samples for which they use decision trees. This situation exists because it is an extremely rare in science that one can make millions of experiments with millions of materials to get large training data sets. Yet, machine learning is fashionable, so many research groups apply it to their tiny datasets just to get published. How reliable are their predictions given the size of their data sets?

Is there a formula, or an algorithm where I can insert the number of my training data, the number of parameters of my model, and maybe few other numbers that distinguish between the models (e.g. feed-forward neural networks, SVM, kernel ridge regression, decision trees) and this formula show that my training data size is sufficient or insufficient for a chosen method?

Is there statistical research that shows quantitatively that a certain machine learning model requires at least this amount of data?


texts:
Has anyone created statistics on how fast and accurate Inception V3 can classify an image based on criteria such as: different models of GPUs/CPUs, input image size, input image ratio, file format, how it runs on iOS and Android, etc?

Example chart I'd hope to see:

GPU | CPU | Ratio |   Size   |  Speed  | Accuracy
 x           1:1     100x100    1000ms     93%
       x     3:4     3k x 4k   40000ms     95%

X as in type of GPU/ CPU


texts:
My question is: how can I mask or crop the results, using R, of IDW interpolation to only the area containing the original set of data points?

In the example below, 20 random points are used to interpolate a surface using the IDW function in gstat. A convex hull is obtained from the points set and plotted on the interpolated map. But, I would like to crop the map so only areas within the point cloud show the interpolation result.

Thanks for any suggestions!

###
#                             IDW in R
#
library(gstat)
#
set.seed(1234)
x <- rnorm(20)+10
y <- rnorm(20)+10
z <- rnorm(20)+10
#
xyz <- data.frame(x,y,z)
coordinates(xyz) <- ~x+y
xr <- range(x)
yr <- range(y)
b <- round((xr[2]-xr[1])/100,2)
#
grd <- expand.grid(x=seq(from=xr[1],to=xr[2],by=b),
y=seq(from=yr[1],to=yr[2],by=b))
#
coordinates(grd) <- ~ x+y
gridded(grd) <- TRUE
#
idw<-idw(formula=z~1, locations=xyz, newdata=grd)
#
image(idw, col=terrain.colors(16))
contour(idw, add=T)
points(x,y, col=4, pch=19)
#
library(spatstat)
conv<-convexhull.xy(x,y)
plot(conv,add=TRUE)
#


texts: What's the order in applying SMOTE transformation in a pipeline?

texts:
Here's the thing, I have an imbalanced data and I was thinking about using SMOTE transformation. However, when doing that using a sklearn pipeline, I get an error because of missing values.
This is my code:
from sklearn.pipeline import Pipeline

# SELECAO DE VARIAVEIS
categorical_features = [
    "MARRIED",
    "RACE"
]

continuous_features = [
    "AGE",
    "SALARY"
]

features = [
    "MARRIED",
    "RACE",
    "AGE",
    "SALARY"
]

# PIPELINE
continuous_transformer = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("scaler", StandardScaler()),
    ]
)

categorical_transformer = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="median")),
        ("onehot", OneHotEncoder(handle_unknown="ignore")),
    ]
)

preprocessor = ColumnTransformer(
    transformers=[
        ("num", continuous_transformer, continuous_features),
        ("cat", categorical_transformer, categorical_features),
    ]
)

pipeline = Pipeline(
    steps=[("preprocessor", preprocessor), ("classifier", LogisticRegression())]
)

X = df[features]
y = df[['binary_response']]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=0.8, random_state=42
)

X_train_smote, y_train_smote = oversample.fit_resample(X_train, y_train)

pipeline.fit(X_train_smote, y_train_smote)

That doesn't work because I have missing data. But I'm not sure what to do because of the pipeline and the order I should use.
Any thoughts on that?


texts:
Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions,
I want to know how can we take the derivative of $f \text{ w.r.t. } X$? I understand differentiation w.r.t one variable, but how does differentiation of another vector/matrix w.r.t a vector work?

Differentiating a function w.r.t a variable gives us the rate at with the function changes when we change the variable by a small amount. What does differentiating w.r.t a vector signify?


texts: Passing data to RNN with Sliding window approach

texts:
I having hard time with LSTM's and RNN so my apologies if this question sounds like a very basic question. I would appreciate if you can help in any way.
I am trying to train my RNN with LSTMs, but I am not sure on how to setup my model and data pipeline.
My data looks like below. It has 2 features for the input and I am predicting single label from multiple classes
Input =
array([[A, 0.250],
       [B, 0.654],
       [C, 0.643],
       [D, 0.859],
       [E, 0.135],
       [F, 0.913],
       [G, 0.493],
       [H, 0.267]])

Output =
array([[2],
       [3],
       [4],
       [4],
       [0],
       [1],
       [1],
       [2]])

I do have many samples for my data. For sake of simplicity here let's say I have 100 sample. So my input and output shapes are like this  input_shape(100,8,2) output_shape=(100,8,1) I hope its clear that my RNN time steps are 8.
Now based on this Sliding window leads to overfitting in LSTM?, I have concluded that I need a sliding window
approach to split my data so that it can learn all the relatioship between different time steps, since that's the key for my data.
Let's say my sliding window is 6 then my shapes become like this  input_shape(100,4,8,2) and output_shape(100,4,8,1)

How do I split my input and output data in a way to get the sliding window of 6 and in a way that I get the above shape ?

Now I have RNN defined like like this
model = Sequential()
model.add(Bidirectional(LSTM(units=128), , return_sequences=True, input_shape=(8,2))
model.add(Dense(8, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

How will pass these input and outputs to my RNN ?


texts: How could I implement POS Tagger using HMM with Viterbi Algorithm in Python(IPython)?

texts: I was trying to implement a POS tagger for resource-scarce language. For the implementation, I've my own manually tagged corpus in a quite small amount. Having this corpus, I've been dealing with the development of PoS Tagger in supervised mode on the Jupyter environment. Over the course of the attempt, I have challenged by applying the performance optimization algorithms(Viterbi) as I was naive to python merely but not to others. Thus I need good provisions from this community if one who could provide me with the simplest way of applying this algorithm along with HMM. I am just worried!


texts: I have a corpus on which I want to perform sentiment analysis using LSTM and word embeddings. I have converted the words in the documents to word vectors using Word2Vec. My question is how to input these word vectors as input to Keras? I don't want to use the embeddings provided by Keras.


texts: Image classification: Predict all objects' classes without bounding boxes

texts:
I'm trying to identify the problem here: I want to input an image and output the classes associated with all the objects in that image, not necessarily predict bounding boxes.

I have been looking for the solution but it seems like image classification won't work with images that contain more than one object. Otherwise it is the object detection problem, which is unnecessary because the output does not require bounding boxes.

Could you let me know what type of problem it is?


texts: I'm worried that I'm training my model wrong

texts:
So I'm trying to classify some fashion mnist like photos into either boots or sneakers. I'm using a perception from sklearn to do so. The data set is a CSV containing pixel values. The model is trained through cross validation. Here is the code:
def perceptron_train(ds,target,data,splits):
    clf = linear_model.Perceptron()
    kf = model_selection.KFold(n_splits=splits,shuffle=True)
    X = data
    y = target

    accuracy_model = []
    training_times = []

    for train_index, test_index in kf.split(X):
        X_train, X_test = X.iloc[train_index],X.iloc[test_index]
        y_train, y_test = y.iloc[train_index], y.iloc[test_index]
        train_start = time.time()#time training time
        model = clf.fit(X_train,y_train)
        train_stop = time.time()
        overall_train_time = train_stop-train_start
        training_times.append(overall_train_time)

        #test_start = time.time()
        accuracy_model.append(metrics.accuracy_score(y_test,model.predict(X_test),normalize=True)*100)
        #test_stop = time.time()
    print(np.mean(accuracy_model))
    print(np.mean(training_times))

What I'm worried about in particular is this line:
X_train, X_test = X.iloc[train_index],X.iloc[test_index]
y_train, y_test = y.iloc[train_index], y.iloc[test_index]

The accuracy is very high and I'm worried that I'm either overfitting or I'm training on the wrong part of the dataset. Sorry, I'm just learning cross-validation and just looking for some confirmation that this is not wrong.
Thank you


texts: How to calculate and print unseen time series values for LSTM after train, valid and test data

texts:
My dataset is time series data. It contains data of 10 packages. trainX = (1410,3,15), trainY= (1410,1), testX=(110,3,15), testY=(110,1). Input is three feature vectors of last three months and i am trying to predict vulnerability of 4th month. one month feature vector has 15 features. I have implemented statefull LSTM. I am training in batches by putting data of one package in one batch.

n_batch=141
# create and fit the LSTM network
model = Sequential()
model.add(LSTM(45, batch_input_shape=(n_batch, 3,15), stateful=True,
return_sequences=True))
model.add(LSTM(45, batch_input_shape=(n_batch, 3,15), stateful=True,
return_sequences=True))
model.add(LSTM(45, batch_input_shape=(n_batch, 3,15),stateful=True))
model.add(Dense(1, activation="relu"))
rmsprop = opt.RMSprop(lr=0.001)
model.compile(loss="mse", optimizer= rmsprop,metrics=['mse'])
model.summary()

for i in range(1500):
history=model.fit(trainX12, trainY12, epochs=1,
batch_size=n_batch,validation_split=0.1,verbose=2, shuffle=False)
model.reset_states()

#creating new model for test data
n_batch=11

new_model = Sequential()
new_model.add(LSTM(45, batch_input_shape=(n_batch, 3, 15), stateful=True,
return_sequences=True))
new_model.add(LSTM(45, batch_input_shape=(n_batch, 3, 15), stateful=True,
return_sequences=True))
new_model.add(LSTM(45, batch_input_shape=(n_batch, 3, 15),stateful=True))
new_model.add(Dense(1, activation="relu"))
rmsprop = opt.RMSprop(lr=0.001)
new_model.compile(loss="mse", optimizer= rmsprop)
new_model.summary()

#setting weights
old_weights = model.get_weights()
new_model.set_weights(old_weights)

#predicting from the predictions themselves (gets the training data as input
to set states)
new_model.reset_states()
#Prediction
testPredict= new_model.predict(testX12,batch_size=n_batch)
#printing history keys
print(history.history.keys())

#training and test loss
print(history.history['loss'])
history.history['val_loss']

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

Isuess: 1. when i am trying to plot the history['loss'] and history['val_loss'],it is plotting nothing. When i am printing both then it is showing only one value. How can i plot all the training and testing loss values?
2. How can i print and calculate the values for future in time of unseen data after train,test and predict. Like what wil be the value of vulnerability in for this package in next 6 months or 9 months. How to write that function. Can somebody do this magic for me. I wil be really thankful.


texts:
I am facing some issues to understand how cross_validation function works in fbprophet packages.

I have a time series of 68 days (only business days) grouped by 15min and a certain metric :

00:00 5
00:15 2
00:30 10
etc
23:45 26

And I really don’t know how to set up my cross_validation function.

I have understand that initial parameter is the training data, but what are period and horizon ?

I have 6528 rows, I want to forecast the next day so my period is 96 ? Because one day = 96 step of 15 min.

Please can you explain me how I have to fill period and horizon parameters. The official documentation of Facebook Prophet is not very understandable.

Thanks a lot.


texts:
While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore:

What is a surrogate models?

Why are surrogate models good for explainability?


texts: Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do fillna.(alldata.mean()) or will that cause data leakage. What would be a better way because doing them separately would have different mean for same column.


texts: Pandas copy() different columns from different dataframes to a new dataframe

texts:
I have 2 dataframes that are coming from 2 different Excel files. I want to extract some columns from one file and other columns from the second file to print a new dataframe with the copied columns.
I copied 2 columns from different dataframes (df1 and df2) but I get print only one of them (the last one) in df3. How can get all of them in the df3?
    import pandas as pd

    df1 = pd.read_csv('ASX_public_DB.csv', sep=';', skiprows=5)
    df2 = pd.read_excel('ASX_delisted_6months.xlsx')

    df3 = df1[['ticker_symbol']].copy()
    df3 = df2[['ASX code']].copy()

    print(df3)


texts: Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?

texts:
I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an idea on what the different dimensions "mean" (their interpretation) based on their values for different words. For example, there's a column that's dark blue for every word except WATER, so that dimension may have something to do with the word representing a person.

Secondly, there's a famous example that "king" - "man" + "woman" ~= "queen", where the word in quotation means the embedding of that word.
My questions are:

I don't quite understand the mechanism as to how any dimension of an embedding goes on to have a tangible, interpretable meaning. I mean, the individual components of embedding vectors could've very well been completely arbitrary devoid of meaning, and the whole embedding approach still could've worked in that scenario, since we're interested in the vector as a whole. Is there an online explanation or a paper that I can look at to understand this phenomenon?
Why does this addition/subtraction of vectors to give the relevant embedding vector for "queen" work so nicely? In one source, the explanation is given as follows:

This works because the way that the neural network ended up learning about related frequencies of terms ended up getting encoded into the W2V matrix. Analogous relationships like the differences in relative occurrences of Man and Woman end up matching the relative occurrences of King and Queen in certain ways that the W2V captures.

This seems like a broad, vague kind of an explanation. Is there any online resource or paper that explains (or better yet, proves) why this property of embedding vectors should hold?


texts: How to configure null hypothesis or what's the null hypothesis when using sklearn?

texts:
I'm predicting how BMI, GDP, ... factors affect life expectancy.
Firstly, I tried to select topK features.
selectorK = SelectKBest(f_regression, k=5)
x_k = selectorK.fit_transform(norm_x, y)
f_score = selectorK.scores_
p_values = selectorK.pvalues_
... PRINT

Output
    features            F-score    p-values

   HealthcareExpenditure    9937.45   0.000000000000000000
   WorkingHours              625.52   0.000000000000000000

Ok, HealthcareExpenditure is having p-value 0, meaning I can reject the NULL hypothesis between HealthcareExpenditure and Life expectancy.
However, what's the definition of null hypo here?
Given HealthcareExpenditure(X) , Life expectancy(Y), what's the null hypo here? And how to see it visually?
I can imagine the relation between  HealthcareExpenditure and Life expectancy should be correlated.
And what's the null hypothesis(H0) here? In the below chart, the p value should be large, and we can not reject H0 , what is H0 here? So confusing.
.
Is that H0 always default to X!=Y in sklearn world?


texts: How can calculate Efficiency for predictive models based on accuracy or error over time?

texts:
I was wondering if I could express the efficiency of prognostic models according to their accuracy(error, e.g. MAPE or MSE) over time [sec]. So let's imagine I have the following results for different predictive models:

models MSE   MAE     MAPE       predicting Time[sec]
LSTM   0.12  0.13   15.67%          456789
GRU    0.06  0.05   5.89%           688741
RNN    0.45  0.51   25.33%          55555

What is the best way to illustrate the efficiency of predictive models over predicting time? Is the following equation right? how about its unit when we use MSE or MAE instead of MAPE which is expressed by %?

\begin{equation}
\text{Efficiency} = \frac{Accuracy}{predictingtime }= \frac{error (e.g.MAPE)}{predictingtime }= \frac{percentage}{sec}
\end{equation}
which graph demosntraete efficiency scientifically for prognestic models?

I read this article and the introduce following criteria under name of prediction efficiency PE. is it possible to use such that?


texts: What are some solutions for dealing with time series data that are recorded at uneven intervals?

texts:
Let's say I have a time series data which is a bunch of observations that occur at different time stamps and intervals. For example, my observations come from a camera located at a traffic intersection. It only records when something occurs, like a car passes, a pedestrian crosses, etc... But otherwise doesn't record information.
I want to produce a LTSM NN (or some other memory based NN for time series data), but since my features don't occur at even time intervals, I am not sure how having a memory would help. For example let's consider two sequences:
Scenario 1:

At 1PM, I recorded a car passing.
At 105, some people cross.
At 150 some more people cross.
At 2PM, another car passes.

Scenario 2:

At 1 PM a car passes
At 2 PM a car passes

In the first scenario, the last car passed 3 observations ago. In the second scenario, the last car passed 1 observation ago. Yet in both scenarios, the last car passed 1 hour ago. I am afraid that any model would treat the last car passing in scenario 1 as 4 time periods ago and the last car passing in scenario 2 as 1 time period ago, even though in reality, the time difference is the same. My hypothesis is that the time difference is a very important feature, probably more so than the intermediate activity between the two car passing. In other words, knowing that the last car passed 1 hour ago is equal or likely more important than knowing that there were some people crossing in the last hour. With that said, knowing that people crossed is important too so I can't just remove that feature.
Another example of my issue can be scene below:
Scenario 1

1PM Car passes
2PM Car passes

Scenario 2

1PM Car passes
10PM Car passes

Once again, in my data set, this would be treated as adjacent observations, but in reality, the time gap is vastly different and thus, the two scenarios should be viewed as very dissimilar.
What are some ways to solve these issues?

I've thought of just expanding the data set by creating a row for every possible time stamp, but I don't think this is the right choice as it would make my dataset humongous and most rows would have 0s across the board. I have observations that occur in microseconds so it would just become a very sparse dataset.
It would be nice to include time difference as a feature, but I am not sure if there's a way to include a dynamic feature in your data set. For example, in the first scenario, at 105, the 1PM observation needs a feature that says this occurred 5 minute ago. But at 150, that feature needs to be changed to this occurred 50 minutes ago, and then at 2PM, that feature needs to now say that it occurred 1 hour ago.
Would the solution to just give the NN the raw data and not worry about it? When building a NN on word prediction, I guess if you give the model enough data, it'll learn the language even if the relevant word happened 10 paragraphs ago...  However, I am not sure if there are enough examples of the exact same sequences (even with the amount of data) for it to obtain the predictability I want.

Any ideas on ways to solve this problem while keeping in mind that the goal is to build a NN? Another way to think about it is, the time when a data point occurred relative to when the prediction will be made, in my situation, is a crucial piece of information for prediction.


texts:
Problem statement :

We have documents with list of words in them.
Overall these documents are classified into 2 group (say, good quality vs bad)

docs -

doc1 = [w1,w2,w3,w4]
doc2 = [w4,w3,w3,w4]
doc3 = [w2,w4,w8,w1]
doc4 = [w5,w4,w0,w9]

doc group -

good_grp = [doc2, doc1]
bad_grp = [doc3, doc4]

Now we have to find out which words actually are important to make the document good vs bad ?

Idea 1:
Merge all words from documents that belong to document group 1 into single document say (good quality doc) and other one being (bad quality doc) and calculate tf-idf score per doc; but in this case we lose information of document level words and now just see document group level word importance.

doc1 = [w1,w2,w3,w4]
doc2 = [w4,w3,w3,w4]
doc3 = [w2,w4,w8,w1]
doc4 = [w5,w4,w0,w9]

good_grp = [w1,w2,w3,w4,w4,w3,w3,w4]
bad_grp = [w2,w4,w8,w1,w5,w4,w0,w9]

Can someone help me to direct to a better approach tf-idf or any other technique to solve this problem?


texts: Variable with extra small Pearson coefficient has bigger positive impact on ML model performance than variable with bigger Pearson

texts:
I made some machine learning models using Python sci-kit learn library and I found some strange situation for me regarding the real importance of some variables (features) to the ML model. I found that a variable that has a smaller Pearson coefficient has higher importance on the ML model (when excluding variable from the model using backward elimination principle) than variables which has higher Pearson.

Below I send the real results of three models where the first model includes all three variables and another two models exclude some variables (- means variable is excluded). I use Random Forest method.

Model Name MAE (Mean Absolute Error)

$ Model_{V1V2V3}=0.92$ $ Model_{V1V2} = 3.86$  $ Model_{V1-V3}= 2.96 $

$ Pearson_{V1}=0.99 $, $ Pearson_{V1}=0.82 $, $ Pearson_{V3}=0.02$

When I exclude variable which has no importance based on Pearson (0.02) and I got to model with better performance compared to a model which includes another variable ($V2$) which has far higher Pearson (0.82). Why? Please, help me to explain this situation. Is there any explanation in literature or similar case reported?


texts: How to get true positive, true negative, and false positive false negative rate correctly using Keras?

texts:
I'm using Keras to do binary classification, and the label was encoded using one-hot encoding. I tried to get the number of True Positives, True Negatives and false positives and false negatives using
    model.compile(loss=keras.losses.binary_crossentropy,
              # optimizer = keras.optimizers.Adam(learning_rate = lr_schedule),
              optimizer = keras.optimizers.Adam(learning_rate = lr_schedule),
              metrics = [
                         tf.keras.metrics.BinaryAccuracy(),
                         tf.keras.metrics.TruePositives(),
                         tf.keras.metrics.TrueNegatives(),
                         tf.keras.metrics.FalsePositives(),
                         tf.keras.metrics.FalseNegatives()
                         ]
              )

but the results seems wrong:
1/1 [==============================] - 0s 408us/step - loss: 0.2671 - binary_accuracy: 0.8800 - true_positives: 22.0000 - true_negatives: 22.0000 - false_positives: 3.0000 - false_negatives: 3.0000
    Epoch 12/20
    1/1 [==============================] - 0s 419us/step - loss: 0.2206 - binary_accuracy: 0.9200 - true_positives: 23.0000 - true_negatives: 23.0000 - false_positives: 2.0000 - false_negatives: 2.0000
    Epoch 13/20
    1/1 [==============================] - 0s 410us/step - loss: 0.2280 - binary_accuracy: 0.8800 - true_positives: 22.0000 - true_negatives: 22.0000 - false_positives: 3.0000 - false_negatives: 3.0000

The number of true_positives and true_negatives got exactly the same value, false_positives and false_negatives got the same value. There are 25 training samples.
My questions are:

Why is this wrong?
How to access the number of true_positives and true_negatives  false_positives and false_negatives correctly?

Thanks in advance


texts: Purpose of trace-decay parameter in eligibility traces

texts:
In TD/SARSA-lambda, eligibility traces are decayed after each step by multiplying by the discount rate and the trace-decay parameter.

I understand that:

The discount rate is used to reduce the value of future actions relative to a state.
An eligibility trace should decay over time, as it represents how "responsible" a state-action pair is for an obtained reward.

What I don't understand why the trace-decay parameter is required, given that multiplying by the discount rate after each step will cause it to decay over time anyways. What does the trace-decay parameter add to the algorithm?


texts: Early stopping on validation loss or on accuracy?

texts:
I am currently training a neural network and I cannot decide which to use to implement my Early Stopping criteria: validation loss or a metrics like accuracy/f1score/auc/whatever calculated on the validation set.

In my research, I came upon articles defending both standpoints. Keras seems to default to the validation loss but I have also come across convincing answers for the opposite approach (e.g. here).

Anyone has directions on when to use preferably the validation loss and when to use a specific metric?


texts: Multilabel Classification - Overfitting?

texts:
My task is the following:
To input drug combinations and output renal failure-related symptoms from the drug combinations.
Both the drug combinations and renal-failure related symptoms are represented as one-hot encoded (for example, someone getting symptom 1 and symptom 3 out of a total of 4 symptoms is represented as [1,0,1,0]).
So far, I have ran the data through the following models and they have produced this interesting graph. The left-hand graph depicts the training and validation loss of the models over epochs and the right-hand graph depicts the training and validation accuracy over epochs.

It is almost clear from the left hand graph that in the Baseline model there is overfitting occurring as the training loss is decreasing and the validation loss is increasing over epochs. However, the accuracy graph on the graph suggests that validation accuracy continues to improve, despite the increase in validation accuracy.
After adding dropout and L2 regularisation (Baseline Reg), the validation loss does not go up so much, which seems to resolve the overfitting issue, but the accuracy is very sporadic and is on average worse than the Baseline model.
The question is the following:
Is my intuition correct in that both model's results is showing overfitting and should I continue my effort to reduce this effect?
Thanks in advance!
Model Architectures are below:
Model: "baseline"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_6 (Dense)              (884, 800)                603200
_________________________________________________________________
dense_7 (Dense)              (884, 800)                640800
_________________________________________________________________
dense_8 (Dense)              (884, 4)                  3204
=================================================================
Total params: 1,247,204
Trainable params: 1,247,204
Non-trainable params: 0
_________________________________________________________________

Model: "baseline_reg"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense_9 (Dense)              (884, 800)                603200
_________________________________________________________________
dropout (Dropout)            (884, 800)                0
_________________________________________________________________
dense_10 (Dense)             (884, 800)                640800
_________________________________________________________________
dense_11 (Dense)             (884, 4)                  3204
=================================================================
Total params: 1,247,204
Trainable params: 1,247,204
Non-trainable params: 0
_________________________________________________________________


texts:
We are having this requirement of extracting information from a credit history document. Usually it is a PDF and a computer generated document.
Because these PDFs are generated by different sources, the layout of the document will be different for each source. The columnn header labels will also be different.
Presently, there are 4 sources which are generating this document, but going forward, it will be from many sources. From each of these documents, we will need to extract information such as lender name, lending amount, outstanding balance etc;

I need to know what are the steps and practical approach involved in extracting the data I want such as lender name, amount, balance etc;

Do we have an established Machine Learning / Deep Learning approach that can be implemented here? Just getting to know the basics of ML/DL, therefore need a direction please


texts: Calculate Information Value and Weight of Evidence in R

texts:
I want to ask a question on how to calculate WOE and IV from weighted good bad indicator in R.

I found the woe package in R, but it only calculate WOE and IV from non weighted good bad indicator and it seems the indicator must be on 0 and 1 only.

Does anyone knows how to manipulate the good bad indicator in this woe package or got information of WOE and IV calculation package with weighted indicator?


texts: How to implement handmade features in a Keras Sequential model?

texts:
I wanted to know how to implement my handmade features in my Keras model.
Below I have some code where the embedding_matrix is the Stanford Glove NLP dataset.

def BidLstm(maxlen, max_features, embed_size, embedding_matrix):
   inp = Input(shape=(maxlen, ))
   x = Embedding(max_features, embed_size, weights=[embedding_matrix],
                 trainable=False)(inp)
   x = Bidirectional(LSTM(300, return_sequences=True, dropout=0.25,
                          recurrent_dropout=0.25))(x)
   x = Attention(maxlen)(x)
   x = Dense(256, activation="relu")(x)
   x = Dropout(0.25)(x)
   x = Dense(6, activation="sigmoid")(x)
   model = Model(inputs=inp, outputs=x)

I want to implement some extra features (preferably numpy array of features) into my keras model.

Any suggestions?


texts: Sentence segmentation when tokens are annotated with offsets

texts:
I have a dataset of documents annotated with entities used to train a NER model. The entity label locations in the text are specified with index offsets. So a document (one sample) in the dataset looks like this:
doc = {
    "text": "Here comes some lengthy text with entities such as Barack Obama...",
    "entities": [
       {"label": "PER", "start": 100, "end": 105},
       {"label": "LOC", "start": 300, "end": 311},
       ...
    ]
}

Here are some statistics about the length of the text in a document:
mean length: 431 chars
max length: 1605 chars
min length: 141 chars

My problem is that I am currently treating the text in a document as one sample but I would like to do sentence segmentation on those texts, i.e. split the sentences. The desired outcome would be to split a sample like this
unsplit_doc = {
   "text": "This...some sentence. And here...another sentence",
   "entities": [
           {"label": "PER", "start": 100, "end": 105},
           {"label": "LOC", "start": 300, "end": 311},
           ...
   ]
}

into
doc_sen1 = {
    "text": "This...some sentence",
    "entities": [
       {"label": "PER", "start": 100, "end": 105},
       ...
    ]
}

doc_sen2 = {
    "text": "And here...another sentence",
    "entities": [
       {"label": "LOC", "start": 300, "end": 311},
        ...
    ]
}

However if I use sentence segmentation like the one in stanza I lose information about the offsets. I don't think it's as easy to just subtract the length of the previous sentence from the offsets in the current document as stanza also tokenizes the sentence (and expands tokens).
Does someone know of an implementation that solves this problem?
Thanks


texts: What does the long curve-shape t-SNE mean?

texts:
I use 1-D CNN input 1*512 size time series data which randomly fragment segment, the output will classify input into 10 classes. After training the CNN, I apply t-SNE to the prediction which I fed in testing data.
In general, the output shape of the tsne result is spherical(for example,applied on MNIST dataset). But now I apply t-SNE on my own dataset. No matter how I adjust perplexity early, learning rate or maximum number of iterations. It will give me the result of long-shaped output, just like the pic below. Does the long-shaped t-SNE mean anything? Thanks everyone beforehand.

Edit: Explaining about dataset contents. p.s. before feeding into CNN for training, I randomly split it into training/testing dataset.


texts: BERT : text classification and feature extractionn

texts:
I have tried multi-label text classification with BERT.

Here is the sample input: $15.00 hour, customer service, open to industries

One of the labels is Billing_rate and prediction score looks quite good.

Now my question is if I want to extract $15.00 hour basically feature value out of BERT. Can you please suggest what are my next step options?


texts: Suppose I want to apply cross validation without any inbuilt function. I have multi classifier problem which is perfectly  divided in n section where n is the number of target features. If I divide my training data sets in to k folds sets then I might end up having single target feature in one set. So to solve this problem I shuffle my training set and target and I am able to achieve 0.96 accuracy compared to 0.98 when I use inbuilt method in sklearn. Is there a better approach to divide your data perfectly in to k folds?


texts: How to determine the best number of the fully connected layers in CNN? Can I use only one fully connected layer in CNN? How to determine the dimension of the fully connected layer output?


texts: What is the difference between an autoencoder and an encoder-decoder?

texts: I want to know if there is a difference between an autoencoder and an encoder-decoder.


texts:
I hope all of you are doing well. I am a high school student, studying machine learning for my interest.I am studying the Backpropagation Algorithm in recent days. I got stuck in a problem:

After the forward pass,  the value of the terminating variable (green) is 1400 (I found it) but I can't find the value of the derivative at z, If backpropagation is run, beginning with the terminating variable in the graph.
To my knowledge, I have to set the weight and biases in such a way that the value of z will be minimum. I need to define a function through gradient ( mainly partial derivative).

Initial values of w and b are randomly chosen.
Epsilon (e) is the learning rate. It determines the gradient’s influence. w and b are matrix representations of the weights and biases. Derivative of C in w or b can be calculated using partial derivatives of C in the individual weights or biases.
The termination condition is met once the cost function is minimized.

Or, I will define some name to the intermediate node, Like w1,w2; then, I will get f(x,y) = a linear combination of W1,w2 with automatic differentiation.
I stuck on creating the function for it as It looks like a new graph rather than a neural network graph where the algorithm is used. If someone explains to me the way to approach with the solution, I hope I can solve this type of problem easily.
Thank you so much.


texts:
Recently I've been thinking about the proper use of encoding within cross-validation scheme. The customarily advised way of encoding features is:

Split the data into train and test (hold-out) set
Fit the encoder (either LabelEncoder or OneHotEncoder) on the train set
Transform both the train and test set using fitted encoder.

This way is claimed to prevent from any data-leakage. However, this seems to often be omitted during cross-validation. Let's suppose I am performing cross validation on the aforementioned train set. If I encode train set and then perform cross-validation it doesn't really mimic the steps above. Shouldn't the encoding be performed "within" cross-validation then? For example, assuming that we perform 5-fold cross-validation, shouldn't we fit the encoder on 4 folds and transform on 5th fold in each cross-validation step? I believe it is what's usually done in target encoding, but not really with label or one-hot encoding.
My questions therefore are:

Am I right about the necessity to fit the encoder on 4 folds and not on the 5th validation fold in each cross-validation step if we really want to prevent overfitting?
If not, why is it really necessary to perform all 3 steps mentioned before while dealing with train and test (hold-out) set?


texts: How to best use geographical information as a factor?

texts: I am trying to predict crime rates and I have naively used lat and long as two separate factors (which seem to work well!). Are there any best practices for location as a factor?


texts: Using regex in redshift to find dollar values

texts:
I have a field in a Redshift table that has user-generated text.  The field is where users can say how much they think something costs.

Ideally it'd just be a decimal, but it's varchar.  So users can type
"I think this is worth \$25", or "I'd pay 55" or "\$117".

So I'm trying to use regexp_substr to pull this out.  Specifically regexp_substr(f.comment_text, '\\$?[0-9]*').  But this doesn't work on a subset of entries for some reasons (eg Could do for $115).

If I remove the ? it works on that, but no longer on entries that don't use $.  Why?  And what should I use instead?


texts: What is the best way to predict time series data?

texts:
I have monthly price data for tomatoes for the last 9 yrs for a particular town and I'm looking to predict the prices of tomatoes 6 months into the future.

I had considered using Linear Regression in Tensorflow (something I learnt about a week ago), because there is a direct relationship between price and rainfall for the location in question. That is, high rainfall means a supply glut and a price drop, low rainfall means scarcity and price hikes.

However I've found that I can't get accurate weather forecasts for 6 months into the future and have to use some other way other than Linear Regression to predict future tomato prices.

What is the best way to do this?


texts: How to derive the sum-of squares error function formula?

texts:
I'm attending a Machine Learning course and I'm studying linear models for classification right now. Slides present approaches to learn linear discriminants (Least squares, Fisher's linear discriminant, Perceptron and SVM), more specifically, how to compute the weight matrix $\tilde{\textbf{W}}$ to determine the discriminant function:

\begin{equation}y = \tilde{\textbf{W}}^T \tilde{\textbf{x}} + w_0. \end{equation}

My problem is about least squares: I don't understand how the minimization of sum-of-squares error function:

\begin{equation}E(\tilde{\textbf{W}}) = \frac{1}{2} Tr\Bigl\{(\tilde{\textbf{X}}\tilde{\textbf{W}} - \textbf{T})^T(\tilde{\textbf{X}}\tilde{\textbf{W}} - \textbf{T})\Bigr\} \end{equation} (where $Tr$ is the trace).

is derived and how it is possible to reach the closed formula solution:

\begin{equation}\tilde{\textbf{W}} = (\tilde{\textbf{X}}^T\tilde{\textbf{X}})^{-1}\tilde{\textbf{X}}^T\textbf{T}\end{equation}

Can someone explain me the main steps in the simplest and clearest possible way to make sense of these formulas? I'm a beginner.

P.S. These formulas come from C.Bishop. Pattern Recognition and Machine Learning.


texts: activation functions in multiple layers in CNNs

texts: An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation function is needed to apply only once, on 1 final layer.


texts: my xgboost model accuracy decreases after grid search with

texts:
I tried grid search for hyperparameter tuning in XGBoost classifier but the best accuracy is less than the accuracy without any tuning

// this is the code before the grid search
xg_cl = xgb.XGBClassifier(objective='binary:logistic', seed = 22)
xg_cl.fit(x_train, y_train)
y_pred = xg_cl.predict(x_test)
print(metrics.confusion_matrix(y_test, y_pred))
print(metrics.accuracy_score(y_test, y_pred))
[[11  1]
[ 1 26]]
0.9487179487179487

also, worth mentioning that the dataset shape is (195, 11) after PCA, and i am trying to classify whether a patient has a parkinsons disease or not.

// this is the grid search code
clf_xgb = xgb.XGBClassifier(objective = 'binary:logistic')
params__grid = {
    'n_estimators' : range(50,150,10),
    'max_depth': range(2, 12),
    'colsample_bytree': np.arange(0.5,1,0.1),
    'reg_alpha' : np.arange(0,0.6,0.1),
    'reg_lambda' : np.arange(0,0.8,0.1)

}
search = GridSearchCV(estimator=clf_xgb, param_grid=params__grid, scoring = 'accuracy',
                            cv = 4 )
search.fit(x_train,y_train)

print('best score:/n',search.best_score_)
print('bestparams:/n' ,search.best_params_)

best score:
0.9038461538461539

best params:
{'colsample_bytree': 0.5,
     'max_depth': 7,
     'n_estimators': 50,
     'reg_alpha': 0.2,
     'reg_lambda': 0.1}

then I used these parameters to build and train a new classifier

clf_xgb_1 = xgb.XGBClassifier(objective = 'binary:logistic', max_depth = 7, n_estimators = 50, reg_alpha = 0.2 ,
                              reg_lambda = 0.1, colsample_bytree = 0.5 )
clf_xgb_1.fit(x_train,y_train)
y_pred_2 = clf_xgb_1.predict(x_test)
print('accuracy:/n', metrics.accuracy_score(y_test,y_pred_2))
print('confusion matrix:/n', metrics.confusion_matrix(y_test,y_pred_2))

accuracy:
0.8974358974358975

confusion matrix:
array([[ 9,  3],
       [ 1, 26]], dtype=int64)

How come my results are worse? I would expect that GridSearch would improve on the results.


texts: RESOURCE EXHAUSTED error with Kfold on a facial landmarks data

texts: i am currently working with helen dataset to predict facial landmarks. Currently running the program on a Geforce 1070. The highest batch size i can give is 230 and i am getting a loss of 3 and accuracy of 40%. i am unable to use KFold cross validation as it gives a RESOURCE EXHAUSTED error. Any suggestions.


texts: How many words should be taken as features in a ML problem?

texts:
I would like to ask you how many words should be taken as features in a ML program.
For example, if I have 30000 distinct words to make a vocabulary, what would a good number be?
I am currently removing stopwords, words with a few characters, numbers; applying lemmatisation; removing punctuation; analysing word frequency to keep the top 50 words.
However I do not know if out of 30000 distinct words, 50 would be too low.
What would be a good way to determine how many to keep or select?


texts: Text summarization with limited number of words

texts:
I am reviewing summarization techniques and haven't (yet) found an approach to limit the length of a summary. So for example a summarization function that gives me a summary that is < 500 words.

Can you point me in the right direction? Are there approaches/implementations out there that try to solve this challenge?

Appreciate your replies!


texts: Truncating float/doubles for reproducibility

texts:
I deploy machine learning models (typically GPU) to a variety of environments. I work sort of at the edge of ML R&D and devops, so I am really big into reproducibility, and one thing that drives me nuts is when models output similar but not byte-for-byte identical values, frustrating any hash-based automated testing. For example, here is a score from the same sample, inference model, code, container image, etc, but one is on a Titan, one is on an RTX 2080.

Titan X  = 0.9887396097183228
RTX 2080 = 0.9887396693229675

That's a relative error of 6.0e-08 or about 60 parts-per-billion. Obviously, this is the "same number", well within the weirdness of IEEE 754 and GPU processing.

I'd like to truncate the output of my algos, as this simplifies the job of automated testing, but the question then becomes, how to round to achieve this, and how much precision? Sounds simple, but take for example this note on numpy.around

  Results may also be surprising due to the inexact representation of
  decimal fractions in the IEEE floating point standard [R9] and errors
  introduced when scaling by powers of ten.

One part-per-million (for typically 0.0-1.0 confidence values) seems reasonable but I'm not sure if there are any subtle gotchas with this approach.


texts: grad-cam implementation on mobilenet SSD network

texts:
Below is a gradcam implementation for a standard image classifier :

from keras.applications.vgg16 import VGG16, preprocess_input, decode_predictions
from keras.preprocessing import image
import keras.backend as K
import numpy as np
import cv2
import sys
model = VGG16(weights="imagenet")
img_path = sys.argv[1]
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = model.predict(x)
class_idx = np.argmax(preds[0])
class_output = model.output[:, class_idx]
last_conv_layer = model.get_layer("block5_conv3")

grads = K.gradients(class_output, last_conv_layer.output)[0]
pooled_grads = K.mean(grads, axis=(0, 1, 2))
iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])
pooled_grads_value, conv_layer_output_value = iterate([x])
for i in range(512):
    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]

heatmap = np.mean(conv_layer_output_value, axis=-1)
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)

img = cv2.imread(img_path)
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
superimposed_img = cv2.addWeighted(img, 0.6, heatmap, 0.4, 0)
cv2.imshow("Original", img)
cv2.imshow("GradCam", superimposed_img)
cv2.waitKey(0)

I would like to implement a similar gradmap on my MobileNetSSD architecture (model is stored as caffemodel)

net = cv2.dnn.readNetFromCaffe('MobileNetSSD_deploy.prototxt.txt', 'MobileNetSSD_deploy.caffemodel')
image = cv2.imread(PATH_FOLDER + 'images/img_01.jpg')
(h, w) = image.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5)
net.setInput(blob)

The mobilenetSSD network may detect many classes in one image, so ideally I would like to build one heatmap per class.

Here is how I get the class_ids present in the image :

output= net.forward()

for i in np.arange(0, output.shape[2]):
    confidence = output[0, 0, i, 2]
    if confidence > 0.8:
        class_idx = int(output[0, 0, i, 1])
        ...

In order to build the gradmap, I am trying to retrieve class_idx (done),class_output and last_conv_layer

To get last_conv_layer_output, I did :

last_conv_layer_output = net.forward(outputName = 'conv17_2_mbox_conf')
class_output = net.forward(outputName = 'mbox_conf_softmax')[:,7]

grads = K.gradients(class_output , last_conv_layer_output)[0]

Problem here is that my gradient is None, meaning that either class_output or last_conv_layer_output (or both) are miscalculated. I am a bit lost on how to debug this?