texts: Convnets : do we have separate activation maps for images in a batch

texts:
I know that if we input an image of shape [6,128] to a convolutional layer with 5 filters each of shape[3,128] with S=1 and P=0 , then there will be 5 activation maps as output from the layer ..each map has a shape of [4,1]

But what about the number of maps outputted when we use batching ?

I mean if we are using a batch with size of 2 images (each of shape[6,128]) as input to a convolutional layer with 5 filters each of shape[3,128] with S=1 and P=0 also , then there will be 10 activation maps as output from the layer instead of 5 (each image in the batch has 5 maps) ?? or we will have also 5 maps but each of shape[2,4,1] ???

I think there will be 10 maps since if we apply then max pooling , we want to choose the max from each image independently , right? so each image must have separate maps from the maps of the other image..


texts: Compute angle of vector in word2vec models

texts:
If I understand correctly, the most_similar function computes the cosine similarity of the vector with all other vectors and finds the closest one.  The vectors then viewed from a certain origin of (0,0,0,..0) for N dimensions (for N dimensional word vectors).

Is there by any means that I could compute the angle of these vectors from the origin ?.
If I compute the similarity between the vector and the origin vector and take the inverse cosine, will the resultant angle be the angle of the vector on origin space.

If I get such an angle, on which plane does it lie as the vector is high dimensional.

The reason is that, when we try to reduce it to 2 dimensions and plot them, only magnitude of the vector is considered right, so the direction information is missing. I would like to plot it in a polar plot to view both.

Is there any such function in gensim?.


texts: How to structure and train an LSTM for real-time predictions in Keras?

texts:
I am trying to create an AI to play a 3D melee fighting game using an LSTM. The NN receives as input the relative positions of the enemies' joints (e.g. head, hands, legs) and should output the correct movements for each timestep. Currently, I am using a vanilla feed-forward NN made with Keras and it performs well, but it needs information from multiple timesteps to be more effective.
I am now trying to implement an LSTM for this problem. I want to receive an output at each timestep rather than feeding in a whole sequence of data before receiving output.
I believe that a stateful LSTM is suited for this task, but I am unsure of how to train it and when to reset the state. Ideally, the AI would have no fixed number of timesteps of memory and would just be able to continuously look back as it's making predictions in real time for each timestep, without having its state reset abruptly during fights.
My training data is one long, continuous sequence of enemy positions (input) and keystrokes (output) for each timestep. How would I structure and train an LSTM to learn from this data to make real time predictions at each timestep using Keras?


texts: Which time-serie model is good when using concatenate data?

texts:
I would like to make a model using sequence datasets.
As I've searched that , many LSTM, RNN tutorials use long-sequential data such as weather in 1950-2020. However, my datasets are short sequential data but concatenate data. I attached a figure.

The labelled datasets are collected from 100 students. It has ground truth and the W vector (W1-Wt) can be used to infer the ground truth. The t is less than 50 (short sequence).
After fitting a model, I will validated it by showing confusion matrix.
The fitted model is used for unlabeled data. This is collected from 10,000 anonymized people. I will use W vector (W1-Wt) of the 10,000 people for inferring.

In this case, which model is good for me? The LSTM, RNN or HMM still have benefit for me?


texts:
My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on how to monitor if it is still doing well:

  Suppose Amazon would use Reinforcement Learning for optimizing the order of a search:

  An episode starts when a user starts a search
  An episode ends after a threshold or when the user buys things.
  The observation the agent gets is the search terms as well as 20 products for which he has to give an ordering.
  He gets a reward if one of those 20 things is bought.

Of course, one can (should?) let the agent learn all the time as products change and probably search terms / language changes. But when do I know that the agent learned something weird / that I should stop it?

I can imagine the following:

Case-based single examples
Having a ground-truth for some trivial searches + products and checking if the model does those right (in non-exploratory mode)
Letting the agent learn in batch-mode (e.g. update the model only once a week) and doing A/B tests for the current model / new model.
Measuring mean reward and setting a threshold. If the mean reward of the agent drops below the threshold, reset the agent to a past "save" state.

Is there literature about it? Blog posts?

I know at least one example where RL went wrong / monitoring didn't quite work:

Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day (The Guardian)


texts: Digitize graphs. Pull out data from points in a scatter plot

texts:
I have a project where I would like to extract data from a series of scatter plots that are image files (Jpeg or png). The plots are similar but the axes scales are not always exactly the same. I have looked online and there are some web app solutions that are very “manual” and require a lot of clicking. This would get tiresome if there are a lot of graphs.
Does anyone know of any solutions that are partly or wholly automated? A python API solution would be ideal but other languages are also ok.


texts: Some robust perhaps more complex as neural networks to measure the distance between data?

texts: Now after certain preprocess we can repeatedly get two sets of data points, exhaustively enumerate one from the first set and one from the second set to form pairs. Each pair of these two data points have the same dimensionality, say two dimensional or one dimensional. The target is to measure the top n nearest ones among the distances of these pairs each time and perhaps finally the average of them. Euclidean distance is the simplest way to measure, but is there any more robust perhaps more complex way even applying neural networks to measure the distance between the pair? For instance, add a linear neural network layer after each data point, and measure the distance between the two outputs of the layer.


texts: Getting very low/ wrong accuracy from RandomizedSearchCV

texts:
I am currently using RandomizedSearchCV to optimize my hyper-parameters. However the reported scores of each iteration is very low. When I then evaluate the highest scoring candidate I get very high accuracy (0.97), while the RandomizedSearchCV reports something much lower (0.32).
search = clf_rand_search.fit(x_traintest, y_traintest)
print(search.score(x_validation,y_validation))
0.32

print(accuracy_score(y_validation.flatten(), search.predict(x_validation).flatten()))
0.9798260869565217

My input and output are both 2-D matrix with (100,9) and (100,230) shape for the train/test data. With lower samples for the validation data.
Should I format my data differently for the RandomizedSearchCV?
Input features first two are normalised and the last one-hot encoded.

Output classification 0 or 1 for 230 nodes.

clf = MLPClassifier(solver = 'adam',
                max_iter=9999,
                alpha=1e-5
                )

hidden_layers = 8
neurons = list(range(10,210,5))

m = [0]*(hidden_layers*len(neurons))

for i in range(1,hidden_layers+1):
    for idx,i2 in enumerate(neurons):
        m[((i-1)*len(neurons)) + (idx)] = [neurons[idx]]*i

param_space = {
    'hidden_layer_sizes':   m,
    'activation':           ['identity', 'logistic', 'tanh', 'relu'],
    'learning_rate':        ['constant','invscaling','adaptive'],
    'learning_rate_init':   np.arange(1e-4,0.1+1e-4,1e-4)
    }

clf_rand_search = RandomizedSearchCV(clf, param_space, n_iter=10,
                                        scoring="accuracy", verbose=True, cv=2,
                                        n_jobs=-1)


texts: How to aasign a new data point to a cluster?

texts:
I have a userdataset which contains fields like ['age','gender', 'computer_literacy', 'vision', 'colour_blind', 'education', 'font_size','colour'].
I clustered these data and assigned the new cluster to the existing data. Now I want to know how can I find a similar cluster for a new user?
As an example, If I am a new user I only submit few details like ['age', 'gender', 'computer_literacy', 'vision'] and I want to know, for which cluster I belong.
What are the possible approaches to solve this problem?


texts: Can someone explain the following error in my python code?

texts:
I am analyzing a dataset in Python for strictly learning purpose.
In the code below that I wrote, I am getting some errors which I cannot get rid off. Here is the code first:

plt.plot(decade_mean.index, decade_mean.values, 'o-',color='r',lw=3,label = 'Decade Average')
plt.scatter(movieDF.year, movieDF.rating, color='k', alpha = 0.3, lw=2)
plt.xlabel('Year')
plt.ylabel('Rating')
remove_border()

I am getting the following errors:

1. TypeError: 'str' object is not callable
2. NameError: name 'remove_border' is not defined

Also, the label='Decade Average' is not showing up in the plot.

What confuses me most is the fact that in a separate  code snippet for plots (see below), I didn't get the 1st error above, although remove_border was still a problem.

plt.hist(movieDF.rating, bins = 5, color = 'blue', alpha = 0.3)
plt.xlabel('Rating')

Any explanations of all or some of the errors would be greatly appreciated. Thanks

Following the comments, I am posting the data and the traceback below:
decade_mean is given below.

year
1970    8.925000
1980    8.650000
1990    8.615789
2000    8.378947
2010    8.233333
Name: rating, dtype: float64

traceback:

TypeError                                 Traceback (most recent call last)
<ipython-input-361-a6efc7e46c45> in <module>()
      1 plt.plot(decade_mean.index, decade_mean.values, 'o-',color='r',lw=3,label = 'Decade Average')
      2 plt.scatter(movieDF.year, movieDF.rating, color='k', alpha = 0.3, lw=2)
----> 3 plt.xlabel('Year')
      4 plt.ylabel('Rating')
      5 remove_border()

TypeError: 'str' object is not callable

I have solved remove_border problem. It was a stupid mistake I made. But I couldn't figure out the problem with the 'str'.


texts:
I am trying to implement a gradient descent algorithm for a simple linear function:
y(x) = x

Where initial hypothesis function is:
h(x) = 0.5 * x

and learning rate:
alpha = 0.1

Target function graph is blue and hypothesis is green.
Cost function:
J = 1/2m * sum[(h(x) - y(x)) * (h(x) - y(x))]

Gradient descent:
q = q - alpha/m * sum[(h(x) - y(x)) * x]

My implementation does not converge:
import numpy as np
import matplotlib.pyplot as plt

def y(x):
    return x

def get_h(q):
    """ Create hypothesis function

        Args:
            q - coefficient to multiply x with

        Returns:
            h(x) - hypothesis function
    """
    return lambda x: q*x

def j(x, y, h):
    """Calculte a single value of a cost function

        Args:
            x - target function argument values
            y - target function
            h - hypothesis function

        Returns:
            Value of a cost function for the given hypothesis function
    """
    m = len(x)
    return (1/(2*m)) * np.sum( np.power( (y(x) -h(x)),2 ) )

def df(h, y, xs):
    """Calculate gradient of a cost function

        Args:
            h - hypothesis function
            y - target function
            xs - x values

        Returns:
            differential of a cost function for a hypothesis with given q

    """
    df = np.sum((h(xs)-y(xs))*xs) / len(xs)
    return df

xs = np.array(range(100))
ys = y(xs)
hs = h(xs)

costs = []
qs = []
q = 0.5
alpha = 0.1
h = get_h(0.5) # initial hypothesis function
iters = 10
for i in range(iters):
    cost = j(xs,y,h)
    costs.append(cost)
    qs.append(q)
    print("q: {} --- cost: {}".format(q,cost))
    df_cost = df(h, y, xs)
    q = q - alpha * df_cost  # update coefficient
    h = get_h(q) # new hypothesis

What am I doing wrong? Should I account for q0 even if my target function intercept is zero?
Update
Answer: https://stats.stackexchange.com/questions/484750/linear-function-gradient-descent


texts: How do I group similar type of skills together?

texts:
Suppose that I have a file which has thousands of skills starting from A-Z. Now, I would like to create a model that can group similar skills together (example neural network and SVM can group together). I know that I can use NLP for this problem, but I'm not sure about the algorithm that I can use to get the best result.

I'm new to NLP so any help is greatly appreciated.


texts: Compare cross validation and test set results

texts:
I am having a hard time understanding the results of a cross validation test and a test run on a test set.
First I made the following pipeline:
pipe=Pipeline([('clf',DecisionTreeClassifier(random_state=0))])
Then I use cross validation on a scaled training set(75% of the original dataset):
>>> cross_val_score(pipe, X_train_scaled,Y_train,cv=7).mean()
0.7257796129913106
I then fit the pipeline with the training data and run the classifier on the training data.
>>> pipe.fit(X_train_scaled,Y_train)
>>> pipe.score(X_train_scaled,Y_train)
0.7734339749126584
Finally, I checked the models performance on the test set:
pipe.score(X_test_scaled, Y_test)
0.941353836876225
Question 1: have I done the right steps, do I even need to run the pipeline on the training data for the training data score?
Question 2: why is the test data so much more accurate than the cross validated one. Is the data underfitted, or is it okay for this to happen ?


texts: Updating 1st dataframe columns from 2nd data frame coulmns

texts:
I have two data frames df1 and df2 which look something like this.

    Name    Age    Sex
0   Sam      25     M
1   John     22     M
2   Larry    34     M
3   Mark     32     M

    Name    Age    Sex
0   Kelvin   51     M
1   Bob      26     M
2   Larry    37     M
3   Jerry    17     M
6   Nick     68     M
8   Sam      98     M
9   Jax      78     M
10  John     23     M
11  Mark     47     M
12  Jerry    68     M

My output should ideally be this:

    Name    Age    Sex
0   Sam      98     M
1   John     23     M
2   Larry    37     M
3   Mark     47     M

I have tried join and merge but the columns doesn't line up


texts: Fully-Connected DNN: Compute the numbers of free parameter in a DNN

texts:
A fully-connected DNN has layer sizes of 3-3-4-2, where the first layer size represents the input layer. We assume that all layers are affine ones (no ReLU). Give the dimensions of all weight matrices and all bias vectors in the network and compute the total number of free parameters in this DNN.
According to this task the first layer size represents the input layer, so it must be 3.
a(0) = x = 3

If I know the input layer, so 3-4-2 are also the sizes of the bias vectors.
I know now the dimension of the input layer and the bias vector. The columns of W must be also 3, because of the size of the input layer. The rows of W must be equal to the size of the bias vector.
So I computed the following dimensions:
W(1) = W33, b3, a(1) = 3

W(2) = W43, b4, a(2) = 4

W(3) = W24, b2, a(3) = 2

But how to compute the total number of free parameters in this DNN?


texts: Python, Logistic regression - How to calculate Nagelkerke pseudo r squared

texts:
I am doing logistic regression in sklearn and I would like to calculate (Nagelkerke) pseudo r squared, which makes more sense for logistic regression analysis. I don't see it is available in sklearn metrics. Does anyone have a python code, which calculates this, to share?

Some context:
https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/


texts:
I have used Neo4J to implement a content recommendation engine. I like Cypher, and find graph databases to be intuitive.

Looking at scaling to a larger data set, I am not confident No4J + Cypher will be performant. Spark has the GraphX project, which I have not used in the past.

Has anybody switched from Neo4J to Spark GraphX? Do the use cases overlap, aside from scalability? Or, does GraphX address a completely different problem set than Neo4J?


texts: Reward is converging but actions taken by trained agent are illogical in reinforcement learning

texts:
I am training a reinforcement learning agent using DQN. My state space has 6 variables and the agent can one action which is discretized into 500 actions

My reward structure looks like

thermal_coefficient = -0.1

        zone_temperature = output[6]

        if zone_temperature < self.temp_sp_min:
            temp_penalty = self.temp_sp_min - zone_temperature
        elif zone_temperature > self.temp_sp_max:
            temp_penalty = zone_temperature - self.temp_sp_max
        else :
            temp_penalty = 0

        reward = thermal_coefficient * temp_penalty

my temp_sp_min is 23.7 and temp_sp_max is 24.5. When i train the agent based on epsilon greedy action selection strategy, after around 10000 episodes my rewards are converging.  When I test the trained agent now, the actions taken by the agent doesn't make sense, meaning when zone_temperature is less than temp_sp_min it is taking an action, which further reduces zone_temperature.

I don't understand where am I going wrong. Can someone help me with this?

Thanks


texts: What does GlobalMaxPooling1D() do to output of LSTM unit in Keras?

texts:
The keras model looks like this

features_input = Input(shape=(features.shape[1],))
inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool,features_input])
outp = Dense(6, activation="sigmoid")(x)

What does GlobalMaxPooling1D()(x) really do to the output of LSTM?
I know the input to LSTM layer is of dimension (batch_size, steps, features).

Does GlobalMaxPooling1D take max across num_filters/hidden units of each LSTM unit?


texts: Implementing a model for a language to another

texts:
I have a dataset of sentences of language X and Y (2 columns, for example, "abc def lang" ==> "xyz pqrt mno uages"). I want to have a output as a table that translates word by word (abc => xyz, def => pqrt mno, lang => uages).
I think we should use deep learning but idk how to implement and what to use. Can you give me some ideas?
Thanks all


texts: Finding threshold value for right-skewed data (bivariate)

texts:
Intuition of the Problem
Suppose you have a dataset of two columns, X and Y, and I plot them using a bar plot. The bar plot shows that there are a lot of values in the first two bars on the left, and the Y value quickly goes to 0 as X is increased. Visually, one can look at the bar plots and tell which X value is the "cut-off" for which any value greater than that has low probability of having high bars. Can we do so mathematically?
Problem formulation attempt
With this right-tailed plot, I want to determine what is the value $x^{*}$ such that any $x > x^{*}$ has significantly low values of Y.
Tried approaches
I tried to fit a Poisson regression on the data using statsmodels package, but the Chi-squared estimate is 2x10^5, very insignificant. Same goes for a logistic regression. I am assuming that there is a lot of bias in these generalized linear models, as this type of univariate regression may not explain the response variable with just the single X variable I am looking at.
I then tried the Mahalanobis distance, looking at the set of $(x, y)$ pairs that are "outliers" to this dataset. However, this yields outlier values due to high $x$ and high $y$ values. Perhaps outliers may not be the appropriate approach and I should instead find the leverage?
Is there any thoughts on how I can frame the problem better and solve it with a graceful statistical approach? Thanks!


texts: Choosing the periodicity in a SARIMA model

texts:
Given the order (P,D,Q,s) of a SARIMA model, s is an integer representing the number of periods in a season. Intuitively, I suppose it would be 12 for monthly data and 4 for quarterly data.

But if I have hourly data (for a whole year) and I'm using only a small number of days as training data (say 10 or 14), what is a proper choice for s?


texts: why does tox21 loads as having 12 classes in pytorch geometric?

texts:
Why is the tox21 dataset loaded with predefined 12 classes when it should be binary classification, thus two classes?
Code sampled modified:
import rdkit
from torch_geometric.datasets import MoleculeNet

# Load the Tox21 dataset
data = MoleculeNet(root=".", name="Tox21")
print("Dataset type: ", type(data))
print("Dataset features: ", data.num_features)
print("Dataset target: ", data.num_classes)
print("Dataset length: ", data.len)

> Dataset type:  <class 'torch_geometric.datasets.molecule_net.MoleculeNet'>
> Dataset features:  9
> Dataset target:  12

I'm borrowing code from this post.
There are 12 assays, so one could train and test by assay, in which case we would end up with a Toxic|non-toxic (two classes) for each assay.
How do I setup the data class to work by group (e.g., by assay)?


texts:
In a regression model is there a need to scale/normalize a dependent/response variable when doing the same for independent variables? If not then why?
An example will be helpful.


texts:
My problem is that in PyTorch I cannot reproduce the MSE loss that I have achieved in Keras.

I have trained the following model in Keras:

from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(10))
model.add(Dense(1))
model.compile(optimizer = "adam", loss = "mean_squared_error")
model.fit(X_train, y_train,
      batch_size = 32,
      epochs = 200
     )

The shape of the training data is:

print(X_train.shape)
>>>(3550, 10)
print(y_train.shape)
>>>(3550,)

After training the MSE is ~0.15:

mse_train = model.evaluate(X_train, y_train)
>>>3550/3550 [==============================] - 0s 18us/step
print("Train MSE: ", mse_train)
>>>Train MSE:  0.1499910642017781

Then I initialize the same model in PyTorch:

import torch
import torch.nn as nn

class NN(nn.Module):
 def __init__(self):
  super(NN, self).__init__()
  self.dense1 = nn.Linear(10, 10)
  self.dense2 = nn.Linear(10, 1)

 def forward(self, x):
  out = self.dense1(x)
  out = self.dense2(out)
  return out

net = NN()
criterion = nn.MSELoss()

And assign the weights I have achieved in Keras:

from keras.models import load_model
keras_model = load_model(MODEL_PATH)

dense_weights = keras_model.layers[0].get_weights()
weights = torch.tensor(dense_weights[0].swapaxes(0,1))
bias = torch.tensor(dense_weights[1])
net.dense1.weight.data = weights
net.dense1.bias.data = bias

dense_weights = keras_model.layers[1].get_weights()
weights = torch.tensor(dense_weights[0].swapaxes(0,1))
bias = torch.tensor(dense_weights[1])
net.dense2.weight.data = weights
net.dense2.bias.data = bias

Now I try to calculate the MSE loss:

X_train_torch = torch.tensor(X_train, dtype=torch.float)
y_train_torch = torch.tensor(y_train, dtype=torch.float)

outputs = net(X_train_torch)
loss = criterion(outputs, y_train_torch)
print("Train loss: ", loss)
>>>Train loss:  0.338391376896338

The MSE is now ~0.34 and is twice as big as calculated in Keras.

What could be the reason? Is there a bug in my calculation?


texts: How to start with receipt OCR text detection

texts: I'm thinking about an OCR system for digitalizing receipts. On the input system would take a picture of receipt and then return classified data (total_sum = Y, date = X, etc.). My question is regarding how to start. My initial thought was that I should start with detecting classes (name of the shop, receipt id etc.) on image and splitting it, then I would send parts of image OCR. My second idea is more NLP based. I would normally pass an image to OCR system and then try to do some classification on text data. Which approach would be better?


texts: tensorflow: is there a way to specify XLA_GPU with tensorflow?

texts:
The following code is used to specify device on which tf node is running

with tf.device('/gpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

I have already known:
this post, tensorflow doc and xla demo

What i want to know is:

Is there any way to specify XLA_GPU as the device on which tf node is running?

with tf.device('/XLA_GPU:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

executing the code above gives ValueError: Unknown attribute: 'device' in '/job:localhost/replica:0/task:0/device:/XLA_GPU:0'

this is 100% reproducible on google colab.


texts: How to save prediction values for the whole data in Keras

texts:
I am using pre-trained VGG16 model to classify images located in the folder. Currently, I am able to classify only one single image.

How can I modify the code to classify all the images in the folder
How can I save the prediction values for each image ?

Below is my code :

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.applications.imagenet_utils import decode_predictions
import matplotlib.pyplot as plt

filename = 'cat.jpg'
# load an image in PIL format
original = load_img(filename, target_size=(224, 224))
print('PIL image size',original.size)
plt.imshow(original)
plt.show()

# convert the PIL image to a numpy array
# IN PIL - image is in (width, height, channel)
# In Numpy - image is in (height, width, channel)
numpy_image = img_to_array(original)
plt.imshow(np.uint8(numpy_image))
plt.show()
print('numpy array size',numpy_image.shape)

# Convert the image / images into batch format
# expand_dims will add an extra dimension to the data at a particular axis
# We want the input matrix to the network to be of the form (batchsize, height, width, channels)
# Thus we add the extra dimension to the axis 0.
image_batch = np.expand_dims(numpy_image, axis=0)
print('image batch size', image_batch.shape)
plt.imshow(np.uint8(image_batch[0]))

# prepare the image for the VGG model
processed_image = vgg16.preprocess_input(image_batch.copy())

# get the predicted probabilities for each class
predictions = vgg_model.predict(processed_image)
print (predictions)

# convert the probabilities to class labels
# We will get top 5 predictions which is the default
#label = decode_predictions(predictions)

Thank you


texts: How to convert subword PPL to word level PPL?

texts:
I'm using this formula to covert subword perpexity to word perplexity:
PPL_word = exp(log(PPL_subword) * num_subwords / num_words)
The question is do I need to include the [SEP] and [CLS] tokens when counting subwords?


texts: using movielens dataset build recommendation engine

texts: Where I can get the complete guide (step by step )on building a recommender system for example using movielens datsets building content based, collaborative or may be hybrid system.


texts: Text extraction / mining from specific templates (ML)

texts:
believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different.

Small context: just started studying TensorFlow, and I am new to the machine learning world, too...so sorry for any stupid thing I will write in this post.
I am studying the foundation of this subject, of course, but I would like to know if at the end I will be able to solve a specific scenario, and to obtain a good direction to follow.

Scenario: I have a lot of documents (paper documents converted to PDFs/images) that have the same structure...a template with few user-filled content.
Let's give an example (variables in bold):

  Police Department of New York [...] The driver John Doe,
  born on 2019-04-03,  owner of a some_car_model_here
  with the driver license XY123ZZ" etc...

So basically I need to extract the entities name, the birthdate, the car model, the driver license and so on. Additionally, there can be different names and different dates on the document, so the context is important. Example: I only need the name that comes after the words "The driver".

What I am doing now:

Extract the content of the document (in blocks) with Google Vision
Checking the first sentences of the document to understand which set of regex to apply (in this case, I should apply the "Police Department" of "New York" set of regexes)
Apply the regexes order to extract entities from this document

What I want to do: I was hoping to have a smartest way that increases the chance to correctly extract what I need, if I periodically add tons of documents to a hypothetical training set.
This is why I was thinking about "machine learning", and this is why I have identified TensorFlow as a possible library for it.

I have already seen that NLTK could be an option, but at the end (correct me if I am wrong) I should write my own rules, probably regex again, to extract some custom content.
I have read also about NER, but hardly thinking how it can help.

So basically what I am asking is: can I perform this task with Tensorflow (or a "machine learning"-related library)? Is there a specific subject/branch to look for? Do you have any other suggestion (if I am on the wrong track)?

Thanks,
  Lorenzo


texts:
In some resources, the belman equation is shown as below:
$v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big] $
The thing that I confused is that, the $\pi$  and $p$ parts at the right hand side.
Since the probability  part -  $p(s',r|s,a)$- means that the probability of being at next state ($s'$), and since being at next state ($s'$) has to be done via following a specific action, the $p$ part also includes the probability of taking the specific actions inside it.
But then, why the $\pi(a|s)$ is written at the beginning of the equation? Why do we need it? Isn't the possibility of taking an action stated at the $p(s',r|s,a)$ part already?


texts: How to train a deep neural network with time-series images and unbalanced dataset?

texts: I have images that represent a fixed-length time-window of different serials. Serials have time-series of different size, so e.g. serial1 has length 30, serial2 length 110 and so on. I have multiple images for every serial. Images are obtained with Grahamian Angular Field. My problem is that i have an unbalanced dataset: 1 serial of disk failed every 100 of disk. In Google Colaboratory I don't have so much memory to train the network. I have a total 40.000 images (15 channels) for 800 serials, so 8 failed serials e 792 healthy ones. If I train the CNN with 50% healthy and 50% failed serials and, on validation set and test set I have 1% my performance are really bad. What should I do? Please, don't ask me to post code. It's too long. I need a theoretical answer.


texts: LSTM Validation MSE always lower than Train MSE

texts:
I am trying to train a LSTM network to forecast time steps further.
I have a list of queries and the current question is based on one among them.

The validation loss (using mse) is always lower than the Train loss (mse), I know I am under fitting hence, generalizing pretty badly.

So atleast overfit for sometime, what changes can be brought in the network. Below is the code for the same.

learning_rate = 0.001
n_neurons = [150, 80, 60, 40, 25, 10]
dropout = 0.2

def fit_lstm(train, n_lag, n_seq, n_batch, nb_epoch, n_neurons, dropout=dropout, reset_state=False):
    # reshape training into [samples, timesteps, features]...
    X, y = train[:, :n_lag], train[:, n_lag:]
    X = X.reshape(X.shape[0], 1, X.shape[1])
    # design network..
    model = Sequential()
    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]),
                 dropout_U=dropout, stateful=True, return_sequences=True))
    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]),
                dropout_U=dropout, stateful=True, return_sequences=True))
    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]),
                 stateful=True, return_sequences=False))
    model.add(Dense(y.shape[1], activation='tanh'))
    tic = time.time()
    model.compile(optimizer=adam, loss='mse') #, metrics=[mean_absolute_percentage_error]
    # fit network..
    loss, val_loss = list(), list()
    for i in range(nb_epoch):
        print('Running Epoch ==> %s' %i)
        history = model.fit(X, y, nb_epoch=1, batch_size=n_batch, validation_split=0.1,
                callbacks=[early_stop,reduce_lr], verbose=2, shuffle=False)
        loss.append(history.history['loss'])
        val_loss.append(history.history['val_loss'])
        model.reset_states() ## clears the state...
    toc = time.time()
    print('====='*10)
    print('Total computation time to train the Model : %0.2f' %((toc - tic) * 100) + ' secs')
    return model, loss, val_loss

Any help would be highly appreciable.

Thanks in advance.


texts: Backpropagation with a different sized training set?

texts:
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$

and whose output is a similarly sized array:

$$\vec{h}_{\theta,i} = [h_{\theta,i1},h_{\theta,i2},h_{\theta,i3}], \hspace{5mm}i=1:m $$

BUT, my only training data is not 3d vectors but rather the magnitude/norm of such vectors (with no knowledge of the vector components ($\lambda's$) themselves):

$$y_i= ||[\lambda_{i,1},\lambda_{i,2},\lambda_{i,3}]||, \hspace{5mm}i=1:m $$

So, my concept is to use the cost function:

$$ J = \frac{1}{2m}\sum (||\vec{h}_{\theta,i}|| - ||y_i||)^2 $$

Note, the difference between this and the more usual quadratic cost function ($ J = \frac{1}{2m}\sum (\vec{h}_{\theta,i} - \vec{y}_i)^2 $) which I would use if $y_i$ was the same shape as the output.

In the typical cost function case I've mentioned above the backpropagation algorithm always starts with computing the output layer error (which based on the simple derivative of that cost function is just $\delta^L = a^L - y_i $). From there you follow to the next layer error which depends on $\delta^L$.

My problem is what should my output layer error be? I've tried just using the derivative of $J$ with respect to $a^L$ using my new cost function (I won't write out that derivative here because it's a bit hairy), but I can't seem to simply slot that in as my output error and continue on with normal back propagation with any kind of sensible results?

If you can't suggest how I should approach my BP algorithm, then perhaps it's because you think my cost function is a non-starter? As a note: I have sort of proven the concept to myself using a linear model rather than an NN and training it with this dataset and linear derivatives rather than BP. The accuracy was low though given the nonlinearity of the data so I would like to use an NN approach.


texts:
I have a medical dataset which looks like this:

patient_id disease_id
1111111111  DISEASE:1
1111111111  DISEASE:2
1111111111  DISEASE:3
1111111111  DISEASE:4
1111111111  DISEASE:5
1111111111  DISEASE:6
1111111111  DISEASE:6
1111111112  DISEASE:1
1111111112  DISEASE:2
1111111112  DISEASE:4
1111111113  DISEASE:1
1111111113  DISEASE:5

which I need to feed into a neural network/random forest model. So, the only natural data representation to feed into the models I thought of was:

    patient_id   DISEASE:1  DISEASE:2  DISEASE:3  DISEASE:4  DISEASE:5  DISEASE:6  ...
    11111111111     1           1           1           1           1        1     ...
    11111111112     1           1           0           1           0        0     ...
    11111111113     1           0           0           0           1        0     ...

But my dataset is very big (~50GB, 1.5 GB compressed) and has tons of disease_ids so that the reshaping this data in the most efficient way possible in R requires 11.7 TB of space in compressed in RDs format (I know this because I divided the dataset into 100 chunks and reshaping of a single one resulted in 117 GB heavy RDs file; merging 100 of these would produce something larger than 11.7TB).

Now, I have 5 dataset this big that I need to merge together, so I feel a bit stuck. I need to come up with a more efficient data representation but don't know how as I am dealing with categorical variables which will require 1-hot encoding. Can anyone suggest any alternative ways to deal with a data like this.


texts: A Machine learning model that has a undefined input size but a fixed output?

texts: I don't know too much about ML but I can seem to figure out how to train something like this. If you guys could list some possible ways to do this, thank you.


texts: Understanding the "Wide" part of Google's wide and deep

texts:
Google's wide and deep recommender model sounds really cool, but I'm struggling to believe I'm grasping the wide section right so wanted to check my understanding.

Their paper says the following:

  The wide component consists
  of  the  cross-product  transformation  of  user  installed  apps
  and impression apps

  Each example
  corresponds to one impression

Let's say we have 5 apps, A through E. My understanding is that the cross-product transformation would represent that as 20 columns, representing each possible combination of installed and impressed app (making 25, but then presumably the 5 "matching" cross-products like and(installed=App_A, impressed=App_A) would be removed because presumably Google is smart enough not to impress Apps the user already has). Let's also say we have 3 Users, called X - Z. X has installed apps A and C, and is shown app B and D. Y has installed App B and is shown A and E. Z has installed apps A, C and D and is shown apps B and E. With that dataset, the cross-product transformation should look (I think) like this:

My question is; is my understanding of the transformation there correct? If so that's going to be one gigantic matrix in fairly short order, particularly given they have over a billion users and a million different apps.


texts: Definition - center of the cluster with non-Euclidean distance

texts:
I have cluster consisting of two points and I am working with non-Euclidean distance. I wonder if it is appropriate to talk about center of cluster in this case and how we can interpret it? We can start with simpler case when distance is weighted Euclidean.

Update: Here is clarification. Assume we are using Manhattan distance to find centroid of our 2 point cluster. As far as I understand centroid is not unique in this case if we use PAM algorithm. We need to work with whole set of centroids  for one cluster. Hence, clustering might produce random results on each iteration. It looks to me that problem is not well posed.


texts: Does policy optimization learn policies to make better actions with higher probability?

texts:
When I talk about policy optimization, it is referred to the following picture, and it is linked to DFO/Evolution plus Policy Gradients.

I would like to know is it correct to say: Policy Optimization learns policies to make better actions with higher probability?

Also, what is the location of  Proximal Policy Optimization in the picture?


texts: Can a Neural Network Measure the Random Error in a Linear Series?

texts:
I have been trying to develop a neural network to measure the error in a linear series. What I would like the model to do is infer a linear regression line and then measure the mean absolute error around that line.

I have tried a number of neural network model configurations, including recurrent configurations, but the network learns a weak relationship and then overfits. I have also tried L1 and L2 regularization but neither work.

Any thoughts? Thanks!

Below is the code I am using to simulate the data and a fit sample model:

import numpy as np, matplotlib.pyplot as plt

from keras import layers
from keras.models import Sequential
from keras.optimizers import Adam
from keras.backend import clear_session

## Simulate the data:

np.random.seed(20190318)

X = np.array(()).reshape(0, 50)

Y = np.array(()).reshape(0, 1)

for _ in range(500):

    i = np.random.randint(100, 110) # Intercept.

    s = np.random.randint(1, 10) # Slope.

    e = np.random.normal(0, 25, 50) # Error.

    X_i = np.round(i + (s * np.arange(0, 50)) + e, 2).reshape(1, 50)

    Y_i = np.sum(np.abs(e)).reshape(1, 1)

    X = np.concatenate((X, X_i), axis = 0)

    Y = np.concatenate((Y, Y_i), axis = 0)

## Training and validation data:

split = 400

X_train = X[:split, :-1]
Y_train = Y[:split, -1:]

X_valid = X[split:, :-1]
Y_valid = Y[split:, -1:]

print(X_train.shape)
print(Y_train.shape)
print()
print(X_valid.shape)
print(Y_valid.shape)

## Graph of one of the series:

plt.plot(X_train[0])

## Sample model (takes about a minute to run):

clear_session()

model_fnn = Sequential()
model_fnn.add(layers.Dense(512, activation = 'relu', input_shape = (X_train.shape[1],)))
model_fnn.add(layers.Dense(512, activation = 'relu'))
model_fnn.add(layers.Dense(  1, activation = None))

# Compile model.

model_fnn.compile(optimizer = Adam(lr = 1e-4), loss = 'mse')

# Fit model.

history_fnn = model_fnn.fit(X_train, Y_train, batch_size = 32, epochs = 100, verbose = False,
    validation_data = (X_valid, Y_valid))

## Sample model learning curves:

loss_fnn = history_fnn.history['loss']
val_loss_fnn = history_fnn.history['val_loss']
epochs_fnn = range(1, len(loss_fnn) + 1)

plt.plot(epochs_fnn, loss_fnn, 'black', label = 'Training Loss')
plt.plot(epochs_fnn, val_loss_fnn, 'red', label = 'Validation Loss')
plt.title('FNN: Training and Validation Loss')
plt.legend()
plt.show()

UPDATE:

## Predict.

Y_train_fnn = model_fnn.predict(X_train)
Y_valid_fnn = model_fnn.predict(X_valid)

## Evaluate predictions with training data.

plt.scatter(Y_train, Y_train_fnn)
plt.xlabel("Actual")
plt.ylabel("Predicted")

## Evaluate predictions with training data.

plt.scatter(Y_valid, Y_valid_fnn)
plt.xlabel("Actual")
plt.ylabel("Predicted")


texts: How is SVD from scikit-surprise handles empty values

texts: I am studying the surprise lib for recommender system. SVD from this lib doesn't require all value input in user-item matrix. But it is a must of the original SVD method. The official doc doesn't explain this. Anyone knows how surprise works?


texts:
I was looking through the documentation for SVM.SVC where it said:

C: float, default=1.0
Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive. The penalty is a squared l2 penalty

Does this mean it's equal to inverse of lambda for our penalty function? ( Which is l2 in this case )
If so, why cant we directly alter the lambda instead of the inverse of it since that's the hyperparameter in the formula?


texts: Multivariate time series forecasting with LSTM

texts:
I have the following problem:

$\mathbf{Y}(t)$ = real valued random vector of observations at time t, $Y_i(t) \in R_{(0, 1)}$

$\mathbf{X}(t)$ = real valued random vector of observations at time t, $X_i(t) \in R$

I want to forecast $\mathbf{Y}(t+1)$, given $\mathbf{X}(t+1)$ and $\mathbf{Y}$ history.

To sum up I'd like to forecast a vector of observations given its history and another vector of observations as the same timestamp.

To make it clearer: let's say that I want to forecast daily temperatures in London and Dublin, knowing the same day temperatures in Manchester and Liverpool and London and Dublin (and Manchester and Liverpool) historical data. It's a bit like a kalman filter where you want to estimate the state variables given some output.

How would you set up this problem in an LSTM?


texts:
I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In the test phase, all units are present, but we multiply each of them with the probability.

Now, is it like, let we have 4 input units originally named a,b,c,d. In the training stage, after applying dropout, with a dropout rate of $0.5$, we are left with units a and c. So, As in the test stage, all the units are present, so, is it like, we multiply each of the units with $0.5$? Also, Is $p$ defined for each of the units in the network, or for the entire Neural Network?

Also, In doing so, how is the result same for training and test stage?


texts: What is the state-of-the art ANN architecture for MNIST?

texts: What is actually the best neural network architecture for the classic MNIST digit classifying task? I couldn't find any that would claim to be the winner...


texts: What are the default values of nodes and internal layers in Neural Network models?

texts:
What is the default number of internal layers and internal nodes in training a neural network?

My data has 62 observations with roughly 200 predictors. I have a target variable with two classes and implemented a neural network with one internal layer and one internal node without repeats. Also, I tried with two internal layers, with 5 internal nodes in one, and 2 internal nodes in second layer. I want to find the accuracy, first, on default values and then I will try to optimize the model performance.

What is the criterion to choose the number of layers and internal nodes in a neural network training model? In the case of Random Forest, we can choose to try to be roughly equal to the square root of the number of predictors.


texts: sklearn FutureWarning message when running a CNN model

texts:
When I run my model, I am receiving the following error message:
FutureWarning: Pass classes=[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24], y=[20 16  4 ...  2  2  2] as keyword args. From version 0.25 passing these as positional arguments will result in an error
  FutureWarning)

I am assuming I need to pass them as keyword args. I am new to the Sklearn library and wondered if anyone has had experience removing this warning before or passing arguments as keyword args? I can ignore the warning, but I don't think that would be best.


texts: What is the state of the art in the field of NLP?

texts:
I am new to Natural Language Processing, I think NLP is a challenging field, the syntax and semantic ambiguities could cause a lot of problems. For example I think for these problems machine translation is a hard task.

Therefore there are probably many approaches and methods that have been applied to this field. But what are the latest and most promising approaches and methods in the field of NLP?

Are these techniques highly dependent on the target language?


texts:
We are developing a classification system, where the categories are fixed, but many of them are inter-related.

For example, we have a category called, "roads" and another one called "traffic". We believe that the model will be confused by the text samples, which could be in roads category and also in traffic.

Some of our text samples are suitable for multi class labelling too.  For example, "There is a garbage dump near the footpath. The footpath is broken completely". This text could be categorized into garbage bucket or footpath bucket.

We are going to build a training set for this classifier, by manually annotating the text. So, can we put multiple labels for one issue? How should we deal with  text with multiple labels for it? Should they be added into all categories to which it is tagged to, as training sample ?

For example, "There is a garbage dump near the footpath. The footpath is broken completely". This text could be categorized into garbage bucket or footpath bucket. So, should this text be added as a training sample for garbage and footpath? How should we consider the labels?

Can you please give your insights?


texts:
In R, stem() produces the graph with "stems" in ascending order. Is there a code for obtaining them in descending order?

Example:

stem (cars$dist)

The decimal point is 1 digit(s) to the right of the |:

   0 | 14004678
   2 | 002466668822244466
   4 | 002668024466
   6 | 046806
   8 | 04523
  10 |
  12 | 0

The desired output:

  12 | 0
  10 |
   8 | 04523
   6 | 046806
   4 | 002668024466
   2 | 002466668822244466
   0 | 14004678


texts:
Ok, maybe this is a really dumb question, but has anyone ever considered extending some of the statistical learning methods to live in some space other that $\mathbb R^n$? My guess is that you're just shuffling parameters around, but I wonder if there's any intuition to be gained.

This question came up as I was working through some course notes on neural nets, and realized that computer scientists may be even more cavalier with tensor indices than physicists :)


texts: Vector dimensionality seems to be implemented incorrectly

texts:
I'm trying to implement a fuzzy topic modeling approach in Python based on a paper, which is accommodated with an R implementation from GitHub. In one of the first steps a document term matrix is created in which each row represents a document j and each column represents a word i, this is an n x m matrix (with n the total number of documents and m the vocabulary size).
The paper describes four ways to perform a global term weighting of the words in the corpus. Described as follows:

Although the four GTW methods are all different, they have in common that we need to perform some calculation on the summation over j (documents). Since we take this summation over j, I expect each formula's output dimensionality to be a vector of length m (size of vocabulary).
This is implemented in R as follows:
document_term_matrix <- as.matrix(DocumentTermMatrix(corpus, control = list(stemming = FALSE, stopwords=FALSE, minWordLength=3, removeNumbers=TRUE, removePunctuation=TRUE )))
gw_idf <- gw_idf(document_term_matrix)

However, gw_idf is a vector of length n (instead of m). Is this implemented correctly and am I missing something in my interpretation? Or should the output indeed be a vector of length m and is it implemented incorrectly?


texts: How to use PCA in CNN for image recognition using Keras?

texts:
I created a CNN model for image classification and I want to use Principal Component Analysis (PCA) but when I run pca.fit() code, the code still running for hours and the RAM become full. So, I want to know how to use PCA in CNN for image recognition using Keras?

My code:

#Data files
train_iris_data = 'Iris_Database_01/Training'
valid_iris_data = 'Iris_Database_01/Validation'
test_iris_data = 'Iris_Database_01/Testing'

#Image data generator
train_iris_datagen = ImageDataGenerator(
rotation_range=10,
shear_range=0.2,
zoom_range=0.1,
width_shift_range=0.1,
height_shift_range=0.1
)

test_iris_datagen = ImageDataGenerator()

#Image batches
image_size = (224, 224)
batch = 32

# Training
train_iris_generator = train_iris_datagen.flow_from_directory(
train_iris_data,
target_size=image_size,
batch_size=batch,
class_mode='categorical')

# Validation
validation_iris_generator = test_iris_datagen.flow_from_directory(
valid_iris_data,
target_size=image_size,
batch_size=batch,
class_mode='categorical',
shuffle = False)

# Testing
test_iris_generator = test_iris_datagen.flow_from_directory(
test_iris_data,
target_size=image_size,
batch_size=1,
class_mode='categorical',
shuffle = False)

pca = PCA(n_components=2)
pca.fit(train_iris_generator)

#pca = PCA(n_components=0.8)
#pca.fit(train_iris_generator)


texts: How to properly predict data with LSTM using train/test?

texts:
I see one paradox here. If we use Train/Test split and evaluate our Test data, we might get a good score, but any further prediction will not be credible, because model didn't train the Test data and include it's sequences in memory.

On the other side, we can train the data on Train and Test sequence as train data, but then we can not evaluate our predictions, because we have no testing reference.

How do you properly predict LSTM models?


texts: How to choose negative examples for recommendation system?

texts:
I am building a search recommendation system for e-commerce which generates most relevant results given an input query. I have framed it as a classification problem (learning to rank) and using pointwise ranking to compute relevance scores.

Let us say for every input query Q1, we have to generate 30 most relevant results. Suppose, a user initiates a search query and chooses the 3rd result as a relevant result. My question is how should this interaction be used to generate training dataset for the model :

Q1 - Result 3 : Positive Label
Q1 - Result 1,2 : Negative Label (User obviously saw result 1,2 and didn't like it. Also mentioned as skip-highs in some papers/blogs).
Q1 - Random : Negative Label (just add some random results to add random negatives to the dataset).

What about ranks 4-30. It's possible user saw ranks 4-8, but after those ranks user didn't even see the results. Should these lower ranks be used to generate implicit negatives to feed to the model ?

Pros

Random sampling from implicit negatives in lower ranks will ensure that the model is trained on the entire feature distribution that it would have to score during predictions. If we just use the explicit positives/negatives from Step 1/2, model would have been trained only on very relevant results. Sampling from lower ranks will ensure that the model will be trained on entire spectrum of relevant results too.

Cons

We are adding noise to the training dataset, since the user didn't even interact with these results.

So, the main question is should we include the results from rank 4-30, which were probably not even seen by the user as negative examples in the training dataset or not?


texts: Improve results of image stitching using OpenCV (material science)

texts:
I'm trying to improve image stitching results with a series of pictures taken via microscope. The object I've photographed is a 3D printed cube that has a number of repeating shapes on the surface. The shapes are repeated so that we can measure their dimensions and perform some machine learning algorithm on the data down the road.

However, as the shapes are identical, and the printer material (resin) that forms the cube present little in the way of distinct features, image stitching performance has not been as good as hoped.

I'm at the point where I'd like to add unique features into the design itself, but haven't found much literature discussing this issue and possible solutions. Given that I'm working with the CAD software SolidWorks, I'm wondering if anyone can provide a suggestion for adding features (possibly non-repeating?) that could theoretically improve performance of OpenCV's image stitcher.

Thanks!

Side note: these cube samples have a length/width of 25mm when printed.


texts: Whats the difference between add.LSTM(num_hidden, droput=0.5) and add.Dropout(0.5) in Keras?

texts:
Could anyone please explain what is the difference between these two cases, specified in the title.
I believe I am not the only one who is confused.

I have read that it is preferrable to add Dropout separately (second case), but it was not explained why that holds true.

Cheers.


texts:
Which machine learning or deep learning model(has to be supervised learning) will be best suited for recognizing patterns in financial markets ?

What I mean by pattern recognition in financial market :
Following Image shows how a sample pattern (i.e. Head and shoulder) looks like:

Image 1:

And Following Image shows how it actually forms in real chart events:

Image 2:

What I'm trying to do is:
Any pattern similar to Image 1 can be defined as Head and Shoulder Pattern but in a Chart (Price Chart) it will not form as clearly as Image 1. Image 2 is the sample of Head and Shoulder Pattern form in Chart (Price Chart).
As it seems in Image 2, it can not be identified as Head and Shoulder Pattern by normal algorithms or analysis (Because there are a lot of highers and low forming a lot of structure, which can easily mislead into a lot of shoulders or head or any other structures).
I'm expecting to train machine to recognize the Head and Shoulder Pattern when similar (as Image 2) pattern is formed.

Thank you for your time.

Let me know if I'm taking it to wrong way.
I only have beginners knowledge on Machine Learning.


texts: About proper Training Model and settings for building Subtitle by using Deepspeech GitHub Project

texts:
I am newbie in speech-to-text AI , but i am trying to find suitable deepspeech setting models for auto extracting the subtitle STR file from the video file, so when i am using the deepspeech-0.6.1 by this setting via this Colab Notebook:
!deepspeech --model deepspeech-0.6.1-models/output_graph.pbmm --lm deepspeech-0.6.1-models/lm.binary --trie deepspeech-0.6.1-models/trie --audio test.wav   --extended --json

gave me a better text like this:

and people fishing up in this wall i disarrayed anaveni an derstand
you angle with the amidei and desol make it a relation to me now i was
angry at first to the twin tione can annihilate them how we can tear
down there or hiawatha known by a truly for policing the others and
tautening to no contraction i get it summoned and you can never the
world with the contents one man fight latest him you apaian why we
could have her own world i guess when behind they are creators to die
oh many a unnumbered things out on to donneraile mere the moment were
i to tipple gave them a long amain ere hardey created on a new nothing
pertinacity of monomania he is ugly man and this was his rage

But when i am using this GitHub project :
https://github.com/abhirooptalasila/AutoSub
which is using the deepspeech-0.8.2, by this kind of command:
!python3 autosub/main.py  --model /content/AutoSub/deepspeech-0.8.2-models.pbmm --scorer /content/AutoSub/deepspeech-0.8.2-models.scorer --file test.wav

It create this kind of STR file which have too differences with deepspeech-0.6.2 result as show above ( lower quality!):

00:00:00,15 --> 00:00:07,90 and people the tooth allies
2 00:00:09,15 --> 00:00:09,80 this
3 00:00:19,05 --> 00:00:53,65 and isaiah pan i understand your anger
with him and maybe your ride out should exist
4 00:00:54,05 --> 00:00:55,25 a red
5 00:00:55,70 --> 00:00:56,10 and
...

So i don't know what is differences of using the --model /content/AutoSub/deepspeech-0.8.2-models.pbmm --scorer /content/AutoSub/deepspeech-0.8.2-models.scorer setting instead of --model deepspeech-0.6.1-models/output_graph.pbmm --lm deepspeech-0.6.1-models/lm.binary --trie deepspeech-0.6.1-models/trie setting and how i could improve the setting to have better quality subtitle for videos, by using some more welt trained models via others open-source databases like said here:

2.1 Data Collection
We want to use some open-source
datasets, that are available online

    National Speech Corpus: Contains 2000 hours of locally accented audio and text transcriptions
 LibriSpeech: Dataset consists of a large-scale corpus of around 1000 hours of English speech
TIMIT:  A collection of recordings of 630 speakers of American English
L2-ARCTIC:  A non-native English speech corpus

or etc methods.
Also asked here:
https://github.com/mozilla/DeepSpeech/issues/3330
Thanks.


texts: Supervised learning vs reinforcement learning for a simple self driving rc car

texts:
I'm building a remote-controlled self driving car for fun. I'm using a Raspberry Pi as the onboard computer; and I'm using various plug-ins, such as a Raspberry Pi camera and distance sensors, for feedback on the car's surroundings. I'm using OpenCV to turn the video frames into tensors, and I'm using Google's TensorFlow to build a convoluted neural network to learn road boundaries and obstacles. My main question is, should I use supervised learning to teach the car to drive or should I provide objectives and penalties and do reinforcement learning (i.e, get to point B as fast as possible while not hitting anything and staying within the road boundaries)? Below is a list of the pros and cons that I've come up with.

Supervised learning pros:

The inputs to the learning algorithm are pretty straightforward. The car learns to associate the video frame tensor and sensor distance readings with forward, backward, and angular wheel displacement
I can more or less teach the car to drive exactly how I want (without overfitting, of course)
I've done tons of supervised learning problems before, and this approach seems to comfortably fit my existing skill set

Supervised learning cons:

It's not clear how to teach speed, and the correct speed is pretty arbitrary as long as the car doesn't go so fast that it veers off the road. I suppose I could drive fast during training, but this seems like a crude approach. Maybe I could manually add in a constant variable during training that corresponds to the speed for that training session, and then when the learning algorithm is deployed, I set this variable according to the speed I want?

Reinforcement learning pros:

If I build my car with the specific purpose of racing other people's self driving cars, reinforcement learning seems to be the natural way to tell my car to "get there as fast as possible"
I've read that RL is sometimes used for autonomous drones, so in theory it should be easier in cars because I don't have to worry about up and down

Reinforcement learning cons:

I feel like reinforcement learning would require a lot of additional sensors, and frankly my foot-long car doesn't have that much space inside considering that it also needs to fit a battery, the Raspberry Pi, and a breadboard
The car will behave very erratically at first, so much so that maybe it destroys itself. It might also take an unreasonably long time to learn (e.g., months or years)
I can't incoporate explicit rules later on, e.g., stop at a toy red-light. With supervised learning, I could incorporate numerous SL algorithms (e.g., a Haar Cascade classifier for identifying stoplights) into a configurable rules engine that gets evaluated between each video frame. The rules engine would thus be able to override the driving SL algorithm if it saw a red stoplight even though the stoplight might not have been part of the training of the driving algorithm. RL seems too continous to do this (ie., stop only at the terminal state)
I don't have a lot experience with applied reinforcement learning, although I definitely want to learn it regardless


texts: Data splitting for a binary classification model

texts:
I'm trying to build a binary classification model that will tell who's going to buy the product and who's not. I've heard that splitting a dataset into two different subsets is a common way when you prepare an input data.

  [ ================ Training Data 80% ================= ] [ ==== Test Set 20% ==== ]

Is it just mindlessly splitting a chunk of dataset by some amount of proportion like above? Is it that simple?

Imagine I have this simple dataset below.

UserId,UserName,AppId,Purchased
1,Lianne,1,1
1,Lianne,2,1
1,Lianne,3,1
1,Lianne,4,1
1,Lianne,5,1
1,Lianne,6,0
1,Lianne,7,0
1,Lianne,8,0
1,Lianne,9,0
1,Lianne,10,0

As the common recommended way, I splitted it into two groups.

// Training Data Set
UserId,UserName,AppId,Purchased
1,Lianne,1,1
1,Lianne,2,1
1,Lianne,3,1
1,Lianne,4,1
1,Lianne,5,1
1,Lianne,6,0
1,Lianne,7,0
1,Lianne,8,0

// Test Set
UserId,UserName,AppId,Purchased
1,Lianne,9,0
1,Lianne,10,0

Would this work? well it seemed not and it turned out it actually didn't. The model was wrong about predicting on the appId of 6,7,8,9. It thought the user number one would buy them with a slightly high chance. The metrics look like...

TP : 5
FP : 4
FN : 1
Accuracy : 0.5
Auc : NaN
F1Score : NaN
Precision : 0
Negative Precision : 1
Negative Recall : 0.5

To make a proper model, what my test dataset should look like on this sample training data?


texts:
I am following this tutorial and am trying to use it for another OpenAI Gym environment (MountainCar-v0). This environment has a discrete action space instead of a box one. By this I mean if you were to print(env.action_space) for this particular environment you get Discrete(3) as the result as opposed to some Box(3,) action space.

Here is the code:

def create_critic_model(self):
    state_input = Input(shape=self.env.observation_space.shape)
    state_h1 = Dense(24, activation='relu')(state_input)
    state_h2 = Dense(48)(state_h1)

    action_input = Input(shape=self.env.action_space.shape)
    action_h1    = Dense(48)(action_input)

    merged    = Add()([state_h2, action_h1])
    merged_h1 = Dense(24, activation='relu')(merged)
    output = Dense(1, activation='relu')(merged_h1)
    model  = Model(input=[state_input,action_input],
        output=output)

    adam  = Adam(lr=0.001)
    model.compile(loss="mse", optimizer=adam)
    return state_input, action_input, model

As you can see, the critic model takes in both the state and action as input. Because I am working in with a discrete action space I get this error: AttributeError: 'Discrete' object has no attribute 'shape'. This is because the discrete action space has no shape but I have to pass it through as an input. And when just passing env.action_space as the input, I get this error: TypeError: 'Discrete' object is not iterable

The Critic model uses an action input which is the action space shape. Being a discrete action space, mountaincar does not have an action space shape. How would I reconfigure this code to suit my purpose(use action input for a discrete action shape space).

Also, on this line:

future_reward = self.target_critic_model.predict([new_state, target_action])[0][0]

it seems we are trying to predict on 2 diff arrays, and I get an error regarding that. So what's going on there?

Error is:

ValueError: Error when checking : the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 arrays but instead got the following list of 2 arrays: [array([[-0.47361678, -0.00322982]]), array([[ 0.00318837, -0.05581997,  0.06615788]], dtype=float32)].

UPDATE

So after switching to my new machine, installing fresh python, keras, tensorflow, and gym(which got rid of the other error) I am left with a new keras error:

AssertionError: Please provide to Input either a `shape` or a `batch_shape` argument. Note that `shape` does not include the batch dimension.

Also still getting the array related error.


texts:
So i'm trying to execute the code provided here: https://github.com/eriklindernoren/Keras-GAN/blob/master/infogan/infogan.py

Both for InfoGAN and CGAN i get the same error:

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-489e5ad1b538> in <module>()
      1 infogan = INFOGAN()
----> 2 infogan.train(epochs=50000, batch_size=128, sample_interval=50)

7 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/cbook/__init__.py in to_filehandle(fname, flag, return_opened, encoding)
    390             fh = bz2.BZ2File(fname, flag)
    391         else:
--> 392             fh = open(fname, flag, encoding=encoding)
    393         opened = True
    394     elif hasattr(fname, 'seek'):

FileNotFoundError: [Errno 2] No such file or directory: 'images/0.png'

Can someone please help?


texts: Evaluating a IR system (Precision and Recall)

texts:
I am studying by now IR system, in the field of valuation of IR system outputs related to a specific query but I need some help to understand it properly.

My book states that when an IR system has to be evaluated, we need a test document collection, a set of query examples, a valuation (relevant or not) for each couple of query/document, defined by experts in the field. So we need two measures to know quantitatively if a IR system is good: Precision and Recall.

My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?

I'll explain: before we calculate Precision and Recall related to a specific query example (see above), we need to know how many elements belong to the relevant set, which is impossible if there isn't a valuation (relevant or not) for the query we are using. My book says we can increase Recall in a search engine by using the relevance feedback technique (query expansion and term reweighting): in this case do we assume the Recall value is unknown?

For example, everyday many documents are shared on the Internet and Google can find them. So it is impossibile to apply Recall and Precision to this scenario, in which information grows and there is no valuation for every new document for each specific query. It is also impossibile to predict all the possibile queries a user can do on a search engine.


texts: Pandas loads a csv file incorrectly, but without throwing an error

texts:
So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas loads the data improperly. And it does so consistently. Let me show you what I mean.
This is a snapshot of the dataset (a csv file)

I think you can more or less see how it looks here. However, when I load it in Pandas, I get this:

As you can see there's a clear distortion. For one thing, the Date column has been pushed from the first column in the spreadsheet to the second in the csv. Its values have also been replaced with what looks to me like a copy of the "Travels" column. Moving on, you can also see that the column I'm interested in ("Total Direct Remittances") is totally incorrect. In the spreadsheet it starts with 342, but when loaded it starts with 23423, and the rest of its values are completely off.
Here's the code:
import pandas as pd

data = pd.read_csv("/home/user/Downloads/IntPayments26102020.csv")

You can get a copy of the data here. Any ideas guys?


texts:
I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a sentence, the result is a tuple with the form below:

[('__label__positif', '__label__négatif', '__label__neutre', 0.8947999477386475, 0.08174632489681244, 0.023483742028474808)]

but when I tried to apply it to the column "text" I did this :

from pyspark.sql.types import *
from pyspark.sql.functions import udf, col
import fasttext

schema = StructType([
    StructField("pos", StringType(), False),
    StructField("neg", StringType(), False),
    StructField("ntr", StringType(), False),
    StructField("pr_pos", DoubleType(), False),
    StructField("pr_neg", DoubleType(), False),
    StructField("pr_ntr", DoubleType(), False)
])

udf_label = udf(lambda words : predictClass(words), schema)
df = df.withColumn("classe", udf_label(col('text')))

df.select('classe').show()

I get this error:
PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects


texts: How i can generate the probabilistic graph for my dataset?

texts:
Im doing my academic project. im having the base paper for reference  the paper is IEEE paper "effective and efficient clustering methods for correlated probabilistic graph". i wish to do this in R tool. in this paper two algorithm are implemented. i like to implement the peedr algorithm in the paper. how can i give the input for that algorithm.? suggest the packgages in R tool
 the paper can be found here
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6570474


texts: After anaconda installation spyder not working

texts:
I am using ubuntu 16.04.
after anacoda installation sypder not open.
Anaconda version: 4.4.6

Anaconda environment

my_root                  /home/shiva/.conda/envs/my_root
py27                     /home/shiva/.conda/envs/py27
py35                     /home/shiva/.conda/envs/py35
base                  *  /home/shiva/anaconda
myenv                    /home/shiva/anaconda/envs/myenv

I get the following error

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spyderlib/spyder.py", line 2331, in main
    mainwindow = run_spyder(app, options, args)
  File "/usr/lib/python3/dist-packages/spyderlib/spyder.py", line 2215, in run_spyder
    main.setup()
  File "/usr/lib/python3/dist-packages/spyderlib/spyder.py", line 752, in setup
    from spyderlib.plugins.inspector import ObjectInspector
  File "/usr/lib/python3/dist-packages/spyderlib/plugins/inspector.py", line 41, in <module>
    from spyderlib.widgets.ipython import IPythonControlWidget
  File "/usr/lib/python3/dist-packages/spyderlib/widgets/ipython.py", line 28, in <module>
    pygments_patch.apply()
  File "/usr/lib/python3/dist-packages/spyderlib/pygments_patch.py", line 22, in apply
    from pygments.lexer import _TokenType, Text, Error
  File "/usr/lib/python3/dist-packages/pygments/lexer.py", line 19, in <module>
    from pygments.filters import get_filter_by_name
  File "/usr/lib/python3/dist-packages/pygments/filters/__init__.py", line 20, in <module>
    from pygments.plugin import find_plugin_filters
  File "/usr/lib/python3/dist-packages/pygments/plugin.py", line 39, in <module>
    import pkg_resources
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2952, in _initialize_master_working_set
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 956, in subscribe
    callback(dist)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2952, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2515, in activate
    declare_namespace(pkg)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2097, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2047, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2066, in _rebuild_mod_path
    orig_path.sort(key=position_in_sys_path)
AttributeError: '_NamespacePath' object has no attribute 'sort'

how can I solve this problem?


texts:
I have used pip to install orange and the installation seemed fine.  There was not Orange icon on my ubuntu 16.04 desktop and tried orange3 in the command line, it didn't work, and tried python3 -m orange and python3 -m orange 3 and neither worked as well.

How can I use orange?


texts:
I am reading [1], where the researchers do a logistic regression, but add to the loss function the following penalization term for fairness

$ R^{AVD}_{FP}(\theta; S) = \left\lvert \dfrac{\sum\limits_{x \in \mathcal{D}_{00}} \theta^T x}{\lvert \mathcal{D}_{00} \rvert} -  \dfrac{\sum\limits_{x \in \mathcal{D}_{10}} \theta^T x}{\lvert \mathcal{D}_{10} \rvert} \right\rvert $

where FP=False Positive, $x$ are the predictors, $\theta$ are the parameters to learn, and $\mathcal{D}_{ab}$ is a partition of the dataset defined as follows:

$\mathcal{D}_{ay} = \{(x^i, y^i) \in \mathcal{D}: x^i_1 = a, y^i = y\}$

where $x_1$, the first predictor, is a "protected" attribute, for example, race. We want a prediction model that is not sensible to $x_1$. For example, we want to classify the probability that a person will commit a heist independently if the person is white or black.

But I am not understanding the penalization term. It would make sense if the sum was over the misclassified points. But there is no such concept there? Also because the penalization is added to the loss function, so during the learning phase, where of course there is not yet such a concept as misclassified or not.

For example, the first sum seems to be over all the samples for which the protected attribute is 0, and the true label is 0. The $\theta^T x$ can be negative (true negative) or positive (false positive!).

How does it make sense?

[1] Bechavod, Y., & Ligett, K. (2017). Penalizing Unfairness in Binary Classification. Retrieved from http://arxiv.org/abs/1707.00044


texts: Why is PyTorch's DataLoader not deterministic?

texts:
I've set the seeds like this (hoping to cover all bases):

random.seed(666)
np.random.seed(666)
torch.manual_seed(666)
torch.cuda.manual_seed_all(666)
torch.backends.cudnn.deterministic = True

The below code will still output DIFFERENT batches for both namesTrainLoader1 and namesTrainLoader2 but they should really be the same. How come creating the model is affecting the deterministic values?

namesDataset = NamesDataset()
namesTrainLoader1 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader1:
    print(each)

model = TorchRNN(inputSize, hiddenSize, outputSize)

namesTrainLoader2 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader2:
    print(each)

Output for namesTrainLoader1:

('saiki', 'close', 'sloan', 'horos', 'roman')
...

Output for namesTrainLoader2:

('david', 'abeln', 'hatit', 'holan', 'protz')
...

I also tried using worker_init_fn (e.g. with lambda x: 0) in the DataLoader, but that made no difference.

Why is this not deterministic? How can I make it deterministic? i.e. reset the internal seed of the DataLoader?


texts: Data Visualization of Feature Distribution Plot

texts:
I have been trying my best to plot this type of feature distribution plot using Python.  I looked for what they are called to replicate the plot, but couldn't find the name of it?

The Reference of the plot is :
https://link.springer.com/article/10.1007/s13278-016-0358-5


texts: Fraud risk propagation in large scale network

texts:
What's the best approach to do some graph analytics and risk propagation in a network using python where multiple accounts are connected through a relationship and few of the accounts in the network are marked as bad accounts and the rest are unknown?

I tried using networkx but it seems to run forever. I have about 8MM edges and 40K nodes


texts: sklearn predict: IndexingError: ('Too many indexers', 'occurred at index ')

texts:
The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to predict and compare the prediction 'REVIEW_ACTION' to the actual 'REVIEWER_ACTION'. Ignoring for the moment why this is A BAD THING TO DO, can we focus on how to achieve this?

use_cols = ['FIRST_NAME', 'LAST_NAME', 'PERSON_STATUS', 'DIVISION_NAME',
            'PERSON_TYPE', 'JOB_CHANGE', 'JOB_TRANSFER', 'IDENTIFY_DATE',
            'SSO', 'USER_ID', 'ASSET_ID', 'ROLE', 'HPA',
            'LAST_LOGIN_DATE', 'ASSET_NAME', 'AUDIT_ID',
            'REVIEWER_ACTION', 'USER ID', 'LABELED_ROLE', 'REVIEW_ACTION']
prediction_col = ['REVIEW_ACTION']

categorical_features = ['DIVISION_NAME', 'PERSON_TYPE', 'JOB_CHANGE', 'JOB_TRANSFER',
                       'SSO', 'USER_ID', 'ASSET_ID', 'ROLE', 'HPA', 'LAST_LOGIN_DATE',
                       'USER ID', 'LABELED_ROLE']

categorical_transformer = Pipeline(steps=[
    ('si', SimpleImputer(strategy='constant', fill_value='missing')),
    ('ohe', OneHotEncoder(handle_unknown='ignore'))],remainder='passthrough')

preprocessor = ColumnTransformer(
    transformers=[
        ('cat', categorical_transformer, categorical_features)])

rf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rfc', RandomForestClassifier(n_estimators=100))
                     ])

I then fit the data and try to run the prediction

df['rf_prediction'] = df[categorical_features].apply(rf.predict)

and get the error:

IndexingError: ('Too many indexers', 'occurred at index DIVISION_NAME')

This has something to do with the columns being 'passthrough''d but I'm not sure how to resolve it. I don't want to process some of these columns but want them in the results when I write the file so that I can validate the results.

Somewhere in my debugging I took a step back and am seeing different errors when I call fit. If X = data I see:

ValueError: could not convert string to float: 'Herve'

Note that "Herve" is a field that is passed through (remainder='passthrough') so the model shouldn't be seeing it -- or so I've been led to believe.


texts: Can I fine tune the xgboost model instead of re-training it?

texts: I am using the xgboost library. My system runs a cronjob each night, where it pulls the data from the database and trains the model. However, I would like to remove the re-training of the model again and again, and just fine-tune it with any new data that came in the database. In sklearn's implemantation (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) one could use warm_start option, what about xgboost ?


texts: Deconvolutional Network in Semantic Segmentation

texts:
I recently came across a paper about doing semantic segmentation using deconvolutional network: Learning Deconvolution Network for Semantic Segmentation.
The basic structure of the network is like this:

The goal is to generate a probability map in the end. I'm having trouble figuring out how to realize the deconvolution layer. In the paper, it says:

The output of an unpooling layer is an enlarged, yet sparse activation map. The deconvolution layers densify the sparse activations obtained by unpooling through convolution-like operations with multiple learned filters. However, contrary to convolutional layers, which connect multiple input activations within a filter window to a single activation, deconvolutional layers associate a single input activation with multiple outputs.
The output of the deconvolutional layer is an enlarged and dense activation map. We crop the boundary of the enlarged activation map to keep the size of the output map identical to the one from the preceding unpooling layer.
The learned filters in deconvolutional layers correspond to bases to reconstruct shape of an input object. Therefore, similar to the convolution network, a hierarchical structure of deconvolutional layers are used to capture different level of shape details. The filters in lower layers tend to capture overall shape of an object while the class-specific fine details are encoded in the filters in higher layers. In this way, the network directly takes class-specific shape information into account for semantic segmentation.

Can anyone explain how the deconvolution works? I'm guessing it's not a simple interpolation.


texts: What is the best practice to classify category of named entity in sentence

texts:
I have 1-4 gram text data from wikipedia for 14 categories, which I am using for NE classification.
I feed named entity from sentence to lucene indexer which searches named entity from these 14 categories.
Issue I am facing is, for single entity I get multiple classes as a result with same score.
like while search titanic, indexer gives this result

Score    - 11.23
Title    - titanic
Category - Book

Score    - 11.23
Title    - titanic
Category - Movie

Score    - 11.23
Title    - titanic
Category - Product

now problem is which class to be considered?

I already tried with classifiers (NB,ME in nltk,scikit learn), but as it consider each entity from dataset as feature, it works as indexer only.

Why lucene?


texts:
I am new to Bayesian modeling. I am running Bayesian regression model in R using brm function from brms library, which is powered by STAN. I have a data with 10 million records. I took 10% sample out of it so as to run no Prior bayesian on it, and get Betas and est error, which I plan to use as priors for rest of the data. My queries are:

All the beta estimates I am getting are having 0 est. error and non zero coefficent value, hence the beta distribution would be for e.g N(0.7,0) which isnt a distribution. Hence I cant use these results as prior

I have tried multiple itereation with 10%, 20%, 5% data with different number of chains and itereations combinations but the results are almost same. What should be my approach? My objective is to get proper estimates of all the variables and I want to do it with Bayesian regression itself as a part of learning.
Am I missing something here? Please suggest. Thanks in advance for your help.


texts: Automated taxonomic identification of insects – where can I find some good enough software or code?

texts:
Short context: my colleagues study plant-pollinator networks. Insect (pollinator) identification is a task that requires a lot of effort.

I would like to know if there are already trained or trainable models with good accuracy used to identify pollinators (insects) from mobile phone images taken in the field (possibly, a close-up photo of the insect on a flower). Identification to family level would be good enough for their research. Maybe for some easy to identify insects, identification to specie level would be also possible.

Note that, I could find few research articles that provide code (e.g. a GitHub repository with the model and workflows), but they are mostly focused on using sharp, good quality, close-up images taken in the lab from insect collections, with the goal to identify insects to species level, e.g:

Valan, 2019 - Automated Taxonomic Identification of Insects with
Expert-Level Accuracy Using Effective Feature Transfer from
Convolutional Networks
https://github.com/valanm/off-the-shelf-insect-identification
Buschbacher, 2020- Image-based species identification of wild bees    using convolutional neural networks,
https://deepabis.github.io/,    https://github.com/deepabis/deepabis

Valan (2019) presents a great illustration (below) about the increase in difficulty of such classification. This might get even harder when it comes to using mobile phone photos from the field with a lot of background noise.

I also asked this question on research gate, but I think it might be interesting to have it on this platform too.


texts: Using classification to find the best support and confidence measure in associative rule mining

texts:
I have been trying the find the best support and confidence values for associative rules mining. I came across the following approach from an answer on Quora -

  Picking the "appropriate" values for support and confidence can be
  difficult, as it is very much an unsupervised process.

  However, if you transform the output of Apriori algorithm (association
  rules) into features for a supervised machine learning algorithm, you
  can examine the effect of having different support and confidences
  values (while having other features fixed) on the performance of that
  supervised model (ROC, RMSE, and etc.). Then, the best values for
  support and confidence are the values that maximize, or minimize, the
  performance metric.

Being a rookie in this field, I am not sure on how to do this. Could you please help me on this?


texts: Using machine learning to classify text within javascript?

texts:
Apologies in advance, I am completely new to the world of machine learning. But I just wanted a bit of help in areas to explore or even just a blunt answer if the following was possible.

I have a chrome extension that gets the content of a webpage and am wanting to classify this text into what topic it is - ie. math, science, geography, etc. I know there are several APIs that I could send the data of too initially but was wanting to know if this thing would be possible to do within the extension using javascript.

I made a quick and dirty demo in which I trained https://github.com/nemo/natural-synaptic with a small amount of data and exported it to JSON. I then imported this into the extension and used it to classify the text, it works okay. However, I would need more training data to classify it correctly and it also seems to have a small performance hit when loading an already small amount of trained data (4mb) so increasing this may affect performance even more.

My overall questions are - How feasible is a text classifier using javascript all client side without affecting performance too much? And is there any frameworks, resource you could suggest I look into? Thanks in advance.


texts: How to Save Model that has a TensorFlow Probability Regularizer?

texts:
Consider the following minimal VAE:
import tensorflow as tf
import tensorflow_probability as tfp

tfk = tf.keras
tfkl = tf.keras.layers
tfpl = tfp.layers
tfd = tfp.distributions

#Fake dataset
cim = np.random.randint(2, size=(10,10))

#Parameters

vector_size = 10
input_shape = (vector_size,)
encoded_size = 3
latent_dim = 5

#Model

prior = tfd.Independent(tfd.Normal(loc=tf.zeros(encoded_size), scale=1), reinterpreted_batch_ndims=1)
vae = tfk.Sequential([
    #Encoder
    tfkl.InputLayer(input_shape=input_shape),
    tfkl.Dense(
        tfpl.MultivariateNormalTriL.params_size(encoded_size),
        activation=None,
        use_bias = False
    ),
    tfpl.MultivariateNormalTriL(
        encoded_size,
        activity_regularizer=tfpl.KLDivergenceRegularizer(prior)
    ),
    #Decoder
    tfkl.Dense(
        units = vector_size,
        activation = tf.nn.leaky_relu,
        use_bias = False
    )
])

vae.compile(optimizer=tf.optimizers.Adam(learning_rate=1e-3), loss='mse')
history = vae.fit(cim, cim, epochs = 1)

tf.saved_model.save(vae, './VAE')

The final line throws an error: AttributeError: 'Tensor' object has no attribute 'log_prob'
Tracing the issue, it seems that the activity regularizer tfpl.KLDivergenceRegularizer(prior) is not being serialized corrected. Any ideas for how to save this Model with the regularizer intact?


texts:
I am looking  to find some resources about what I wan't to do :I wan't to make some GUI of my machine learning models and  finally deploy them as a web  app.I find R Shiny to be somehow  ok , but it limits me only to use R,and I have also Python things.Looking at the azure ml studio  it brings  somekind of workflow  like in Disco and ProM from process mining, but I am not  sure  how much pain will be to feed everything  like ensembles/grid search etc to those  flow components,also the full paid  version is  very expensive.
Trying to do a web application  from scratch could also not be ok  cause it will be a lot  of pain  when I will modify my  models.
Any suggestions?


texts: how to implement Excel vlookup on Python

texts:
I have a dataset A and dataset B, both data has a column in common which is the ID, i want to check if dataset A ID values can be found in dataset B ID values. Also get the records of dataset B that have the same value with dataset A. I can do this on excel using vlookup, but I want to automate the process using python.

NOTE:
1. dataset B has 10020 rows while dataset A has 2039 rows.
2. The ID column is numeric.

Thanks.


texts: Training on accurate data versus noisy data

texts:
I have data currently available that is very accurate and I would like to train my classification methods on this set of clean data to learn the important markers for distinguishing between classes. But in the future, my trained classifiers will not be seeing and performing decisions on this cleaned data; instead, it will likely have a lot more noise following some unknown distribution(s). Thus I am wondering, is it 'better' to train on noisy data if I'm going to likely see noisy data in the future, or train on good data since the noisy data should (ideally) correspond to the cleaned data if noise was removed?

Intuitively, if my goal is to simply perform classifications, then training on noisy data seems the 'better' approach since this is better representative of my expected future inputs. But if my goal is to learn about the data and what constitutes a particular decision, then training on cleaned data appears the 'better' approach.

But am I overlooking anything? Would training on the clean and/or noisy data be preferable for different reasons (e.g. generalization, overfitting, reducing dimensionality)?


texts:
I have a dataset with physiological measures of subjects along time. I would like to create (or select) a mean prototype example in order to be able to identify in new examples how far are they from the mean prototype. A second issue will be to select a threshold to determine what is considered near or far.
Each example has 20 numeric features and I have around 300 examples per subject.

First ideas (disregarding outliers):

To iterate through all the examples of a subject and find the one who has the minimum mean distance to all the other examples. This will select a specific example from the dataset.
To use an evolutionary algorithm to find a prototypical example which has the lowest mean distance to all the other examples. This will create a new example that can be used as prototype.

Now I would like to determine when a new example is close, far or very far from the prototype (mean). A possible approach is to set two thresholds distance to determine to which class or case corresponds the new example (close, far or very far). How could I determine these thresholds? Possibly using number of standard deviations? What other approaches can be followed to perform all this?

Let's assume the distance metric is already selected.


texts: Xgboost quantile regression via custom objective

texts:
This is my first time posting, so please bare with me if I miss giving necessary info...

I'm new to GBM and xgboost, and I'm currently using xgboost_0.6-2 in R. The  modeling runs well with the standard objective function "objective" = "reg:linear" and after reading this NIH paper I wanted to run a quantile regression using a custom objective function, but it iterates exactly 11 times and the metric does not change.

I just simply switched out the 'pred' statement following the GitHub xgboost demo, but am afraid it is more complicated than that and I can't find any other examples on using the custom objective function. Do I need to take it a step further and take derivatives for the 'grad' and 'hess' part?

Or could it be a problem with xgboost (doubtful)?

qntregobj <- function(preds, dtrain) {
  qr_alpha = .5
  labels <- getinfo(dtrain, "label")
  preds <- ifelse( preds - labels >= 0
                 , (1-qr_alpha)*abs(preds - labels)
                 , qr_alpha*abs(preds - labels)
                 )
  grad <- preds - labels
  hess <- preds * (1 - preds)
  return(list(grad = grad, hess = hess))
}

step1.param <- list( "objective" = qntregobj
                   , "booster" = "gbtree"
                   , "eval.metric" = "rmse"
                   , 'nthread' = 16
                   )
set.seed(123)
step1.xgbTreeCV <- xgb.cv(param = step1.param
              , data = xgb.train
              , nrounds  = nrounds
              , nfold = 10
              , scale_pos_weight = 1

              , stratified = T
              , watchlist = watchlist

              , verbose = F
              , early_stopping_rounds = 10
              , maximize = FALSE

              ## set default parameters here - baseline
              , max_depth = 6
              , min_child_weight = 1
              , gamma = 0
              , subsample = 1
              , colsample_bytree = 1
              , lambda = 1
              , alpha = 0
              , eta = 0.3
  )
  print(Sys.time() - start.time)

  step1.dat <- step1.xgbTreeCV$evaluation_log
  step1.dat

Which produces:

iter train_rmse_mean train_rmse_std test_rmse_mean test_rmse_std nround
 1:    1        122.6362     0.04268346       122.6354     0.3849658      1
 2:    2        122.6362     0.04268346       122.6354     0.3849658      2
 3:    3        122.6362     0.04268346       122.6354     0.3849658      3
 4:    4        122.6362     0.04268346       122.6354     0.3849658      4
 5:    5        122.6362     0.04268346       122.6354     0.3849658      5
 6:    6        122.6362     0.04268346       122.6354     0.3849658      6
 7:    7        122.6362     0.04268346       122.6354     0.3849658      7
 8:    8        122.6362     0.04268346       122.6354     0.3849658      8
 9:    9        122.6362     0.04268346       122.6354     0.3849658      9
10:   10        122.6362     0.04268346       122.6354     0.3849658     10
11:   11        122.6362     0.04268346       122.6354     0.3849658     11


texts: How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?

texts: Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) on a single compute node how do we perform the multiplication?


texts: What is the best Keras model for multi-class classification?

texts:
I am working on research, where need to classify one of three event WINNER=(win, draw, lose)

WINNER  LEAGUE  HOME    AWAY    MATCH_HOME  MATCH_DRAW  MATCH_AWAY  MATCH_U2_50 MATCH_O2_50
3         13    550      571          1.86        3.34        4.23       1.66     2.11
3         7     322     334           7.55         4.1         1.4       2.17     1.61

My current model is:

def build_model(input_dim, output_classes):
    model = Sequential()
    model.add(Dense(input_dim=input_dim, output_dim=12, activation=relu))
    model.add(Dropout(0.5))
    model.add(Dense(output_dim=output_classes, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adadelta')
    return model

I am not sure that is the correct one for multi-class classification
What is the best setup for binary classification?

EDIT: #2 - Like that?

model.add(Dense(input_dim=input_dim, output_dim=12, activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(output_dim=output_classes, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adadelta')


texts: What exactly is the "hyperbolic" tanh function used in the context of activation functions?

texts:
I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows

$$
\tanh(x) = \frac{\sinh(x)}{cosh(x)} = \frac{e^x - e^{-x}}{e^x + e^{-x}}
$$

What exactly is the meaning of term "hyperbolic"?
What is the interpretation of the word hyperbolic in this context? How is the equation above derived?


texts:
Good afternoon!
I'm solving one task: train an LSTM to write an example like Shakespeare
I have torch tensor 'predictions' with dimensions (length_sequence, batch_size, num_of_symbols), num_of_symbols - predicted one hot vector and 'answers' tensor with the same dimensions (num_of_symbols - real one hot vector). How can I find loss? I understand how to use MSE or MAE, but can I use CrossEntropyLoss and how?