texts: Guidance for creating a machine learning model for math equation solving

texts:
I do not have any experience with machine learning and have come to need a ML model for this use case.
Using an already existing ML model I am able to extract the formula to calculate charges from different documents.
For eg: Document 1 may have the equation: charges = x*y/25
Document 2 may have the equation: charges= ((a+b)/100)*55
As the formula will change for each document I cannot use a set formula to solve for every document.
The problem I am facing is that the values of x and y in document 1, and the values of a and b in document 2 are not provided in the respective documents.
The terms are defined in each document i.e, x stands for number of shirts and y is the cost per shirt.
I obtain the values for x, y, a and b from another source. As the values are not in the document itself I cannot use any existing ML model that solves extracted equations.
I was thinking of a solution such as using a model to recognize what each of those terms are and replace the variable with the variable name for that value in my program, thereby I would be able to solve the equation.
i.e, replace x in the equation to numShirt (which would be the variable name in my program) and so on.
=> numShirt*costShirt/25
Does this seem like an appropriate approach or are there better methods to do this?
Any guidance is appreciated.


texts:
I was going term by term through the softmax function for the word2vec (SKIP-GRAM) model.  I found  most definition of these functions to be not 'clear'  so I modified the notation to make sure I understand it.

Is the following  formulation correct?

$$P(w_{-t} | w_{t} ; \theta) = softmax(score(w_{-t}, w_t))$$

$$P(w_{-t} | w_{t} ; \theta) = \frac{exp(score(w_{-t}, w_t))}{\sum_{w' \in \theta} exp(score(w', w_{t}))}$$

where:

$w_{-t} =$ context

$w_{t} =$ target word

$score(A,B)$ a measure of similarity between vectors A and B.

$\theta = $ vector representation for all words in vocabulary

In the simplest case:

$$score(A ,B) = A \cdot B$$


texts: Is there a website where people store their pickled models for others to try? E.g. different people might try different ML approaches on the iris dataset, is there a place where I can find/download models others have constructed?


texts: Is it possible to predict using fewer number of features than the number of features that was used in training the model?

texts:
I'm making a model using sklearn.svm.SVC that would predict machine performance (ErrorID).

For the training of the model, I'm using 6 features, namely, EmployeeID, JobID, MachineID, Speed, RunningDateandTime, Meters and passing in ErrorID as labels.

Now for the prediction, I only have RunningDateandTime because I want to predict the performance in the future. But the model won't accept it because the number of features at the time of training and the number of features at the time of prediction is not same.

Is there any way to force the model to predict using only 1 feature while being trained on 5 features?


texts: decision tree vs neural network for boolean function

texts: Which structure is more powerful in terms of expressiveness (i.e. it can represent a given Boolean function, accurately) — a single-layer perceptron or a 2-layer decision tree? (There are 10 features)


texts: Differences between big data, data warehousing, business intelligence and data science?

texts: I know they are four different areas, but I would like to know what are the main differences between those disciplines, and how they are related to each other, if some of them depend on each other, and what is the specific objective of each one.


texts:
for x in num_cols:
    imp = SimpleImputer(missing_values=np.nan, strategy='mean')
    imp.fit(np.array(ds[x]).reshape(-1,1))
    ds[x] = imp.transform(np.array(ds[x]).reshape(-1,1))


texts: Do all autoencoders perform dimensionality reduction

texts: I want to use Convolutional autoencoder to find patterns in data as well as reduce dimensions. Can it be used for this purpose? Moreover, is removal of multicollinear features through autoencoder better than manual feature selection(plotting correlation and stuff)?


texts: Extract the "path" of a data point through a decision tree in sklearn

texts: I'm working with decision trees in python's scikit learn. Unlike many use cases for this, I'm not so much interested in the accuracy of the classifier at this point so much as I am extracting the specific path a data point takes through the tree when I call .predict() on it. Has anyone done this before? I'd like to build a data frame containing ($X_{i}$, path$_{i}$) pairs for use in a down-stream analysis.


texts: How to compare Random Forest with other models

texts:
I am new to Machine Learning and I am trying to undrestand the Out of Bag Error in Random Forests and its use.
Let's say that we have a dataset. First we use the whole dataset (without splitting it) to get a Random Forest and its Out of Bag error. Then we split the dataset, train a Neural Network on the training part and test it on the test part of the dataset.
Can I choose between the two models by comparing the Out of Bag error of the random forest with the total test error of the Neural Network ? Does it make sense ?


texts:
I'm analyzing a single large table (about 50GB 16M rows, 15 columns perhaps more) currently using PostgreSQL 9.4.
Most of my queries doesn't involve more tables.

Queries run time are very large. I'm comparing it to similar queries in Tableau.
I know this comparison is not fair, Tableau extract was designed for this usage.
Nevertheless i think that i'm not the only one looking for such a solution.

What i'm looking for is:

Database (probably relational)
Analytics oriented (no updates\deletes are required)
Fast (like using a tableau extract)
Supports SQL queries
Preferably open-source
Will not require much optimization (i don't want to be responsible for indexes creation, like in Tableau extract creation)

To illustrate: if i had the ability to connect to a Tableau extract locally like to a localhost database and run SQL quires on it, this will be great.


texts: When is Non-Stochastic Global Optimization Preferable or Necessary?

texts:
Background
I'm specifically referring to non-convex black-box optimization problems of the form:
$ \text{min} f(\vec{x})$

$s.t. \ \ a_i\le x_i \le b_i \ \forall i\in \{1,2,...,n\} \ \ \ \text{and}\ \ \ \vec{a},\vec{b}\in \Bbb{R}^n $
Assume any $n$
Of interest are global optimizers that yield good solutions, but the solutions are not necessarily nor provably the global optimum.
Evolutionary Algorithms
Evolutionary algorithms (EAs) are often the go-to optimizers for these types of problems; such methods include: genetic algorithms, particle swarm optimizers, differential evolution, and all the algorithms based on organismal interactions. Pretty much every EA has stochastic components. Whether it's random initialization, or stochasticism in inter-generational (or intra-generational) subroutines such as selection for crossover or random mutations, stochasticism is pretty ubiquitous in the realm of EAs. Almost everything you'd find in this journal or this one would fall under this category.
Non-Stochastic Optimization vs. Deterministic Global Optimization
I am not interested in deterministic global optimizers. Such methods provide some form of probability/confidence or even guarantee that the found solution is indeed the global optimum. These are more commonly seen in discrete/combinatorial optimization, but occassionally some deterministic optimizers become tangentially related when the user has some form of a priori knowledge/assumptions. The preference/necessity of deterministic optimizers is clear, even when they just give associated confidence with the solutions they find. So again, I am not referring to these.
Non-Stochastic Global Optimizers
I only know of a few non-stochastic global optimizers. Probably the most famous are the many variants of direct search (also called pattern search) algorithms. Conceived by Fermi and Metropolis, then popularized by Hooke and Jeeves, and extended to a generalized pattern search (GPS) making heavy use of positive bases as meshes, the direct search algorithms are similar to the classic Nelder-Mead method in that they use a neighborhood of points with an underlying geometric structure to (deterministically) explore the search space. Of course some non-stochastic variants exist as well, including Luus-Jaakola's sampling of a uniformly distributed neighborhood or the more popular mesh adaptive direct search (MADS) and all of its spin-offs.
There are some other non-stochastic global optimizers hiding out there on the internet, like this one, but I have yet to find one that explains the practical significance of the non-stochasticism.

Question
What are some concrete use-cases for a non-stochastic global optimizer as described in the above-mentioned background?
Are there situations where non-stochastic optimization is necessary? Possibly mission-critical optimization, or where you need repeatability? Maybe something medically-oriented? Or for interpretability?
The only example I can think of (coming from an ML/DL background) is a situation where it would be slightly preferred, but certainly not necessary. In particular, we could train an ML model using a non-stochastic optimization algorithm, which would let us observe the effects of the ML model hyperparameters. In other words, eliminating the stochasticism in the optimizer could help interpret/tune the actual ML model hyperparameters as you'd be able to see causes/effects of modifications, where currently there's uncertainty due to the randomness involved in training.


texts: Can feedback loops occur in machine learning that cause the model to become less precise?

texts:
In discussions about ML algorithms, in for instance crime prediction, it is often claimed by non-experts that there are problems with feedback loops causing the model to become biased and give the wrong results.
Basically saying that the model's predictions give more attention to that type of data, and when retraining with the results, the predictions become skewed so even more attention is given to the same data type, and so on.

Is this true?

I would think that retraining the model with new data would make it more precise, regardless of how that data originated.


texts: How to add extra word features other then word Embedding in Recurrent Neural Network model

texts: I am building a deep learning model for NLP. I am pretty comfortable with adding word embedding from word2vec or Glove vectors as extra word features but I wanted to add other word features like POS tag of a word, NER tag of word along with embedding as features. How can I do this. Should I give these word features by concatenating their vector with the word vectors. Or is there some other method. Please suggest.


texts: Getting Scikit-Learn RandomForestClassifier to output Top N results

texts:
I'd like to see the top N results for a RandomForestClassifier prediction, ordered by descending probability.

The answer may be predict_proba, but I have no idea how to interpret the results.

Help appreciated!


texts: Synchronizing Multiple Cameras in Autonomous Driving

texts:
Please forgive the naivity of this question, it's just due to lack of experience.

It goes without saying that self-driving cars have up to 8 cameras and more that do various vision related tasks:

object detection and localization,
2D to 3D depth perception,
semantic segmentation,
and probably more that I have yet to learn about.

My question is on the synchronization of these cameras:

I am assuming there's a hardware sync to ensure all inputs are captured at the "exact same" time.  Is that true?

Is there a way to account in the model / protocol of capture for the fact that the inputs might have some delay in between them?

Even though this question makes me sound very novice, feel free to discuss advanced topics. I have just started reading this paper on depth perception using a single camera and am able to understand it.


texts: Word list as a baseline for measuring a classifier's performance?

texts:
I am working on a simple Naive Bayes classifier that categorizes text messages as either "positive" or "negative". I was told that the simplest baseline to measure the classifier's performance would be to compare it against a word list.
However, I have no clue as to how this might be helpful.
How can a word list be a baseline for a classifier?
P.S. I am new to machine learning and data science, so please don't hate on my question.


texts: Classification - Divide the interval (0 - 1] to lets say 100 classes and use each class to make a calculation

texts:
class-1 represents 0.01, class-i represents 0.01*i, class-100 represents 1.00.

Thus, when the classifier predicts the class-y and it should have predicted class-(y+1) there is a small error so we can accept class-y.

Is there a way to express this behaviour in a neural network? Maybe with a distribution or something?

PS: Not interested in regression.


texts: What are the ways to identify a good attribute test while constructing a decision tree?

texts:
I'm working through a decision tree by hand to learn it. From my research, I have found the following three ways of determining which variables to split on:

Minimum remaining values - The variable with the fewest legal values is chosen
Degree heuristic - The variable with the most constraints on remaining values
Least Constraining value - The variable that rules out the fewest remaining values in the remaining variables

Do I have these right? What are some other ways of determining the splits?


texts: DBSCAN clustering on document [updated]?

texts:
I am new in topic modeling and text clustering domain and I am trying to learn more. I would like to use the DBSCAN to cluster the text data. There are many posts and sources on how to implement the DBSCAN on python such as 1, 2, 3 but either they are too difficult for me to understand or not in python.
I have a CSV data that has userID and message that they wrote as follows:

user.csv (number of csv rows:400 (#message))

userID         messages
112   The car was broken and Kevin fixed it
.
.
.

I know some steps to apply DBSCAN such as:

Remove stop words
Find similarity distance ( I have a code that does the cosine similarity)

I am also aware that sci-kit learn has the demo at 4 but I prefer the manual implementation that I can see what's going on in the code.

It would be great if you can provide your help with code that I can run in my side to learn it.


texts: Approximation of function with neural networks

texts:
I am looking for the relevant techniques to approximate a function on $[0,1]$ with a neural network when the function has a huge amplitude?

Here is an example with a simple neural network. For test = 1 everything works well. For test = 2 then the algo "converges" towards a constant. Why?

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import matplotlib

sample_size = 5000000

test = 1

if test == 1:
    learning_rate = 1e-2
    alpha = 2.
elif test == 2:
    learning_rate = 5e-10
    alpha = 2000000.

xs = np.random.normal(0.,1.,sample_size).reshape(-1,1)
ys = alpha * xs * xs

xs_test = np.arange(-3.,3.,0.01).reshape(-1,1)
ys_test = alpha * xs_test * xs_test

matplotlib.interactive(True)
fig = plt.figure()
ax = fig.gca()
ax.plot(xs_test, ys_test, 'r')
fig.canvas.draw()
fig.canvas.flush_events()

x = tf.placeholder(tf.float32, shape=(None,1))
y = tf.placeholder(tf.float32, shape=(None,1))

batch_size = 50
nb_batches = int(np.ceil(sample_size/batch_size))

with tf.name_scope('nn'):
    hidden = tf.layers.dense(x, 50, name="hidden", activation=tf.nn.elu)
    outcome = tf.layers.dense(hidden, 1, name="output")
with tf.name_scope('loss'):
    loss = tf.reduce_mean(tf.square(outcome - y))
with tf.name_scope('train'):
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    training_op = optimizer.minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(nb_batches):
        indices = slice(i*batch_size, (i+1)*batch_size)
        mini_batch_x = xs[indices]
        mini_batch_y = ys[indices]
        if i % 100 == 0:
            print('loop: ', i, 'loss:', loss.eval(feed_dict = {x: xs_test, y: ys_test}))
            ax.clear()
            ax.plot(xs_test, ys_test, 'r')
            ax.plot(xs_test, outcome.eval(feed_dict = {x: xs_test}), 'b.')
            fig.canvas.draw()
            fig.canvas.flush_events()
        sess.run(training_op, feed_dict={x: mini_batch_x, y: mini_batch_y})


texts: If we are using batch normalization as the first layer, can we forego standard scaling of inputs?

texts: It is common practice to use the standard scaler on the inputs before feeding it to a deep learning architecture. I was wondering whether it is necessary if the first layer is a batch normalization layer.


texts: In Fast R-CNN how are input RoIs mapped to the respective RoIs in the feature map before RoI pooling?

texts:
I've been reading the Fast R-CNN paper

My understanding is that the input to one forward pass is the whole input image plus a list of RoIs (generated by selective search or another region proposal method). Then I understand that on the last convolution layer's feature map, each corresponding RoI gets RoI-pooled.

But I'm having trouble with two concepts:

How is the input RoI mapped to the corresponding RoI in the deep feature map? Each "pixel" in the feature map comes from a very a non-linear function on wide perceptive field over the input, so there's no way of making a 1:1 mapping right?
Disregarding that I'm confused in point 1, once we have a bunch of RoIs in FM and we do the RoI pooling, we have N pooled feature vectors. Do we now run each of these through one FC network one by one? Or do we have N branches of FC networks? (that wouldn't make sense to me)

EDIT

I also just finished reading the Faster R-CNN paper. In the same way, I'm also interested to know about how the proposed regions from RPN map to the input of the RoI pooling in the Fast R-CNN layers. Because actually those proposed regions live in the space of the input image, not in the space of the deep feature map.


texts:
I want to make prediction for quantity of stock that will be sufficient over a period of time i.e from one delivery to another.

Assuming, i want a store to have minimum amount of item between deliveries e.g 30 items, delivery interval can be 7 , 14, 30, days.

Also i have history data of past delivery interval, amount of item delivered and the amount of item available before next delivery.

What will be the best way to relate these variable: delivery interval and on hand quantity to quantity delivered.

also can i predict quantity to be delivered in other to get a given amount of on hand stock before next delivery.


texts: How can I get students to stick with data science?

texts:
I am a chief data scientist at one of the largest corporations in the world. I am constantly approached by employees to teach them more about data science and help them on their learning path. The problem is that no one really seems to stick with it over the long term, everyone just wants the "shortcut" to the sexy data scientist job without realizing the work needed.

So, in effect, how can I get more of my "students" to stick with data science? What other techniques have you all used (if any) to teach people successfully? I am posting this because I'm tossing around the idea of starting a small study group where multiple people will help each other learn data science and I will guide them along. I am hoping that this accountability to their peers (instead of just one-on-one with me) will help them.


texts: Learning the task execution steps for a target goal

texts:
I am trying to model a network which can take :-
Input - A Target Goal (e.g. Go To Cinema Hall)
Output - A sequence of steps (e.g. Go To Bus Stop -> Onboard A Bus -> Drop at Location X -> Walk 100 meters.
My training data is a similar kind of Objective - Task Sequence Mappings.
Can anyone please suggest a suitable approach using DL4J to address this problem and model suitable solutions?


texts: How to use bigrams for a text of sentences?

texts:
I have a text which has many sentences. How can I use nltk.ngrams to process it?

This is my code:

   sequence = nltk.tokenize.word_tokenize(raw)
   bigram = ngrams(sequence,2)
   freq_dist = nltk.FreqDist(bigram)
   prob_dist = nltk.MLEProbDist(freq_dist)
   number_of_bigrams = freq_dist.N()

However, the above code supposes that all sentences are one sequence. But, sentences are separated, and I guess the last word of one sentence is unrelated to the start word of another sentence. How can I create a bigram for such a text? I need also prob_dist and number_of_bigrams which are based on the `freq_dist.

There are similar questions like this https://stackoverflow.com/questions/12821201/what-are-ngram-counts-and-how-to-implement-using-nltk?rq=1 but they are mostly about a sequence of words.


texts: Binary classifier on imbalanced dataset yields weird PR curve

texts:
I have a dataset with ~6M points, 9 features and two classes. The minority class represents just under 2% of the data. The data is first divided into 100 batches and a different classifier is trained for each batch using the rest of the data.

So far I have tried using sklearn's Naive Bayes classifier trained first using all the data, then downsampling, and then SMOTE. Every time I get this weird shape on the PR curve. I am now trying with random forest, but I am not confident. Does this suggest that the features design is bad? Any suggestions on what would be my next step?


texts: What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?

texts:
I am solving for a regression use case using tensorflow's DNNRegressor. For EDA purpose, I referred to this post and used pandas boxplot to plot my numerical predictors and target variable(here, pid demand) and scatter_matrix for plotting the distributions and here are the results :
predictor_target_boxplot ; features_label_pdf_scatter_matrix
.

I need help in interpreting these two plots, specifically on these fronts:

How come the boxplot shows so many points beyond whiskers (~10%), can there be so many outliers in a dataset?
How do I handle those outliers?
Based on the second plot (feature, label pdf), should I normalize my features to exhibit Gaussian distribution? If so, why?


texts: Difference between Convolution and Pooling?

texts: We uses filters in convolution that make feature map with reduced size(maybe), so does pooling. Then how are they different?


texts: Pros and Cons of Python and R for Data Science

texts:
So, let's start out by saying I am NOT asking which is better.

I like both of these languages for data science and I think this is a matter of and rather than or since there is no need to choose one of them. My general opinion is that R has more data science depth while Python has more breadth in terms of diversity of application.

I will start this out with a few of my own scattered opinions.

Python

I tend to use Python as the overarching language (sometimes using it to execute scripts in R) since:

Installed standard on Macs and many *nix setups
Easy to install/set up in environments with security restrictions
Has a great suite of tools whether I want to:
Set up a website
Interact with the Apache suite of tools
Build out pipelines
Build and score predictive models
Develop an application / front-end
Has a larger audience if I get stuck and need to ask questions on here
Debug / Unit test
Auto Document A good example of this was in building a Robot some facial recognition stuff over the summer. Raspbery Pi comes with Python by default, the sensors had Python API's, and a few of the web API's I bumped up against had Python API's available. This would have been a difficult application to take on using R.

R

I like to use R for early data explanation since data table tends to load larger sets into memory more quickly than Pandas and the general environment seems more mature for exploratory stuff. If I need mixed models or something similar that isn't Machine Learning "bread and butter" R is more likely to have what I need.  sklearn is pretty good but doesn't feel quite as polished as what I have in R.

On the data viz side, ggplot2 is obviously better than matplotlib (the html based stuff is tough to share in an environment with heavy IP and security restrictions so I tend to lean towards the simple stuff.)

How do you use them?

How do you interact with these languages? Do you mix them together? Specialize in one? What's your rationale? Strengths/weaknesses?


texts: Is a neural network able to learn to map a completely different feature vector to the same class

texts: Is a neural network (for example a MLPClassifier in Python) able to learn to map a completely (or very) different input feature set to the same output class? Or is it better to work in this case with more than one output class and map these recognized output classes afterwards to the same class manually?


texts: Use prediction as feature for a decision tree

texts:
I'm working at classifying documents according to their content.

First I built a decision tree model that gives 90% of goods results.

Then I tried a TFIDF/SVC approach which also gives 90% of good results.

So now i'd like to combine both. My first thought was to add the prediction of TFIDF/SVC as a feature of the decision tree.

I saw this post about bagging/stacking/boosting. For me, adding the feature to the decision tree is equivalent to stacking. Is that correct ?


texts: How to calculate accuracy, precision and recall, and F1 score for a keras sequential model?

texts:
I want to calculate accuracy, precision and recall, and F1 score for multi-class classification problem. I am using these lines of code mentioned below.

from keras import backend as K
def precision(y_true, y_pred, average='None'):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def recall(y_true, y_pred, average='micro'):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def f1(y_true, y_pred, average='weighted'):
    def recall(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall

    def precision(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision

    precision = precision(y_true, y_pred)
    recall = recall(y_true, y_pred)
    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))

Don't know why the value of recall is 1 for both testing and training.
Please help me to calculate accuracy, precision and recall, and F1 score for multi-class classification using the Keras model.


texts: What's the math for real world back-propagation?

texts:
Considering a simple ANN:

$$x \rightarrow
f=(U_{m\times n}x^T)^T \rightarrow
g = g(f) \rightarrow
h = (V_{p \times m}g^T)^T \rightarrow
L = L(h,y)
$$

where $x\in\mathbb{R}^n$, $U$ and $V$ are matrices, $g$ is the point-wise sigmoid function, $L$ returns a real number indicating the loss by comparing the output $h$ with target $y$, and finally $\rightarrow$ represents data flow.

To minimize $L$ over $U$ and $V$ using gradient descent, we need to know $\frac{\partial L}{\partial U_{ij}}$ and $\frac{\partial L}{\partial V_{ij}}$, I know two ways to do this:

do the differentiation point wise, and having a hard time figuring out how to vectorize it
flatten $U$ and $V$ into a row vector, and use multivariate calculus (takes a vector, yields a vector) to do the differentiation

For the purpose of tutorial or illustration, the above two methods might be suffice, but say if you really want to implement back-prop by hand in the real world, what math will you use to do the derivative?
I mean, is there a branch, or method in meth, that teaches you how to take derivative of vector-valued function of matrices?


texts: Which is better for a beginner: R or Python?

texts:
From what I can tell, R and Python are the two most popular languages for data science.

My question is which one would you recommend for someone just starting out in data science? Does any one have any clear advantages on the other? Is any one easier to learn or does any one have more potential?

Thanks a lot!


texts: How to interpret partial dependence interaction plot for binary classification?

texts: The plot below is an example from pdpbox library https://github.com/SauceCat/PDPbox/blob/master/tutorials/pdpbox_binary_classification.ipynb. The what does 0.900 color saturation to the right mean with respect to age and fare in the below plot? The data is taken from Titanic dataset and target variable is survived or not.


texts:
I have a dataset with a column named Package Size and it has data in two units inches and meters mentioned in words. How do I preprocess the data such that I remove the unnecessary word and also convert them into a single unit?

Example:

Package Size 13 inches4 meters

Package Size(in meters) 0.3302 4


texts: How can I extract bootstrap generated datasets into individual dataframes?

texts:
I am having a bit of trouble understanding Bootstrapping and what/how I can manipulate the bootstrap created dataset.

This is all in R

My original dataset is structured like this:

Time Animal_ID Treatment Volume
0     1          Control   1
2     1          Control   2
4     1          Control   6
0     2          T1        6
2     2          T1        6
4     2          T1        8
0     3          T2        21
2     3          T2        7
4     3          T2        8
0     4          T3        4
2     4          T3        3
4     4          T3        2

I am going to compute a Z-score,mean, and median statistic on the bootstrap generated data sets which is something I can do in the function. However, What I really need as a proof of concepts is a way to extract my bootstrap generated data sets into individual data frames or write them as excel files.

To clarify, I want to showcase a simple example of the statistic I am performing so if I create 5 bootstrap data sets I would need a way to explore them like they were unique individual data sets. I would need to show that this works in concept when looking at R=5 and then I would be able to confidently expand this to R = 100 no extraction required.

Does anyone know how to accomplish this? Or if there are other alternative?


texts:
I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
The problem I observed depends on initial weights. When using sigmoid function it does not matter that much if weights are random numbers in ranges of [0,1] or [-1,1] or [-0.5,0.5]. But when using ReLu the network very often has a huge problem with ever converging when I'm using random weights in range [-1,1]. But when I changed the range of initialization of weights to [-0.5,0.5] it started to work. This only applies to ReLu activation function and I totally don't get it why it won't work for [-1,1]. Shouldn't it be able to converge with any random weights?
Also when I changed initial weights to normal distibution, it has no problem with convergence. I understand that normal distribution should work better and faster than random [-1,1]. What I don't understand is why it can't converge (error remains the same epoch after epoch) with [-1,1] and has no problem with converging with normal distribution... Shouldn't it always be able to converge just slower and faster with different initialization method?
PS. I'm using normal backpropagation with softmax as last layer and MSE as loss function


texts: Digits Localization on Streets View House Numbersm

texts:
I am trying to learn a bit of deep learning playing with the Street View House Numbers data set. I have managed to recognize sequences of digits and I'd like now to train a CNN to localize digits and provide boxes coordinates. The problem is that I have the boxes coordinates related to the original image sizes which are always different.

I have to resize the images in order to have homogenous input for the NN but I don't know hot to transform the boxes coordinates coherently.


texts: Given one language ngram model, how do I compare likelihoods of two texts of different length?

texts:
Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model.

How does one do this? Is it something to do with perplexity?

Would also be glad if someone posted a reference to a reasonable Python implementation of the suggested method [a code snippet or a link to nltk/spacy/whatever method].

Thanks in advance.


texts: How to count number of word embeddings in Gensim Word2Vec model

texts:
I am trying to create a Word2Vec model of the the Pub Med Central corpus using the Gensim library and want to limit the total number of word embeddings to around 1 billion.

I have searched high and low and am unable to find out a) How to count the total number of word-embeddings in a saved model, and b) how to limit the total number of embeddings when training the model (once I hit 1 billion, then stop).

Please forgive my simpleton questions.


texts: Let's say we did some analysis on a network dataset. We have an adjacency matrix which we can use to construct a graph and we can find the nodes with the highest degree's. How would I go about giving a location of these nodes. Would it be okay to refer to the entries of the adjacency matrix (i,j)?


texts: Is the search for a specific n-gram the same like a string search?

texts:
Is the result of a search for a specific n-gram like sherlock+holmes equal to the result of a regex search for "sherlock holmes" in the same document corpus?

So if i read about n-grams for certain words, that's the same like normal string search?

Example:

https://books.google.com/ngrams/

https://books.google.com/ngrams/info


texts:
In below formula

Took this picture from:
https://www.cs.toronto.edu/~duvenaud/cookbook/
what is the difference between x and x_dash ?
Can we use this kernel if x is univariate time series and the time period is not constant (Not deviating a lot but also not exactly constant)
My ultimate goal is fit a periodic function to data using this kernel but is it possible ? If anyone can share any resource on how to fit kernel functions to data I would really appreciate it.


texts: How to use correct weights in linear regression model

texts:
I'm trying to understand can we implement a simple linear regression model.

Let's say we are predicting price currencies. We want to know whether the currency will raise or not.

As i understand, we need to define two vectors for this:

$x=[1,2,3,4,5,6,7,8,9,10,11,12]$ - months

$y=[2.30,2.33,2.29,2.30,2.36,2.40,2.46,2.50,2.48, 2.43,2.38,2.35]$ - average prices.

Let's plot this:

Before sigmoid separator, i will try a simple linear separator.

I'm guessing that at first, I need to choose some random slope (and bias will be the smallest scalar in vector).

$f(x) = 0.026x+2.3$

As we see, the separator is inaccurate, let's try the quadratic cost function at f(1):

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$

$C = \frac{1}{1} \sum_{i=0}^{1}(2.3259999999999996-2.30)^2=0.0006759999999999897$

It seems accurate in the beginning, but it gets worse as it progresses, so somehow i need to improve it.

From my knowledge, the next step is to find the derivative of the function.

Normally, gradient descent algorithm is used for this, but finding a slope of the tangent line is very easy here:

$\frac{dy}{dx} = 0.026$

What is the next step? How can i use this derivative to use proper weights to minimize the cost function?


texts: Algorithm for campaign optimization (Digital Advertising)

texts:
Suppose i am running an Ad thru an Ad exchange A, and i have a set of campaigns running on it. I have

The spend of the campaign.
The budget allocated to it.
The number of hours it took to exhaust it's budget.
Total installs.
Total Revenue.

I want to write an algorithm to divide the total budget x to n campaigns, allocating higher budgets to a campaign delivering higher revenue.


texts:
I was watching Machine Learning A- Z from SuperDataScience but when I was doing below code sample:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, :-1].values

from sklearn.impute import  SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer = imputer.fit(X[:, 1:3])
X[:, 1:3]= imputer.transform(X[:,1:3])

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_features =[0])
X = onehotencoder.fit_transform(X).toarray()

I got this warning message:

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
  If you want the future behaviour and silence this warning, you can specify "categories='auto'".
  In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
    warnings.warn(msg, FutureWarning)
  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
    "use the ColumnTransformer instead.", DeprecationWarning)
  And this below message also

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
  If you want the future behaviour and silence this warning, you can specify "categories='auto'".
  In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
    warnings.warn(msg, FutureWarning)

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
    "use the ColumnTransformer instead.", DeprecationWarning)
  I was reading ColumnTransfer in sklearn website library I didn't understand how to fix these error messages

SampleFile:Data.csv


texts: How to select random data for two different recommender systems?

texts:
The business problem: We have two different vendors that offer personalized recommender engines and want to do A/B testing with them. The recommendation will give the user a personalized offer via a push message on the phone. During the testing period, we should give each provider a dataset with different details regarding the customers (purchase history, in-app events, etc). Each vendor will receive a dataset with identical info but from different clients.

What is the best method to choose the two datasets so that they would be similar in terms of client behaviour?

I assume that giving them random data from our database wouldn't be a rigorous method so one idea that I have in mind is applying dbScan clustering on our database and further randomly picking clients from each cluster - I don't know if this is the best approach. The full database has 200k clients and each dataset should contain 5k clients.

Example: After dbScan clustering there are k=10 clusters so I randomly pick elements from each cluster and split them into Dataset01 and Dataset02.

Any suggestions?


texts: Keras Conv1D model Input_shape value error

texts:
I am not sure why I am receiving this value error. Additionally, I haven't found a tutorial that explicitly talks about the appropriateness of size of filters and kernel. I would appreciate some input and some links. I am predicting the next to the last or last column.

Here is my code:

import pandas as pd
import keras
from keras import Input, layers , Model
from sklearn.model_selection import train_test_split
from keras.layers import Dropout

input_tensor=Input(shape=(6,))
x= layers.Conv1D(filters=128,kernel_size=18, padding='same', activation='relu')(input_tensor)
#x=layers.MaxPooling1D(5)(x)
#x = Dropout(0.5)(x)

x= layers.Conv1D(256,5, activation='relu')(x)
#x = Dropout(0.4)(x)
#x= layers.Dense(4, activation='relu')(x)
#x = Dropout(0.2)(x)
x= layers.Conv1D(256,5, activation='relu')(x)
x=layers.MaxPooling1D(5)(x)

x = Dropout(0.3)(x)
#x= layers.Dense(16,1, activation='relu')(x)

callbacks_list=[
        keras.callbacks.EarlyStopping(monitor='acc',
                                      patience=3,),
                                      keras.callbacks.ModelCheckpoint(
                                              filepath= 'C:/Users/vtodorova/results3/APIFunctional.py',
                                              monitor='accuracy',
                                              save_best_only=True),
                                              keras.callbacks.ReduceLROnPlateau(
                                                      factor=0.1, patience=10,)]

model.compile(optimizer='RMSprop', loss='mse', metrics=['accuracy'])
#callbacks=callbacks_list
model.fit(X1_train, y1_train, epochs=3, batch_size=256, verbose=1)
model.fit(X2_train, y2_train, epochs=3, batch_size=256, verbose=1)

score1=model.evaluate(X1_train, y1_train)
score2=model.evaluate(X2_train, y2_train)

output_tensor=layers.Dense(1)(x)

model=Model(input_tensor, output_tensor)

Here is the head of the data. The names of the columns and the columns got a little misplaced when I copied and pasted, but I hope its ok.

AutoLeadID  leadage  leadstatustypeid  hasCob  hasSRE
0   695746319        5                 1           0                 0
1   695746320        5                 1           0                 0
2   695746321        5                 1           0                 0
3   695746322        5                 1           0                 0
4   695746323        5                 1           0                 0

   hasSRC   hasCRE                                                hasCRPC
0                0                      0                         0
1                0                      0                         0
2                0                      0                         0
3                0                      0                         0
4                0                      0                         0


texts: Why does streamlit effect Keras callback code?

texts:
I use Streamlit for an image classification project in Keras. I defined a lambda callback:

def batch_begin(batch, logs):
    st.write(logs)
    if stop_training:
        model.stop_training = True

testmodelcb = LambdaCallback(on_batch_begin=batch_begin)

I use stop_training parameter to stop training when a button is pressed.

    if st.button('Stop training!'):
        stop_training = True

Problem is this: When I remove the st.write(logs) line, callback does not work.


texts: Neural Networks with a list of undetermined length as input

texts:
I have a list of strings and for each input (each list), there is a target that is again a string. The idea is to let the network learn to generalize from the inputs.
Here is one example:
input_1 = ['abxd', 'abyd', 'abzd']
target_1 = 'ab.d'

input_2 = ['abmdexg', 'abndexg', 'abodexg', 'abmdeyg', 'abndeyg', 'abodeyg']
target_1 = 'ab.de.g'

As you see the length of the lists is not predefined. I want to check if Neural Networks are capable of generalizing more complex structures.


texts: Can a neural network be created that will derive the pythogorean theorem?

texts:
I tried training a neural network with various numbers of hidden lawyers etc where the training set was just pairs of different side length and the answer was length of the hypoteneuse, but I could not get it to give a reliable answer.  Is it possible to do this kind of thing with a neural network?

If so, what would the network look like?  just two input cells and one output cell for the answer?  What should be in the middle?


texts:
I have a dataset similar to the following one:
Feature 1:   Feature2:  Feature3:    Time:    Output:
   10.          5.         0.89.      0.25.       0.1
   10.          5.         1.18.      0.50.       0.2
   10.          5.         2.21.      1.00.       0.24
   20.          5.         0.51.      0.25.       0.05
   20.          5.         0.55.      0.50.       0.10
   20.          5.         1.12.      1.00        0.22
   20.          10.        0.13.      0.25.       0.10
   20.          10.        0.89.      0.50.       0.19
   20.          10.        3.56.      1.00        0.35

While this sample set has only three entries per occurrence, the others in the full dataset have times ranging from 3 to 20 entries.
My objective is to develop a model that can 'predict' the next entry for each of these. For example lets take the first occurrence:
time = [0.25, 0.5, 1.0]
output = [0.1, 0.2, 0.24]

I would want to predict what the value should be at t=1.5 or t=2.0, and then do this again for the next occurrence.
I started by assuming that the the input features and time would be my X, and my single output would be my y.
I know for a fact that the values towards longer time periods are no longer linear, and linear regression showed very disappointing results -  as would be expected.
I tried using an MLP approach which gave better results:
regr = MLPRegressor(activation='tanh', alpha=0.1)

Which gave me an r^2 of roughly 0.55 when comparing expected and predicted values on testing data.
I am hesitant to use statsmodels AutoReg library as, from what i understand, it does not 'learn' the features of other occurrences in the data in order to make predictions on future data.
Do you have any other suggestions on what libraries exist that can help in this situation?
Thanks!


texts: Adding time as a feature with xgboost/random forests

texts:
I am trying to use xgboost for performing some regression and the features I have are rather simple and limited. I have the time stamp associated with some measurements. The measurements are customer counts and the dependent variable is predicting the average customer wait time. There is dependence on time of the day and weekends as well. For example, I notice that wait times are longer in thee afternoon than in the morning and evenings. There is also a dependency on weekday, holiday or weekends. So I also added a boolean variable to indicate whether the given day is a weekend or holiday or not.

The time data I have is in 10 minutes interval and is a regular python time stamp with the date, hours, minutes and second granularity. How should such time features be included with xgboost or random forests or indeed any such modelling paradigms?


texts:
To detect passive voice in sentences, we can use the spacy module to tag each token in the sentence, then build a classifier to classify it as passive or active based on conventional grammatical rules, e.g.

If a clause has all of the following, then it is in the passive voice:

A form of an auxiliary verb (usually be or get)
The past participle of a transitive verb
No direct object
The subject of the verb phrase is the entity undergoing an action or having its state changed

News headlines are written differently. They don't follow conventional grammatical rules. How would one be able to use spacy to detect headlines in passive voice?
For example, if I have this headline, "Church That Defied Coronavirus Restrictions Is Burned to Ground," its part-of-speech tags are ['ROOT', 'dobj', 'compound', 'compound', 'nsubjpass', 'auxpass', 'relcl', 'prep', 'pobj']. We can classify this as passive, based on the fact that it has a nominal subject (passive) and no direct object.
However, if I have this headline, "Rayshard Brooks Fatally Shot By the Atlanta Police," its part-of-speech tags are ['compound', 'ROOT', 'advmod', 'ROOT', 'prep', 'det', 'compound', 'pobj']. We can't use any of the criteria to classify this as passive voice, because the headline has dropped the auxiliary and spacy doesn't detect a nominal subject (passive).


texts: How do you perform multilabel classification that is also a multiclass problem?

texts:
I have a data set in which each row of data belongs to certain classes/labels.

text
class1
class2
class3

text1
pos
neg
na

text2
na
neg
na

text3
na
neu
na

text4
pos
neg
neg

text5
neg
neg
na

There are basically 4 classes with 3 labels each (pos, neg, neu, na). I suppose this is both a multiclass and multilabel problem. How do I approach this? I am using the BinaryRelevance function from multisklearn but the result always returns 2 classes only (0 and 1). What is the correct way to do this?


texts: Epochs and other hyperparameters in Deep Q-Networks

texts:
I was wondering about hyperparameters used in Deep Q-Networks. Considering the use of replay memory and target network, together with the epsilon-greedy policy, are the number of epochs different of 1 useful? In the affirmative case, which are the situations and benefits?
The reason is that we want to explore the environment, thus providing a high number of different samples, but at the same time, my intuition says that it can be useful for fast convergence of the Q-values.


texts: Finding the appropriate CNN Model Architecture and Parameters

texts:
I am currently creating a CNN model that classifies whether the font is Arial, Verdana, Times New Roman and Georgia. All in all there are 16 classes since I considered also detecting whether the font is regular, bold, italics or bold italics. So 4 fonts * 4 styles = 16 classes.

The data that I have used in my training are the following:

 Training data set : 800 image patches of 256 * 256 dimension (50 for each class)
 Validation data set : 320 image patches of 256 * 256 dimension (20 for each class)
 Testing data set : 160 image patches of 256 * 256 dimension (10 for each class)

Below is the sample screenshot of my data:

Below is my initial code:

import numpy as np
import keras
from keras import backend as K
from keras.models import Sequential
from keras.layers import Activation
from keras.layers.core import Dense, Flatten
from keras.optimizers import Adam
from keras.metrics import categorical_crossentropy
from keras.preprocessing.image import ImageDataGenerator
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import *
from matplotlib import pyplot as plt
import itertools
import matplotlib.pyplot as plt
import pickle

image_width = 256
image_height = 256

train_path = 'font_model_data/train'
valid_path =  'font_model_data/valid'
test_path = 'font_model_data/test'

train_batches = ImageDataGenerator().flow_from_directory(train_path, target_size=(image_width, image_height), classes=['1','2','3','4', '5', '6', '7', '8', '9', '10', '11', '12','13', '14', '15', '16'], batch_size = 16)
valid_batches = ImageDataGenerator().flow_from_directory(valid_path, target_size=(image_width, image_height), classes=['1','2','3','4', '5', '6', '7', '8', '9', '10', '11', '12','13', '14', '15', '16'], batch_size = 16)
test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(image_width, image_height), classes=['1','2','3','4', '5', '6', '7', '8', '9', '10', '11', '12','13', '14', '15', '16'], batch_size = 160)

 imgs, labels = next(train_batches)

 #CNN model
 model = Sequential([
     Conv2D(32, (3,3), activation='relu', input_shape=(image_width, image_height, 3)),
     Flatten(),
     Dense(16, activation='softmax'),
 ])

 print(model.summary())

 model.compile(Adam(lr=.0001),loss='categorical_crossentropy', metrics=['accuracy'])
 model.fit_generator(train_batches, steps_per_epoch = 50, validation_data= valid_batches, validation_steps = 20, epochs = 1, verbose = 2)

 model_pickle = open('cnn_font_model.pickle', 'wb')
 pickle.dump(model, model_pickle)
 model_pickle.close()
 print('Training Done.')

 test_imgs, test_labels = next(test_batches)

 predictions = model.predict_generator(test_batches, steps = 160, verbose = 2)
 print(predictions)

Can anyone suggest how will I know the right network architecture and parameters to get the optimal accuracy? How should I start tweaking my network?


texts: Deriving diagonal approximation of Hessian in a neural network

texts:
Consider the equations relating to the diagonal approximation for the hessian matrix for a neural network in "Pattern Recognition and Machine Learning - Christopher Bishop" (on pg. 250 eq. 5.80)

$\frac{\partial^2 E_{n}}{\partial w_{ji}^2} = \frac{\partial^2 E_{n}}{\partial a_{j}^2} z_{i}^2$
$\frac{\partial^2 E_{n}}{\partial a_{j}^2} = ({h}'(a_{j}))^2 \sum_{k} \sum_{k'} w_{k'j}w_{kj}\frac{\partial^2 E_{n}}{\partial a_{k} \partial a_{k'}} + h''(a_{j})\sum_{k}w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$

$a_{j}$ - input to a unit j

$h$ - activation function

$z_i = h(a_i)$ i.e output of a unit i

$w_{kj}$ - edge from j to k

I am interested in how eq. 2 follows from:

$\frac{\partial E_{n}}{\partial a_{j}} = h'(j)\sum_{k} w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$

where k runs through all the units to which unit j directly sends a connection to (eq 5.56, pg 244).

In particular, I am getting a different result for the first term on the RHS of eq. 2, i.e

$({h}'(a_{j}))^2 \sum_{k} \sum_{k'} w_{k'j}w_{kj}\frac{\partial^2 E_{n}}{\partial a_{k} \partial a_{k'}}$

So, eq. 3 should follow from:

$h'(j)\frac{\partial }{\partial a_{j}}\sum_{k} w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$

Following is the way how I tried to derive eq. 3 from eq. 4 using the chain rule of differentiation:

$h'(a_{j})\frac{\partial }{\partial a_{j}}\sum_{k} w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$

$= h'(a_{j})\sum_{k}\frac{\partial a_{k}}{\partial a_{j}} \frac{\partial }{\partial a_{k}}w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$

$=h'(a_{j})\sum_{k}h'(a_{j})w_{kj}\frac{\partial }{\partial a_{k}}w_{kj}\frac{\partial E_{n}}{\partial a_{k}}$

Used:

$(a_k = \sum_jh(a_j)w_{kj}) \wedge (\frac{\partial w_{kj}}{\partial a_j} = 0 )\rightarrow \frac{\partial a_{k}}{\partial a_j} = h'(j)w_{kj}$

So what remains to be shown is:

$\sum_{k}w_{kj}\frac{\partial }{\partial a_{k}}w_{kj}\frac{\partial E_{n}}{\partial a_{k}} = \sum_{k} \sum_{k'} w_{k'j}w_{kj}\frac{\partial^2 E_{n}}{\partial a_{k} \partial a_{k'}}$

which should complete the proof.

QUESTION 1

How to establish eq. 5 ?

One may want to use:

$\frac{\partial a_k} {\partial w_{kj}} = z_j$

QUESTION 2

If

$y = f(x_1, x_2,...)$

then, is it always true that

$\frac{\partial y} {\partial x_k} = 1 / \frac{\partial x_k} {\partial y}$


texts: I want to run PCA on a data set that will be aggregated by country. Should I aggregate the data before or after I standardize the data, and why?

texts: Basically the title asks my question. I have the results of a survey that was filled out by people from different countries. I have been asked to analyze the data using PCA and see what findings I can get on each country from my analysis. Now of course this means I have to aggregate my data, but should I aggregate the data before or after I standardize the data? From the results that I am getting it seems that aggregating first is the right way to go about it, but I am failing to understand why that would be the case.


texts: converting a Tensorflow model for Tensorflow.js

texts:
(Warning, I'm a Tensorflow/ML noob)

I'm trying to convert and load a TensorFlow model into tensorflow.js

The model is saved:

saver.save(sess, checkpoint_path, global_step=model.global_step)

and potentially restored:

saver.restore(session, ckpt.model_checkpoint_path)

This (and similar lines) creates a directory structure:

model_tmp
  checkpoint
  model.ckpt-100.data-00000-of-00001
  model.ckpt-100.index
  model.ckpt-100.meta
  ...
  model.ckpt-500.meta

...So I should be able to convert to a JS model, right?

#TODO --output_node_names='???' \
tensorflowjs_converter \
    --input_format=tf_saved_model \
    --saved_model_tags=serve \
    model_tmp/model.ckpt-500 \
    model_web

Wrong! When I run the script above I get an error:

IOError: SavedModel file does not exist at: model_tmp/model.ckpt-500

I've tried model_tmp, model_tmp/checkpoint but neither of them work either.

Also, I'm not sure what to provide at output_node_names in the converter call.


texts: Detecting abnormal 'cat' behaviour via Supervised Learning

texts:
A few work colleagues and I were looking through a recently replaced 'cat', we had in the workplace. For those of you which are curious, the 'cat' in this context, refers to a specific type of pump known as a cat pump. It's affectionally called the 'kitty' as we only have one of these pumps.

After one routine inspection, I noticed a number of periods where kitty wasn't performing well. This is labelled in the dataset as Cat Anomaly.

(Accessible from here if anyone wishes to analyse further:

https://drive.google.com/open?id=1e0OhzhaSZP9_A3QdaK9-BvWXbRPZacED)

This is what it looks like when it is abnormal:

Essentially, it looks similar to a sinusoidal pattern and is slightly curved.
I also noted that the cluster of points seem to be quite tight when it is an abnormal state.

Using Python's Sklearn Libraries and some good ol fashioned signal processing, I decided to remove some of the noisy clusters so I can see what the signal looks like 'without' the noise. To do this, I have performed a Savitzy-Golay Filter on the data, which has smoothed this out as per below.

I have then used a feature to subtract the values from the 'normal' cat values - 'smoothed' cat values which gives me a rough indication of when the 'clusters' appear.

This has worked acceptably (~90% accuracy) and I can pick up abnormalities to an extent when I feed this feature into my random forest model, but I feel the 'cluster' approach I have used is very clunky and there are much better ways to capture the 'noisiness' of the cat signal as a variable.

So now I'm looking for a better way of recognizing and separating 'abnormal' cat behaviour.

So, as a better feature or variable to capture 'noisy' clusters, does anyone else have any ideas on improvements on how to capture this?


texts: Unsupervised Learning upon a column of dataset (graph shown)

texts:
I’m new into Machine Learning so here I am asking for a sanity check, if the question I am asking is even reasonable.

I have a Dataset of columns, so I want to call one of the columns from the csv using pandas.

Take one of the number from that column of numbers, do some unsupervised learning to determine if this value is an anomaly in this column and it belongs there or not.

The graph below shows, how I would see this process going. I am not sure on what I would do regarding what unsupervised method would be best for this case.


texts: Survival analysis to estimate kanban tasks completion times

texts:
I am working on a problem to estimate task completion time in kanban (project management tool).
While doing EDA, I looked at tasks that are either done or cancelled. In this case, I defined the completion time as the time taken from task creation to done/cancelled.
I noticed I am running into an issue with that definition. I am disregarding tasks that have not been done yet. If we think of "task = done" as "event = 1", this is like throwing away observations with "event = 0" in survival analysis, giving us a biased result.

How should I handle this?
I would also like to get some inputs on how should I approach "done" vs "cancelled"?


texts:
I am using GoogleColab Pro to train the model for image classification. I have configured TensorBoard inside google colab to debug my model but when I run tensorbord command inside google colab it keeps on getting crashed.
I have used the below command to run tensorbord inside colab.
%load_ext tensorboard

%tensorboard --logdir logs


texts:
I have built my machine learning model and now as a next step I wish to make it publicly available. The plan is:

To build the UI by using Streamlit.
To package the application using Docker image.
To deploy the Docker image into the Google Cloud Run environment.
To start the Docker container in Google Cloud Run environment.

But the problem is that I cannot estimate what will be the costs when starting and running the container in Google Cloud Run (I'm still a beginner). How to estimate the worst-case scenario, i.e. what could be the maximum cost that it should generate? There are CPU, memory, requests and networking properties in the Google Cloud Run pricing table but:

How can I know exactly how much of these resources my application will take?
Could it happen that if the application is publicly available and the requests exceed the free limit quota, I could get an astronomical bill.
Can I set some limits over which my bill cannot exceed?


texts: Do you need a virtual machine for data science?

texts:
I am brand new to the field of data science, want to break into it, and there are so many tools out there.  These VMs have a lot of software on them, but I haven't been able to find any side-by-side comparison.

Here's a start from my research, but if someone could tell me that one is objectively more rich-featured, with a larger community of support, and useful to get started then that would help greatly:

datasciencetoolKIT.org -> vm is on vagrant cloud (4 GB) and seems to be more "hip" with R, iPython notebook, and other useful command-line tools (html->txt, json->xml, etc). There is a book being released in August with detail.

datasciencetoolBOX.org -> vm is a vagrant box (24 GB) downloadable from their website. There seems to be more features here, and more literature.


texts: Deploying keras model with Tenserflow Serving giving error

texts:
I have created an classification model with Keras Deep Learning and I am exporting and saving to deploy using Tenserflow  serving but its giving below  error during starting TensorFlow Serving container step .Please suggest.below are step by steps I am doing.

**Error**: Error facing :2019-07-17 07:38:04.575012: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:362] FileSystemStoragePathSource encountered a filesystem access error: /models/sample_test; Input/output error

**Code:**

# model creation code
model = Sequential()
model.add(Dense(60,input_dim=58,kernel_initializer='normal',activation='relu'))
model.add(Dense(30,kernel_initializer='normal',activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])
model.fit(X_train_pca, ytrain_smote,epochs=2, batch_size=10, verbose=1

config = model.get_config()
weights = model.get_weights()
new_model = model.from_config(config)
new_model.set_weights(weights)

# exporting and saving model
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import utils
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import     build_signature_def, predict_signature_def
from tensorflow.contrib.session_bundle import exporter

export_path = 'D:\\featureTools\\deep learning\\Tenserflow Serving\\ABC
builder = saved_model_builder.SavedModelBuilder(export_path)

import keras.backend as K
from keras.models import Model
signature = predict_signature_def(inputs={'transaction': new_model.input},
                                  outputs={'scores': new_model.output})
K.set_learning_phase(0)

with K.get_session() as sess:
    builder.add_meta_graph_and_variables(sess=sess,
                                         tags=[tag_constants.SERVING],
                                         signature_def_map={'predict': signature})
    builder.save()

#Docker Image  Pull Command
docker pull tensorflow/serving

## Start TensorFlow Serving container and open the REST API port
## ABC is folder directory underwhich model file(saved_model.pb) and other folder variables is saved

docker run -t --rm -p 8501:8501 -v "C:\Users\sarvendra.singh\serving\tensorflow_serving\servables\tensorflow\testdata\ABC:/models/sample_test" -e MODEL_NAME=sample_test tensorflow/serving

##Error facing
Etensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:362] FileSystemStoragePathSource encountered a filesystem access error: /models/sample_test; Input/output error


texts: Where can I find a crowdsourced dataset for multi-label classification with individual participant labels?

texts:
I am looking for a dataset that has been crowdsourced, and corresponds to a multiple answer correct per question setting. I need the individual labels given by the participants from the crowd.

I have searched online (such as here), but all I get is datasets where the answers have been aggregated to create a ground truth. I also found sources like this, which provide such a dataset but apply to the single label case.

I would appreciate any help. Thank you.


texts: what does this doc2vec based ML predict?

texts:
I'm trying to understand what does this ML program - which based on doc2vec - predict:

import logging, gensim
from gensim.models.doc2vec import TaggedDocument
from gensim.models import Doc2Vec
import re
import os
import random
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

model = Doc2Vec.load('reviews_model.d2v') # Already trained

sent = []
answer = []
docvec = []

for fname in ['yelp', 'amazon_cells','imdb' ]:
    with open ('sentiment labelled sentences/%s_labelled.txt'% fname, encoding = ('UTF-8')) as f:
        for i , line in enumerate(f):
            line_split = line.strip().split('\t')
            sent.append(line_split[0])
            words = extract_word(line_split[0])
            answer.append(int(line_split[1]))
            docvec.append(model.infer_vector(words, steps=10))
            print (str(docvec) +  'time')

combined = list(zip(sent, docvec, answer))
random.shuffle(combined)
sent , docvec, answer= zip(*combined)

 clf = KNeighborsClassifier(n_neighbors=9)
 score = cross_val_score(clf, docvec, answer, cv =5)

 print (str(np.mean(score)) + str(np.std(score)) )

The output be something like:

0.7903333333333334

So what does it actually mean it's 79% correct? correct of predicting what exactly?

P.S: the documents learned are positive and negative reviews.


texts: Feature engineering for categorical variables

texts:
I have some categorical variables in my dataset for a regression problem.

1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary encoding will treat all three values as a different class. How can I use it efficiently retaining the information that 'Girls&Boys' include both? Is breaking into two separate columns for girls and boys is the only approach in such cases?

2) Age range: (18-35, 35-50 etc.) I have broken it down to 2 columns of age_min and age_max. Is there any better way to use such kind of features with values in range?

3) Range of percentage: It can take only 5 values (0, 1, 1-5, 5-10, 10). How should I use this variable for training my model? Here I cannot break it down into 2 columns (like with age) because of "1" and "10" fixed values. How to treat variables with fixed as well as range values?

4) Similarity: It can also take 5 values (0, 1, 1-5, 5-10, 10) but it also has an "auto" option. That means if it is "auto" it can take any random value which we do not know about. How should I incorporate that?
Should I create a separate column for "whether it is auto or not"? Then, in the original similarity column, what value should I put for observations with "auto". I cannot put "0" as it is already a value and I am sure that "auto" won't be 0. Does putting it "None" will create a difference?
How to treat unknown values?


texts: Using t-SNE to track progress of a word vector embedding model. Pitfalls?

texts:
I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer)

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

EDIT: I have since realised that the apparent lack of clumping could be due to the data itself. MNIST separates out cleanly because there are generally no weird glyphs that look like halfway mutations between numbers. My dataset (twitter sentiment, 1.6 million tweets) is, for a lack of a better word, filled with unclassifiable drivel, and it seems entirely probable that the homogeneous forest of points in the centre of the plot represents these sorts of tweets.


texts: The right Machine Learning solution for this problem

texts:
Throughout my understanding about using Machine Learning, there was always a column whose name Target or Label, that was causing us to call methods Supervised learning. Now I am dealing with not a supervised or an unsupervised procedure. I do not know what exactly is. I have some data-frames that each of which has 3 columns. The columns have a relation with each other but it does not matter. I am looking for a way to show the effect of samples on each other. In the sense that, in my topic, theoretically, it has been proved that in a specific situation, some samples are more important than the others. Now I need to show it using Machine Learning. I have generated the required data for having that situation but I do not know how to find the effect of samples on each other.? I could have shown some differences on samples but couldn't find any meaningful scenario to be used in machine learning. To clarify, I have defined a value which a computation between each column value and I would like to know in the computation of this value in a specific index, which sample are playing a more important role.  Now The question is that,
**HOW TO FIND THE EFFECT OF SAMPLES ON EACH OTHER USING SIMPLE OR ADVANCED TECHNIQUES?**

To understand better let's see some sort of data here.
an example of data frame is below:
S   C     E
0   0.2   1e-2
1  -0.15  2e-2
2   0.24  2e-3
3  -0.1   3e-1
4   0.3   2e-1

the third column is a combination of the first 2 columns(There is the equation of combination) Now I need to know for example for the E[2], what is the effect of c[0],c[1],c[2],c[4].


texts:
Here is a nice definition of R-squared that I have found on the internet.

  R-squared is a statistical measure of how close the data are to the
  fitted regression line. It is also known as the coefficient of
  determination, or the coefficient of multiple determination for
  multiple regression.

  The definition of R-squared is fairly straight-forward; it is the
  percentage of the response variable variation that is explained by a
  linear model. Or:

  R-squared = Explained variation / Total variation

  R-squared is always between 0 and 100%:

  0% indicates that the model explains none of the variability of the
  response data around its mean. 100% indicates that the model explains
  all the variability of the response data around its mean. In general,
  the higher the R-squared, the better the model fits your data.
  However, there are important conditions for this guideline that I’ll
  talk about both in this post and my next post.

Could someone please explain to me what does "variability of the response data around its mean" stand for, and also "it is the percentage of the response variable variation that is explained by linear model"? I am having trouble understanding these concepts.


texts: Why is there such a mismatch between the Model's predicted probability and theoretical probability in logistic regression?


texts: Support Vector Classification kernels ‘linear’, ‘poly’, ‘rbf’ has all same score

texts:
I build a classification model based on SVM and getting same results after running different kernels. Can you please let me know if is mistake ? also recall for all are identical. Thank you for help.

Adding the location for the notebook and data.
SVC repo with the notebook and data


texts: How to measure correlation between several categorical features and a numerical label in Python?

texts:
I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product Owner". Then I also have some numerical categories such as "Hour of day when started the job", "Number of sub products" and more. In total I have around 16 different categories, where each category can take around 10 different values.

I will now try and train a regression model to see if I can predict the lead time(time it takes for the product to go through the pipeline) based on these features.

I would want to see if there are any of these features which are more correlated with the lead time than others. As I have understod it I have to seperate the numerical and categorical features and perform tests seperately on them. For numerical variables I have read about pearsonr and for correlating categorical and numerical variables I have read about ANOVA but I can't seem to find any way of implementing ANOVA in Python. Has anyone any experience with this? And if not, any other tips on how to do this?

Example data frame

        product_category     product_owner    #sub_products     hour     lead_Time_min
              A                  Bill             5             13         123
              B                  Lisa             14            19          40
              B                  Lisa             2             16          20
              D                  Eric             1             11          10
              C                  Ben              7             11          4
              C                  Lisa             14            10          25
              B                  Lisa             2             19          252


texts:
I have a CNN model using cifar -10 dataset. The model was built using Keras (Tensorflow).

Now based on this model, I have to generate an image embedding (vector). That means - an input image comes and I have to output the embedding vector of that image.

I am not sure how to do that. This is not a straight forward prediction/classification output. Rather I have to output the embedding of the input image (which is off couse the predicted embedding but an embedding vector nonetheless).

Any suggestion?


texts: Keras: apply masking to non-sequential data

texts:
I am trying to do multi-label classification via LSTM in Keras.

As a simple example, suppose I have sequences of length 4. Each sequence element is a 3x3 numpy array with values 0, 1, or -1, like this:

-1  1 -1
 0 -1  0
 1  0 -1

The -1 values indicate that this position of the numpy array should be ignored (no weight calculation, no associated gradient update, no contribution to the cost function, etc). The positions of the -1 values are the same in each 3x3 numpy array. The 0/1 values are the labels for each position.

How can I tell Keras to ignore the values -1 in my input? I have tried one-hot encoding each 0/1/-1 value and using a mask that way (since masking does not accept 1D input). I also saw that masking is applied to dimension 1 of the (num_samples, num_timesteps, num_features) input shape, which is not what I want: I want it applied to the features (the 3x3 numpy arrays).

Does the below code do what I want it to do?

from keras.layers import Input, Masking, LSTM, Dense, Flatten
from keras.models import Model
import numpy as np
import tensorflow as tf
from keras import backend as K
from keras.utils import to_categorical
from keras.optimizers import adam

#Creating some sample data

#Matrix has size 3*3, values -1, 0, 1
X = np.random.rand(3, 3).flatten()
X[X < 0.2] = 0.
X[(X >= 0.2) & (X < 0.4)] = 1.
X[X >= 0.4] = -1

X2 = np.random.rand(4, 3*3)
for i in range(X2.shape[0]):
    X2[i,:][(X==-1.)] = -1.
    X2[i,:][(X !=-1.)] = 0.
    tobeone = int(len(np.where(X2[i,:] == 0.)[0])*0.5)
    selected_ones = np.random.choice(np.where(X2[i,:] == 0.)[0], tobeone)
    X2[i,selected_ones] = 1.

X = np.reshape(X, ((1, 3*3)))
X_new = np.concatenate((X, X2), axis=0)
Y = X_new[4,:]
X = X_new[:4,:]

#Categories are 0, 1, -1 one-hot encoded
X = to_categorical(X, num_classes=3)
X = np.reshape(X, (4, 3, 3*3)) #X needs to be shape (num_samples, num_timesteps, num_features), not sure if I have this right

Y = to_categorical(Y, num_classes=3)
y_true = np.reshape(Y, (1, 3, 3*3)) #predicting a single timestep

#Building the model

mask_val = np.tile([0,0,1], 3*3).reshape((3, 3*3))
input_tensor = Input(shape=(3, 3*3))
masking_input = Masking(mask_value=mask_val)(input_tensor)
lstm = LSTM(1, return_sequences=True)(masking_input)
output = Dense(3*3, activation='sigmoid')(lstm)

model = Model(input_tensor, output)
model.compile(loss='categorical_crossentropy', optimizer='adam')
print(model.summary())

y_pred = model.predict(X)

#See if the model's loss is the same as the unmasked loss (shouldn't be)
print(model.evaluate(np.expand_dims(X[0,:,:], axis=0), y_true))


texts:
Does Apche Kafka support sockets-based communication?

Example: After running both Kafka producer and consumer.if I send a message through putty(TCP/IP client), will the consumer receive it or not?


texts: Does reinforcement learning require the help of other learning algorithms?

texts:
Can't reinforcement learning be used without the help of other learning algorithms like SVM and MLP back propagation? I consulted two papers:

Paper 1
Paper 2

both have used other machine learning methods in the inner loop.


texts: Sklearn linear regression: adding new feature increases training time dramatically

texts:
As the title states, when I add another feature to the previous 200+ ones, the algorithm starts to train about 10 minutes, while earlier it trained only for a minute.

Can anybody please explain me why this can happen?


texts: Minimizing Costfunction in a Feedforward MLP

texts:
I made a sweep on a feedforward MLP changing number of layers and neurons per layer, in order to see an effect on the costfunction. Costfunction = 0.5 (Trainingoutput - Modeloutput)^2.
For the Training samples I use  minibatches. It runs for 100k epochs (1Mio doesnt improve it).
I can see that the ANN is training and the Cost gets reduced, but for all tested setups the Costfunction seems to converge at the same value at 8.
Number of hidden Layers | Neurons per Layer | MAE after 10k epochs | MAE after 100k epochs

Outputvalues reach from 2 to 100, so a mean error of 8 is quite high
Any ideas why the costfunction won't minimize any further than ~8?
EDIT: The Inputdata consists of 36 samples, is it possibly not enough sample material, to get a good fit?


texts: How should I read the following heatmap?

texts:
I've been playing around with the linear regression, and I was thought that before commencing it's always good to plot an heatmap to see whether there are features that somehow is worth testing for their significance/relationship.
Would you agree with the above?

With that being said, after having refined a tiny bit my data, I obtained the following heatmap:

As you can see I now have two features completely blank. I though in the beginning to some sort of rendering glitch, which is not the case.

In reviewing the dataset, I found out that the features have always the same value across the dataset.

I'd assume this is not being printed out due to the fact this feature has became statistically significant for every feature.

Would you say this is a correct way of referring to this? Or is there any other way to express this concept?

This is however not the case for the other feature, which contains a mix of numerical value, hence I'm not able to explain.

Can you please advice?


texts: Why does word2vec try to maximize the dot product of the center word vector and context?

texts:
I am learning about the maths behind word2vec from this tutorial.
u are the embeddings for the center word and v for the context. It appears that this dot product is to be maximized. Why the context words are not necessarily similar to the center. Sure they appear together, but it does not mean they are synonyms.


texts: How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)?

texts:
I'm developing a participatory social environmental diagnostic. To do this, I'm using primary (qualitative data from interviews with stakeholders) and secondary data (local socioeconomic data).

From this data, I distribute the identified factors in a SWOT matrix - identifying opportunities, strengths, weaknesses and threats that could afftect the local community. After that, each factor is given a scale from 0 to 2, being:

0 - Null or small impact
1 - Medium impact
2 - High impact

In this way, I have  this kind of data:

Ok. Now I build a scale with this formula:

= (((strengths + opportunities) - (weaknesses + threats)) / ((strengths + opportunities) + (weaknesses + threats)) * 2

OBS: basically opportunities and strengths are positive values and threats and weaknesses are negative, so I subtract the negative values from the positive ones. The division by the sum of all values are made to have the statistical universe (population, in this case the maximum value). The multiplication by 2 are made to have the possibility to create a range from -200% to 200%, but it could be -100% to 100% too ;)

Now I have a scale ranging from -200% to 200% that I interpret in this way:

Very unfavorable (-200% to -100%)
Unfavorable (-100% to -29%)
Balance (-30% to 30%)
Favorable (31% to 99%)
Very favorable (100% to 200%)

OBS: This scale was used at this website (https://en.luz.vc/template/swot-analysis-excel-tool-template/)

Now, what I need is to convert this to 0 to 10 scale so I apply this formula:

I've discover this  formula here: (https://stats.stackexchange.com/questions/25894/changing-the-scale-of-a-variable-to-0-100)

Finally, my question is how can I achieve this score (from 0 to 10) right from the beginning, without the need to make the conversion from the first scale?

OBS: The first scale give a percentage (-200% to 200%) and the second one a decimal number (0 to 10).

Warm regards ;)


texts: Should we have only binary labels for LSTM?

texts:
Is it possible to have non-binary labels for LSTM? I mean an array like

[ 100 120 140 20 50 70]

Instead of

[1 0 1 0 0 1 1]

for example! Isn't this opposite of LSTM's essence and doesn't reduce it's performance?


texts: NLP Feature creation from phrase matching

texts: I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with only around 11% being classed as "Raised". So far, the Random Forest classifier is providing the highest level of accuracy but the True Positive Rate/Recall is sitting at around 40% and I can't seem to increase upon this. I have been provided with a list of phrases that should they be contained in the email content, then in all likelihood a ticket needs raising. Looking for some tips as to the best method to create a new feature based on phrase matching? Has anyone any experience in the best methods for doing this?


texts: Why do you need to update the number of boosting rounds each time you update a parameter in xgboost?

texts:
I have been reading material that suggesting that, after each grid search you do on a single parameter (e.g say on learning rate), you should update the number of boosting rounds afterwards. This is very expensive (time). I also intend to use GridSearchCV, and from my understanding, you cannot use early stopping rounds and GridSearchCV simultaneously.
Why is it necessary to update the number of boosting rounds each time you update a parameter?


texts: Finding best cluster after running K-Means clustering

texts:
I have bunch of text which I want to segregate based on semantic similarity. Running through K-Means, I was able to divide the complete text into different clusters.

However, I still need to find which cluster contains most similar text. For e.g :-

Cluster 1 :-
'Hey there, how are you'.
'How are you dear'.
'I haven't seen you for past three days. How are you'.

Cluster 2 :-
'As per survey conducted, smoking kills more people than ...'.
'Avoid smoking in public places.'.
'People more likely to avoid smoking if they ...'.
'As per survey conducted, people most likely chose democrats'.

Cluster 1 should rank above Cluster 2 as it has more semantic similarity among text contained within.


texts: Intent Classification in Question Answering

texts:
I am trying to write a question answer intent classification program.

My task is given a set of unlabelled question and answers, I have
to write a program where I may group all the similar questions
and identify their answers.

Once the answers for a group of similar questions are done, I have
to identify the intent or focus of answers.

For example, if I have a set of questions like:

a) Q: where is Texas? A : It is in USA.
b) Q: where is California? A: It is in USA.
c) Q: where is NASA? A: It is in USA.
d) Q: who is Queen Elizabeth II? A: Queen of England.
e) Q: who is Donald Trump? A : President of USA.

Thus,
I am trying to group questions a, b & c as Location oriented question,
and d & e as Official/Person oriented question.

To solve this problem,

I am trying to use a standard classifier, and as it identifies 'It is in USA' as the class,
I am trying to tag it tag it as "It/NA is/NA in/NA USA/LOC" to identify intent/focus
of answer as Location.

I am using a standard classifier like Naive Bayes and a standard Hidden Markov Model
based tagger.

The result is more or less fine.
I am using two training set one for classifier and the other for tagging.

If any one of the esteemed scientists may kindly suggest how I am trying
to solve the problem?

Here, Q means question, A means answer.

Apology for cross-posting


texts:
Say I have a tagging system on an electrical circuit:

Name          Description
-------       --------------
BT104         Battery. Power source
SW104         Circuit switch
LBLB-F104     Fluorescent light bulb
LBLB104       Light bulb
...           ...

I have a hundreds of tags created by people who should have followed my naming conventions but sometimes they make mistakes add unnecessary additional characters to tag names (i.e. BTwq104 etc.).

Up until now I used regular expressions, that I built over time whilst observing various inconsistencies that users introduce whilst naming different parts of their circuits, to parse the names and tell me what the different elements are. For example: name 'BT104' would tell me it's a battery on circuit 104.

I would like to investigate or use a machine learning technique to identify what a tag name is (same way I used regular expressions). Any suggestions and approaches are welcome.

So far I tried Named-entity recognition suggested technique "Bag of words". Followed a few tutorials here and here (latter being the most useful in learning). None of them produced wanted results if any. I think that "Bag of words" are mostly used for real word rather than made up words.


texts: What is the difference between different batch_sizes in Keras Sequential models?

texts:
I am interested to know, what happens when I choose batch_size=1 or batch_size=1000 or any other numbers in Keras.lstm.mode.fit() function for example when I am configuring batch_input_shape? Does this make effect on my final result and changing them?

I need for future prediction using batch_size=1 but affraid of getting bad/wrong result!

May someone explains the important effects of choosing batch_size from 1 to N numbers?


texts: What supervised machine learning model can be used to generate a scorecard-like result?

texts:
A scorecard is typically used in Credit Application. One very common model for developing a credit scorecard is logistic regression since it has well-defined probabilities.

Apart from logistic regression, is there any model that can be used in the scorecard?

For example, I don't know whether Support Vector Machine can be used since it only outputs a decision boundary.

More on the scorecard:

Features are assigned with weightings
All features are categorical
The sum of weightings of all features with value True is the total score (like a checklist)
There will be a cutoff point to classify good/bad (label, +1,-1)
How far from the cutoff point represents probabilities.


texts: Pandas DataFrame: Aggregating multi-level groups by matching keys

texts:
I have some data that looks like this;

data.head()

    stock  date  binNum    volume
0  stock0  d120       2  249500.0
1  stock0  d120       3   81500.0
2  stock0  d120       4   79000.0
3  stock0  d120       5  244000.0
4  stock0  d120       6  175000.0

I can get the average volume for each bin across all days for a particular stock with the following code;

stock0 = data[(data['stock'] == 'stock0')]
binGroups = stock0[['binNum', 'volume']].groupby('binNum', sort=False)
stock0vol = binGroups.aggregate({'volume': np.mean}).reset_index()
stock0vol.head()

   binNum         volume
0       2  174095.238095
1       3  100428.571429
2       4   79880.952381
3       5   73642.857143
4       6   69761.904762

I would like to apply this to all stocks. The result will be a table with a stock column but no date column (since it is an aggregation across all days). Something like this;

    stock  binNum         volume
0  stock0       2  174095.238095
1  stock0       3  100428.571429
2  stock0       4   79880.952381
3  stock0       5   73642.857143
4  stock0       6   69761.904762

I can do this by putting the above code in a loop and bolting on the rows one by one, but I am sure there is a more elegant way to do it via grouping and aggregation. Can anyone shed some light please?


texts: Best statistic books which have solutions

texts:
I saw some questions which are asked about books in data science, so I think my question is not out of topic.

I am looking for a book in statistic which has complete answers for exercises.
The solutions helps me to make sure, that i learned well the concepts.

I have review the book open intro statistic, but it is not easy to find solutions for it.

Can you suggest me any book?


texts: How best to use the resale transaction year in predicting housing prices?

texts:
I'm looking into the classic problem of predicting apartment prices (resale market) based on the their type, size, location, etc. Pretty straightforward and Linear Regression or Regression Trees give some first decent result -- I'm still more in the exploratory phase.
However, I'm not sure how to best incorporate the year of the resale transaction, since there are clear long-term trends over the years. Right now, I just keep it as a feature, which seems to be a valid approach. I just wonder if there might be alternative approaches. For example, I also have to overall price movement on a quarterly basis. So I assume I could adjust the each resale price based on those trends and ignore the year as feature. Would this make sense?
What are other approaches? (Again, I'm not even sure if this an issue at all.)


texts: Binary Classifier making only one prediction

texts:
tl;dr I'm building a binary classifier that always eventually predicts all "0" or all "1" after some number of epochs and I'm looking for possible reasons why/how to proceed.

Below is all just more details on my approach and thoughts:

What I'm doing: I am building a binary classification basic feed-forward neural network for signal processing with tensorflow.

I pre-process my data by separating it into "windows" of length window_size points (say like 50), and that window/example can be considered either a "1" (positive) or a "0" (negative). The official, manually annotated labels are located at specific points (e.g. 403, 875, 1450, etc.) and are relatively evenly spaced. I decided that my example/window is positive if the window contains a label, negative otherwise. Multiple examples are generated from moving the window by some stride_length (say 1, 2, or 10 points, etc.), until my moving window reaches the end of the signal.

For the training data, since most of my examples are negative, I've tried to normalize the ratio of neg to pos examples so it's close to 1:1. I do this by deleting a calculated ratio of negative examples. For test data I skip this last step, but everything else is the same.

I run it through a 3 layer NN with RELU's on hidden layers and sigmoid on the sole output, with <0.5 predicting a negative and >0.5 predicting a positive. Cost is sum of (label - prediction)^2, using AdamOptimizer at default LR (0.001).

My issue:
No matter what parameters I adjust, my NN seems to predict everything as either negative or positive after a certain number of epochs (sometimes even immediately at the first epoch). It's pretty random too. I'll run the same thing multiple times and I'll get something different every time, about half of the predicting all positive, and the other half all negative.

Things I have tried/thought about:

I shuffled my training examples before training my model, with not much improvement.
I cut down on my negative examples as I stated above, since I had this problem before I cut down (was predicting basically all negative immediately). But now that I have close to a 1:1 ratio of pos/neg examples, it's flip-flopping between predicting all negative and all positive, so it's extremely sensitive to this ratio.
I originally had a softmax_cross_entropy_with_logits cost function with one-hot encoding, so 0 1 was positive and 1 0 was negative. (I thought maybe the cost function was rewarding extreme predictions, so I switched to sum of squares which is harsher)
I'm thinking of trying batch normalization and dropout, but I don't expect those to fix my problem.
I think the problem might be in how I am pre-processing the data. My thought process is this: if the ratio of pos/neg examples is so important that it is making my model predict all positive or all negative, then maybe there is no real trend in my data (which is wrong, since anyone with eyes can see the difference). My thought was that because I have a moving window, sometimes the label is near the left edge of my moving window and sometimes the right edge, which means the actual region of interest is getting input into different neurons each time, which confuses the neurons so it feels like there is no real trend.
However, I took a look at the weights and biases and they seem to be stabilizing after a few epochs. Shouldn't they still be jumping around wildly if there really is no trend in the data? Maybe that is wrong.
I was thinking about testing my hypothesis out by running the left-leaning label examples and right-leaning label examples through different neural networks and seeing if I get better results, but maybe it's not worth the effort it will take.

I've been trying this for a while and can't seem to think of a good explanation or approach. I would really appreciate any help I can get on this, and I'm happy to provide more information. Thank you!


texts: Object coordinate detection with capsNet

texts: Where can I find the implementation of object coordinate extraction with capsule networks in keras or pytorch?


texts: Train a model using multiple data sources

texts:
I have to train a classification model to predict if a customer will buy a product or not. I have multiple (eg. 3 or 4) data sources. The variable distributions among the different data sources is quite different (eg. in the first one I have a vast majority of young people, while in the second one there are many adults). When the model will be used in production, the test dataset will be made of records from only one data sources (I don't know in advance which one).
My question is: which is the best way to combine these different sources? The data are not so much, so I cannot train an independent model for each data source. Can I just concatenate the data frame or do I have to perform some other kind of step?
Thank you!