texts: Merging Refinement Step in the BIRCH clustering algorithm

texts:
I'm trying to understand the BIRCH algorithm, more specifically, the workings of a Clustering Feature tree. The paper describes 'A Merging Refinement' step when inserting a new entry into the CF tree, as follows.

If they are not the pair corresponding to the split, we try to merge them and the corresponding two child nodes. If there are more entries in the two child nodes than one page can hold, we split the merging result again.If I understand correctly, we pick the two closest entries that are not the seeds. We can merge the cluster features using the additivity theorem, for example, if the pair $i$, $j$ are to be merged to give the new merged feature $k$ we have $ CF_k = CF_i + CF_j$. However, I do not understand how to deal with the "two child nodes". I'm thinking that if they were just leaf nodes then we can try and merge them into one or two nodes depending whether or not all the CFs fit into one node (page size).

What happens when this is not the case, when the "two child nodes" have further children themselves?

Here is a link to the paper I am referencing.

"If they are not the pair corresponding to the split, we try to merge them and the corresponding two child nodes. If there are more entries in the two child nodes than one page can hold, we split the merging result again."


texts:
I read this article: https://www.tensorflow.org/tutorials/text/image_captioning
And i understand main concepts of this article.
We have label and image in our dataset. We pass image to our model and want to get next single word for image description and for next step give to decoder previous predicted word and cnn_encoder's output. And continue this step 'till we get  token. But i don't understand training part. https://www.tensorflow.org/tutorials/text/image_captioning#training I realize that my question isn't good, but i really want to find out what's going on. Please, help me. If i don't understand some concepts for that, say it to me, i'll watch some videos, read some tutorials.


texts:
I have installed Anaconda, but every time I open Terminal I have to go give the command:

export PATH=~/anaconda3/bin:$PATH

How can I fix this issue?


texts: Devices behavior in one continuous variable vs events rate

texts:
I have devices on which I have time series data of one continuous variable. I have to evaluate the relation between the profile of that variable on those devices and "events".

Those events are given in terms of occurrences on a time period.

My first intention is to make clusters of similar behavior of that variable and compare those clusters with the low/middle/high events rates.

I was thinking about doing a K-means with the min, max, quartilles, mean, normal q-q p value, Kurstosis, etc. as dimensions, but I don't think it's a good idea because:

Those dimensions are not independant
It's "losing" data and so potentially losing classification potential

Do you have some suggestions to group similar devices together?

Also, do you have other ideas to establish that relationship?

Context:

python3 with the scipy stack
~ 3000 devices and hundreds of thousands of data per day; 5 months to consider


texts: Where does the evaluation speed advantage of Transformer-XL come from?

texts:
The Transformer-XL paper claims an advantage in evaluation speed 363x-1874x than that of a baseline Transformer model.
However, I do not understand where this massive difference comes from.
Although states from the previous segments can be cached, I do not see how the model obviates the need for autoregressively generating tokens one by one, which the paper seems to suggest. If the input is [1, 2, 3, 4, 0, 0, 0, 0], and we want to predict the values at the masked (zero) tokens, relying on the output logits at these positions would not be sufficient, as the model cannot attend to its own prediction for the first masked position when predicting the second masked position. Therefore, we need to generate the first masked position first - how do we then allow the model to attend to it without re-running the whole inference process?
I understood that the cached hidden states come from the previous segment, but that comes as a memory input and I don't see how it speeds up the generation of the current segment.
Is there some way we can cache the current hidden states and then cleverly propagate just the changed fields for the token that was added? This seems hopeless to me because of the softmax layer in dot product attention - as soon as one of its input "scores" is changed, its whole output changes due to the normalization that it performs. That would then invalidate the full contents of all subsequent layers and cause them to be recomputed. What am I missing?
I could not find the code for sampling from the model in the Transformer-XL repo so could not study it in more detail. I feel like I'm deeply confused about some of this.


texts:
The data produced from my humidity sensor at different temperature is as follows (forget about purpose of frequency and its source)

on re-arrangement (temp in deg C)

In this case temperature and frequency are independent variables and humidity is dependent on both of them. Humidity seems directly proportional to frequency if analyzed for humidity Vs Frequency and temperature seems inversely proportional to frequency while analyzed Temp Vs Frequency.

I need to be able to make an equation which gives Humidity upon solving with known temperature and frequency. For example if I input F = 8179 and T = 20.5, H should be near to 42%.

I tried polyfit and interp2 from MATLAB for frequency vs humidity and got much errors and tried least square method manually but ended getting nothing near. Specifically I do not have idea on modeling for 3 Variable equation. What may be the way for modeling 3 variable equations upon this small sets of data. Thanks.


texts: What are contexts where a perceptron model could be defined?

texts:
per this post, a perceptron could use the logit function as the activation function.

per wiki

  In the context of neural networks, a perceptron is an artificial
  neuron using the Heaviside step function as the activation function.

so, a perceptron does no have to use Heaviside step function as the activation function.

besides neural networks, what are other contexts where a perceptron is defined and would use other activation functions.


texts: Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

texts: If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model in the beginning and suppress parts of it later?


texts: Categorization of Natural Language Processing Tasks

texts:
Problem

I am currently learning basics of natural language processing. I see many tasks in this area is assigning labels to each individual words in the sentence, including POS tagging, chunking, named entity recognition and semantic role labeling.

My question is

Are there other tasks that do not involve assign labels to each word in the sentence. One thing I could think of is sentiment analysis, which is assigning label to entire sentence/document or different aspects of sentence/document.
Is there some survey that is recommended to read that summarize the general tasks in natural language processing.


texts: How does SVD actually provide the recommendations? I seem to get conflicting answers

texts:
I am reading a text book that basically says the following:
Given a matrix A where A is USERS x ITEMS we can use SVD to decompose the matrix into:

$$A = U \times \Sigma \times V^T$$

Then we can take the first $n$ columns of these matrices to get:
$$A \approx U_n \times \Sigma_n \times V_n^T$$

Then say we have a USER vector, $u$, that has ratings for items, we find out where $u$ is position in the $n$-dimensional space:

$$u_{nd} = u \times U_n \times\Sigma_{n}^{-1}$$

And we can use the position of $u_{nd}$ to discover what items and users are similar to $u_{nd}$ by using a method like cosine similarity.

So this all makes sense to me (I think). Pretty straight forward.

However if I go to this tutorial, and go to the SVD section, she writes that you can provide predictions by just getting a low-rank approximation of a ratings matrix?

Moreover, if you have:
$$A = U \times \Sigma \times V^T$$
We can simply make predictions by:
$$A \approx U_n \times \Sigma_n \times V_n^T$$
where $n$ is the $n$ lower-rank approximation.

Why do both of these methods work? What is the point of the first method if I can just use the second method to make predictions?

FURTHER EDIT: Why does the first method use the inverse of matrix $\Sigma$ whereas the second does not?


texts: Should the lambda for L1 norm regularizer inversely be proportional to the number of trainable weights?

texts:
Say I want to implement Conv2D in keras and for each Conv2D layer, if I apply 20 filters of [2,3] filter on an input with depth of 10, then there will be 20*(2*3*10+1) = 1220 trainable weights.

the value of L1 norm would proportionally increase the more trainable weights there are. Similarly for L2 norm.

So shouldn't the lambda, as in kernel_regularizer=l1(lambda), be inversely be proportional to the number of trainable weights?

intuitively for me, if lambda of 0.1 worked for 10,000 weights, then applying the same or bigger lambda for 1 million weights doesn't make sense to me.


texts: tree.DecisionTree.feature_importances_ Numbers correspond to how features?

texts:
clf = tree.DecisionTreeClassifier(random_state = 0)
clf = clf.fit(X_train, y_train)
importances = clf.feature_importances_

importances variable is an array consisting of numbers that represent the importance of the variables. I wonder what order is this? Is the order of variable importances is the same as X_train?

I am trying to make a plot from this. So order matters.


texts: Best practice for short sentences in a deep learning network

texts:
In a deep learning network (CNN or RNN), we might use word embeddings such as FastText, Glove, etc. to represent the input text. My question is:

If I'm working on a data from Twitter, and I have a variety of text lengths, as in the picture below (example):

sometime I have a few sentences of a length larger than 150 and the average length of the rest of the sentences is 48.

Here, what I noticed in some implementations online that they pad the short sentences with "PAD" word to increase their size to reach the length of the largest sentence (Red spaces in the picture), where this PAD word is filled with a random value. Is it a good practice to do so?

If my dataset contains 5000 tweets, and I have about 20 tweets with a length larger than 150 , the random value will affect the classification task with a high ratio since most of the sentences will be padded with a random value.


texts: Different hidden layer architectures deliver the same classification results, is that normal?

texts:
I have a data set with 600 data points with about 10 attributes (binary).
The dataset has been normalized:

Xnormalized = StandardScaler().fit_transform(X)

The outputs are binary (2 classes)
I tried different MLP architectures using 1 or 2 hidden layers with different number of nodes and performed a repeated stratified cross validation as shown below:

names=[]
classifiersmlp = []
for i in range(5, 26,5):
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd',
                                        random_state=1,
                                        activation='tanh',
                                        hidden_layer_sizes=[i]))
    for j in range(5,26,5):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd',
                                            random_state=1,
                                            activation='tanh',
                                            hidden_layer_sizes=[i,j]))

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=1)
scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1'}
mlpResults = []
for name, clf in zip(names, classifiersmlp):
    print(name)
    cvresultMLP = cross_validate(clf, Xnormalized, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score']))

The results of all architectures are very similar (only 1-2% difference in all 3 evaluation measures (recall (approx 78%) and precision (approx 74%)). Is it normal that the architectures are equally good or should there be greater differences? What does it mean that the results are so similar?

After a comment from @yohanesalfredo that the scaling is outside the cv, I updated the code:

names=[]
classifiersmlp = []
for i in [5,25,50]: #range(5, 5,3)
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd',
                                        learning_rate_init=0.01,
                                        random_state=1,
                                        activation='tanh',
                                        hidden_layer_sizes=(i)))
    for j in [5,20]: #range(1, 5,3):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd',
                                            random_state=1,
                                            learning_rate_init=0.01,
                                            activation='tanh',
                                            hidden_layer_sizes=(i,j)))

scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1', # according to docu only score for the 1 label
          'roc_auc':'roc_auc'}
mlpResults = []
rand_state=1
for name, clf_temp in zip(names, classifiersmlp):
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=rand_state)
    rand_state+=1
    classifier_pipeline = make_pipeline(preprocessing.StandardScaler(), clf_temp)
    cvresultMLP = cross_validate(classifier_pipeline, X, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score']))
    print(np.mean(cvresultMLP['test_roc_auc']))


texts: Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)

texts:
I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options:

Driving a ---- is not fun in London streets.

Apple
Car
Book
King

With the right model in place:

Question 1. What operation/function has to be used to find the best fitting choice? The similarity functions in the library are defined between one word to another word and not one word to a list of words (e.g. most_similar_to_given function). I don't find this primitive function anywhere while it is the main operation promised by CBOW (see below)! I see some suggestions here that are not intuitive! What am I missing here?

I decided to follow the head first approach and start with fastText which provides the library and pre-trained datasets but soon got stuck in the documentation:

  fastText provides two models for computing word representations: skipgram and cbow ('continuous-bag-of-words'). The skipgram model
  learns to predict a target word thanks to a nearby word. On the other
  hand, the cbow model predicts the target word according to its
  context. The context is represented as a bag of the words contained in
  a fixed size window around the target word.

The explanation is not clear for me since the "nearby word" has a similar meaning as "context". I googled a bit and ended up with this alternative definition:

  In the CBOW model, the distributed representations of context (or surrounding words) are combined to predict the word in the middle.
  While in the Skip-gram model, the distributed representation of the
  input word is used to predict the context.

With this definition, CBOW is the right model that I have to use. Now I have the following questions:

Question 2. Which model is used to train fastText pre-trained word vectors? CBOW or skipgram?

Question 3. Knowing that the right model that has to be used is CBOW, can I use the pre-trained vectors trained by skipgram model for my word prediction use case?


texts:
This follows a recent post where I got my answer concerning possible web-based techno for plotting big heatmaps ($600 \times 600$ dissimilarity matrix).

Now, that I have achieved that, I'm struggling with setting the color scale.
I would like some fine continuous color scale. For now I have plugged a discretization into 17 colors... not enough. With Python, pretty much is done with pyplot.pcolor.

How can I obtain such a 'continuous' color scale?

Here the difference between the two heatmaps parameterized by the same matrix:

Using matplotlib.pyplot in Python

Using D3.js and the code below

Here is the code:

<script src="http://d3js.org/d3.v3.js"></script>
<canvas id="heatmap"></canvas>
<script>
var tsvName = 'wasserstein_dists.tsv';
var canvas = document.getElementById('heatmap');
var ctx = canvas.getContext("2d");

var scale = 1;
var colors = ["#6363FF", "#6373FF", "#63A3FF", "#63E3FF", "#63FFFB", "#63FFCB",
           "#63FF9B", "#63FF6B", "#7BFF63", "#BBFF63", "#DBFF63", "#FBFF63",
           "#FFD363", "#FFB363", "#FF8363", "#FF7363", "#FF6364"];
var heatmapColor = d3.scale.linear()
    .domain(d3.range(0, 1, 1.0 / (colors.length - 1)))
    .range(colors);

// On charge le tsv :
d3.tsv(tsvName, function (d) {
  return {
    x: +d.x,
    y: +d.y,
    v: +d.v,
  };
}, function (err, data) {
  if (err != null) { return alert('No tsv file'); }

  var size = Math.sqrt(data.length);
  canvas.width = size * scale;
  canvas.height = size * scale;

  var c = d3.scale.linear()
    .domain([d3.min(data, function (d) { return d.v; }), d3.max(data, function (d) { return d.v; })])
    .range([0,1]);

  data.forEach(function (d) {
    ctx.fillStyle = heatmapColor(c(d.v));
    ctx.fillRect(d.x * scale, d.y * scale, (d.x + 1) * scale, (d.y + 1) * scale);
  });
});


texts: Update machine learning model while retaining the prediction results exact for old data

texts:
New to ML here. In our industry, we are looking for a type ML method/model that can be updated to accommodate new data points while keeping the prediction value of the historical data exactly the same. Of course a simple hash table would do, but are there other better alternatives.
First question is what is this type of method/model called in the ML world. Where could I find some good examples/explanations of such techniques?
Thanks


texts: How to make machine learning model that reports ambiguity of the input?

texts:
Suppose I want to build a neural network regression model that takes one input and return one output.

Here's the training data:

0.1 => 0.1
0.2 => 0.2
0.1 => -0.1

You will see that there are 2 inputs 0.1 that matches to different output values 0.1 and -0.1. So what will happen with most machine learning models is that they will predict the average when 0.1 is fed to the model. E.g. the output of 0.1 will be (0.1 + (-0.1))/2 = 0.

But this 0 as an average answer is an incorrect answer. I want the model to be telling me that the input is ambiguous/insufficient to infer the output. Ideally, the model would report it as a form of confidence.

How do I report predictability confidence from the input?

The application that I find very useful in many areas is that I could then later ask the model to show me inputs that are easy to predict and inputs that are ambiguous. This would make me able to collect the data that are making sense.

One way I know is to train the model then check the error on each training data, if it's high then it probably means that the input is ambiguous. But if you know any other papers or better techniques, I would be appreciated to know that!


texts: How to deal with version control of large amounts of (binary) data

texts:
I am a PhD student of Geophysics and work with large amounts of image data  (hundreds of GB, tens of thousands of files). I know svn and git fairly well and come to value a project history, combined with the ability to easily work together and have protection against disk corruption. I find git also extremely helpful for having consistent backups but I know that git cannot handle large amounts of binary data efficiently.

In my masters studies I worked on data sets of similar size (also images) and had a lot of problems keeping track of different version on different servers/devices. Diffing 100GB over the network really isn't fun, and cost me a lot of time and effort.

I know that others in science seem to have similar problems, yet I couldn't find a good solution.

I want to use the storage facilities of my institute, so I need something that can use a "dumb" server. I also would like to have an additional backup on a portable hard disk, because I would like to avoid transferring hundreds of GB over the network wherever possible. So, I need a tool that can handle more than one remote location.

Lastly, I really need something that other researcher can use, so it does not need to be super simple, but should be learnable in a few hours.

I have evaluated a lot of different solutions, but none seem to fit the bill:

svn is somewhat inefficient and needs a smart server
hg bigfile/largefile can only use one remote
git bigfile/media can also use only one remote, but is also not very efficient
attic doesn't seem to have a log, or diffing capabilities
bup looks really good, but needs a "smart" server to work

I've tried git-annex, which does everything I need it to do (and much more), but it is very difficult to use and not well documented. I've used it for several days and couldn't get my head around it, so I doubt any other coworker would be interested.

How do researchers deal with large datasets, and what are other research groups using?

To be clear, I am primarily interested in how other researchers deal with this situation, not just this specific dataset. It seems to me that almost everyone should have this problem, yet I don't know anyone who has solved it. Should I just keep a backup of the original data and forget all this version control stuff? Is that what everyone else is doing?


texts: Why is the optimal C chosen by GridSearchCV so small?

texts:
I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ridiculously small (~e-18) so that the margin is expanded to contain all samples. Even when I alter the samples so that they are easily separable, the optimal C is still on the scale of e-18. GridSearchCV selects a very small C however I try to alter the samples. Does anyone know why this is happening?
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets.samples_generator import make_blobs
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV

X, y = make_blobs(n_samples = 500, centers = 2, random_state = 6,
                  cluster_std = 1.2)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(X[:,0], X[:,1], c = y, cmap = 'rainbow', s = 30,
           edgecolors = 'white')
ax.set_xlabel(r'$x_1$', fontsize = 20)
ax.set_ylabel(r'$x_2$', fontsize = 20)

svc = SVC(kernel = 'linear')
c_space = np.logspace(-20, 1, 50)
param_grid = {'C': c_space}
svc_cv = GridSearchCV(svc, param_grid, cv = 5)
svc_cv.fit(X, y)
c = svc_cv.best_params_['C']
svc.C = c
svc.fit(X, y)

support_vecs = svc.support_vectors_

x1_min = min(X[:,0])
x1_max = max(X[:,0])
x2_min = min(X[:,1])
x2_max = max(X[:,1])
x1 = np.linspace(x1_min, x1_max, 100)
x2 = np.linspace(x2_min, x2_max, 100)
X1, X2 = np.meshgrid(x1, x2)
points = np.vstack([X1.ravel(), X2.ravel()]).T
boundary = svc.decision_function(points).reshape(X1.shape)
ax.contour(X1, X2, boundary, colors = 'k', levels = [-1, 0, 1],
           linestyles = ['--', '-', '--'])
ax.scatter(support_vecs[:,0], support_vecs[:,1], s = 250, linewidth = 1,
           facecolors = 'none', edgecolors = 'k')

```


texts: A new comer in this field - How to start?

texts:
To be honest I find entire DS field so vague that I can't find actual roadmap of learning. I am actually a programmer for 10+ years and pretty much interested to get into Data related field. My stats and math suck at this moment.

My question is, is it Ok that I use my programming skills and jump right onto Machine Learning/Analysis tools like WEKA, Python Scikit etc and use their algorithm for understanding and keep studying/learning theory in parallel. Right now in terms of Data, I am already involved in:

Data Processing(RDBMS)
Data Scrapping(Python)

What I am thinking the logical steps are:

Data Mining
Data Analysis (sentiment analysis etc)
Data Presentation/Visualization( D3 etc?)
Machine Learning Kit/Libraries

I have given myself a 6 month time to get into this field.

Thanks


texts: Problem with important feature having a lot of missing value

texts:
I am facing a dilemma with a project of mine. One of the variables (numerical) doesn't have enough data i,e almost 99% data are missing. However, upon talking to the domain experts, it appears that the particular variable is important to the problem we are trying to solve (model). Initially, I thought of converting it to a binary variable such that 1 will represent that the variable has a value at that position and 0 will represent the missing value. However, it seems that we are missing information by doing it.

Can anybody suggest any way go forward?

One thought came to me is to discretize the variables using quantiles, but then what to do with the missing values?

Another one is to include both the binary variable along with the original variable in the model with missing values replaced by some imputed values. But I cannot come to any logical reasoning as to why or why not this will work.

Any light on this matter would be greatly helpful (other than of course drop it altogether).

Thanks.


texts: Are pre-trained models vor CIFAR-10 / CIFAR-100 / SVHN available?

texts: I know that there are various pre-trained models available for ImageNet (e.g. VGG 16, Inception v3, Resnet 50, Xception). Is there something similar for the tiny datasets (CIFAR-10, CIFAR-100, SVHN)?


texts: How to use sklearn train_test_split to stratify data for multi-label classification?

texts:
I am attempting to mirror a machine learning program by Ahmed Besbes, but scaled up for multi-label classification. It seems that any attempt to stratify the data returns the following error: The least populated class in y has only 1 member, which is too few. The minimum number of labels for any class cannot be less than 2.

In my data set, I have 1 column which contains clean, tokenized text. The other 8 columns are for the classifications based on the content of that text. Just to note, column 1 - 4 have significantly more samples than 5 - 8 (more obscure classifications derived from the text).

Here is a generic sample from my code:

x = data['cleaned_text']
y = data[['car','truck','ford','chevy','black','white','parked', 'driving']]

x_train, x_test, y_train, y_test = train_test_split(x,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=42)

print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)

Output: (6293,) (700,) (6293, 8) (700, 8)

Adding stratify=y to train_test_split returns the error previously mentioned. Even when I limit y to just one column, I still get the error.

How can I stratify the data so that I give the program a fair look in the training set?


texts: Correlations, p-values and features selection

texts:
By using correlation matrix, I got some results:
Count_words          -0.098857
Count_numbers        -0.008305
Count_symbols        -0.025853
Count_question       -0.031649
Count_equal           0.224223
Count_characters      0.09

I used this line of code (in case you are familiar with Python):     df.drop("Target", axis=1).apply(lambda x: x.corr(df.Target))
If I understand correctly, the above results should suggest that there is not correlation between the variables considered.
Since I would like to add the above variables (or some of them) in a model already built with other features (textual), I would like to know if I can include all of them based on the fact that they are not correlated to each other and that the p-value is less than 0.05. My doubt is  if the above results do not make sense and do not suggest that these variables can be used in the model.
I hope you can give me some advices on that. Thanks


texts: Creating the optimal set of utterances to train a natural language processing engine

texts:
Context

I am using natural language processing engines such as IBM's Watson Conversation and Microsoft's LUIS, which take a sentence and classify its intent. For example, "i want to buy food" => "Buy_Food".

Problem

I am trying to come up with the best set of utterances to train those engines to understand natural language for a given application. My goal is to improve the accuracy of the NLP engine predicting the correct intent.

Suppose I have a few thousand utterances, each with a known intent. I need to come up with a set of just a few hundred best utterances that will be used to create a model in the NLP engine.

What I tried

At first I tried to take a few hundred sentences randomly, create an NLP engine model, and test its accuracy using a few hundred other sentences from my set. Then I would replace one sentence in the NLP engine model, and re-run the test again. At first, Microsoft's LUIS would guess the intent correctly around 90-95% of the time, and after a few dozen iterations the accuracy goes up to 91-96% - very slightly.

I am also thinking about using part-of-speech tagging to not just replace one sentence randomly, but to strive for syntactical diversity of those sentences.

Reason behind doing this

The reason why I am doing all of this is that it seems that dumping more utterance examples into the NLP engine doesn't seem to make it better; in fact, it seems to make it worse.

This is why I am trying to think of a way to create a limited set of utterances that will be used to train the NLP model and improve the rate of how often the engine guesses the correct intent.


texts: Pandas throwing "Error tokenizing data. C error" while loading data sets from URL

texts:
I am trying to work on the Titanic competition to get hands on experience with data science & machine learning.  I tried to load up the datasets from GitHub but pandas threw the following error:

ParserError: Error tokenizing data. C error: Expected 1 fields in line 32, saw 2

I tried to follow the advice of other SO users so I added skiprows=1 parameter in my pd.csv() call to skip the first row but it didn't work.

import pandas as pd

train_dataset = pd.read_csv("https://github.com/oo92/titanic-files/blob/master/train.csv", skiprows=1)
test_dataset = pd.read_csv("https://github.com/oo92/titanic-files/blob/master/test.csv", skiprows=1)
ground_truths = pd.read_csv("https://github.com/oo92/titanic-files/blob/master/gender_submission.csv", skiprows=1)

train_dataset.head()


texts: XGBoost Linear Regression output incorrect

texts:
I am a newbie to XGBoost so pardon my ignorance. Here is the python code :

import pandas as pd
import xgboost as xgb

df = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30]})
X_train = df.drop('y',axis=1)
Y_train = df['y']
T_train_xgb = xgb.DMatrix(X_train, Y_train)

params = {"objective": "reg:linear"}
gbm = xgb.train(dtrain=T_train_xgb,params=params)
Y_pred = gbm.predict(xgb.DMatrix(pd.DataFrame({'x':[4,5]})))
print Y_pred

Output is :

[ 24.126194  24.126194]

As you can see the input data is simply a straight line. So the output I expect is [40,50].  What am I doing wrong here?


texts:
First, apologies if this isn't the right forum for my question -- let me know if there's a better place.

I'm trying to implement Q learning on a two-dimensional grid world, where the agent has four actions available -- movement to the left, right, up, or down. A $4 \times 4$ transition matrix $T$ gives the probability the agent moves in the intended direction. So if $T$ is the identity matrix, and the agent intends to move left, it will move left $100 \%$ of the time; but $T$ can be modified such that the agent moves left $90 \%$ of the time, and up $10 \%$ of the time, etc.

My implementation does the following:

  initializes Q values to 0
  places agent in a randomized start location $s$
  agent chooses an action $a$ according to an epsilon-greedy strategy
  the transition matrix is used to determine which direction $i$ the agent actually ends up moving
  agent moves in direction $i$, ends up in state $s'$, collects some reward $r$, and the Q value $Q(s, a)$ is updated
  repeat steps 3, 4, 5 as necessary

When $T$ is the identity matrix, my implementation returns the correct answer. When I change $T$, my implementation still returns the answer as though $T$ is the identity.

I've spent a lot of time reviewing step 4 above and believe my implementation is correct. When I choose $T$ such that the agent moves in the intended direction $90 \%$ of the time, then $a$ and $i$ match $90 \%$ of the time, exactly as intended.

Is there anywhere else this issue could be coming from?

Thanks!


texts:
I went through basic concepts of ML algorithms but still had few queries on their advantages over each other.

When should we use linear regression vs Neural Networks for regression?
When should be use Logistic Regression vs SVM vs Neural Networks classification vs Decision Trees?


texts: KeyError: Selecting text from a dataframe based on values of another dataframe

texts:
I have the following two dataframes badges and comments. I have created a list of 'gold users' from badges dataframe whose Class=1.
Here Name means the 'Name of Badge' and Class means the level of Badge (1=Gold, 2=Silver, 3=Bronze).
I have already done the text preprocessing on comments['Text']and now want to find the count of top 10 words for gold users from comments['Text'].
I tried the given code but am getting error:
"KeyError: "None of [Index(['1532', '290', '1946', '1459', '6094', '766', '10446', '3106', '1',\n       '1587',\n       ...\n       '35760', '45979', '113061', '35306', '104330', '40739', '4181', '58888',\n       '2833', '58158'],\n      dtype='object', length=1708)] are in the [index]". Please provide me a way to fix this.

Dataframe 1 (badges)
   Id | UserId |  Name          |        Date              |Class | TagBased
   2  | 23     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   3  | 22     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   4  | 21     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   5  | 20     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   6  | 19     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False

Dataframe 2 (comments)
   Id|                    Text                             |    UserId
    6|  [2006, course, allen, knutsons, 2001, course, ...  |    3
    8|  [also, theo, johnsonfreyd, note, mark, haimans...  |    1

Code
for index,rows in comments.iterrows():
  gold_comments = rows[comments.Text.loc[gold_users]]
  Counter(gold_comments)


texts: How to make our own environments for reinforcement learning?

texts: How can we make our own environments for reinforcement learning? I have a heavy doubt that a game engine is used.. And if a game engine is used, which game engine is used that can be downloaded for free?


texts: Encoding entities with features of continuous values

texts:
Given a set of entities, I would like to predict the next in the sequence; for this purpose, I would like to use RNN. However, my first challenge is how to model the entities.
A possible input sequence can be:
EntityType_1 -> \
EntityType_2 -> \
EntityType_1 -> \
EntityType_3

Where each entity, given its type, has a unique set of continuous properties. For instance, the above sequence including features can be:
EntityType_1 (x=0.1, y=0.8) -> \
EntityType_2 (z=0.5) -> \
EntityType_1 (x=0.9, y=0.3) -> \
EntityType_3 (i= 0.2, j=0.7)

I have more than 10,000 entity types, where each has anywhere between 0 to 10s of features.
Any thoughts on related work on modeling/encoding such entities?


texts: What are ways to pick a single training sample to compute gradient in SGD?

texts: instead of computing the full gradient as in GD, SGD computes estimate of the gradient by either computing gradient of a mini-batch or computing gradient for a single training sample, picked randomly at each step of updating parameters. The question is besides picking training samples by a uniform distribution, what are other ways to pick them for SGD? And what are pros and cons of each way?


texts: order of features importance after make_column_transformer and pipeline

texts:
I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor.
The code looks something like this

 xgb = XGBRegressor()

 preprocessor = make_column_transformer(
        ( Fun1(),List1),
        ( Fun2(),List2),
            remainder='passthrough',
    )

    model_pipeline = Pipeline([
        ('preprocessing', preprocessor),
        ('classifier', xgb )
    ])

   model_pipeline.fit(X_trn, Y_trn)

Therefore, the training data which inputted into the XGBRegressor have no labels and resorted due to the make_column_transformer function. Given this, how do I extract the features importance using XGBRegressor.get_booster().get_score() method?

Currently, the output of get_score() is a dictinary that looks like this:
{‘f0’: 123 ,
‘f10’: 222,
‘f100’: 334,
‘f101’: 34,
…
‘f99’:12}

Can I assume that the order of the features provided by get_score() is identical to the order of features after make_column_transformer function (aka, I have to incorporate the feature sorting) such that 'f0' == 1st feature after make_column_transformer, 'f1' ==2nd  feature after make_column_transformer, etc.?


texts: How can I deal with data that is on the format "Image + single number"?

texts: Let say I have a data set where every sample is an image of a landscape and a temperature associated with the landscape. How do I incorporate the temperature into my convolutional neural network for classifying if the data is e.g a winter or summer landscape? Can I simply add the temperature as a feature after the feature learning part of the network? I can not find similar questions but maybe this has a name that I am not aware of?


texts:
The internet is full of pictures like this:

But how are the second/third/etc CNN layers able to extract features when the features are already extracted by the previous layers?
For example, the mid-level feature in the picture has a nose. When we apply this "nose" filter, the output feature map will be an image without the nose, right?
Then, this feature map is passed to the next CNN layer, but how is it able to extract the "high-level feature" if the feature map given to it doesn't contain the nose? And the more layers we stack in CNN, the less meaningful data will be extracted in the latter layers.


texts:
I have been working through Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis where the authors make the following claim (subjection 2.2.1):

...the Laplacian space is more robust to illumination changes and more indicative for face structure.

what is Laplacian space in this context? How is it robust to illumination changes?


texts: How to generate synthetic data from training set for regression?

texts:
I have 41 training samples with 12 features and I am thinking about generating synthetic data that follows the distribution of my training set. All features are continuous variables. I found something called pysynth  link. I don't know what's happening but it gives this error "Unable to allocate 7.28 TiB for an array" .For context 1 TiB is 1100 GB.
Is there any way I could generate synthetic data to add to my training set?


texts: How to Implement a Custom Loss Function with Keras for a Sparse Dataset

texts:
My dataset is composed of an idle system that, at some time instants, receives requests. I'm trying to predict these instants through a clock. Since the requests are sparsely distributed (I've forced them to last for a while so they don't get too sparse), I wanted to create a new loss function that would penalize the model if it only gives out a zero prediction for everything. My implementation attempt is just a penalty for the standard logits:

def sparse_penalty_logits(y_true, y_pred):
    penalty = 10
    if y_true != 0:
        loss = -penalty*K.sum((y_true*K.log(y_pred) + (1 - y_true)*K.log(1 - y_pred)))
    else:
        loss = -K.sum((y_true*K.log(y_pred) + (1 - y_true)*K.log(1 - y_pred)))

    return loss

Is it correct? (I have also tried it with tensorflow). Every time I run it I either get a lot of NaN's as the loss or predictions that are not binary at all. I wonder if I'm doing something wrong at setting up the model also because binary_crossentropy is not working properly either. My model is something like this (the targets are represented by a column with either 0's or 1's):

model = Sequential()
model.add(Dense(100, activation = 'relu', input_shape = (train.shape[1],)))
model.add(Dense(100, activation = 'relu'))
model.add(Dense(100, activation = 'relu'))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(optimizer = 'adam', loss = sparse_penalty_logits)

If I run it, as I said, I get very strange results (boy, do I feel like I've messed up real bad...):


texts: How to implement sequence to sequence models?

texts:
I have a dataset with patient demographics, diagnosis history, hospital visit dates, drugs consumed etc.
All these events have time stamp information (except static info like demographics such gender, age, education etc).
My objective is to predict the next 3 visits to hospitals and risk factors for his next 3 visits based on his previous hospital records such as visit information, drug information, demographics, diagnosis information etc.
Can experts here let me know the list of steps that I need to follow so I can learn and try to achieve my project objective?


texts: How to get common values between two multi sets?

texts:
I am working in R. I have two vectors; A and B of lengths 5913 and 3733 respectively (with entries are repeated). I want to extract those values (with repetitions) that are present in both A and B.
I have done (A %in% B) (lets call it C) and (B %in% A) (lets call it D).
The length of C is 3906 and that of D is 3607 (so 2007 elements in A are not in B and 126 elements in B are not in A).

But How do I find the common values ? I don't think I can use intersect() method as it expects vectors with no duplicate values. I have many many repetitions.

Note: Due to the length of these vectors, I couldn't mention them here.

Any help would be greatly appreciated.

Thanks


texts:
I am working on an automated insights generation use case where I want to generate meaningful sentences from given aggregated data.

For example,
Data:

Student = John
Total_Marks = 96
Class_Average = 85

NLG model-generated insights:

1. You did an excellent job, John! Your score is 96!
2. You have scored 11 marks above the class average.

When I look at classic NLG, they are sentence generation approaches given a starting letter or word.
This might be more of a Neural Machine Transition use case.

What do you think my approach should be?


texts: Looking for more recent dataset for document classfication

texts:
I am trying to develop an NLP - CNN algorithm to detect documents with sensitive information such as passport, license and distinguish them from other documents like resume, email, form or advertisements.
I personally consider this as a document classification problem and looked for open source datasets which had documents from different category/classes.
I found the RVL-CDIP Dataset and tobacco3482 dataset with classes such as Email, form, letter, news, resume, scientific. However, the dataset collection looks from an old collection (ie samples in these datasets represent how resumes were written in 20000 and invoices collected in early 2000 )These datasets doesn't quite represent recent sample set of documents that we might come across on a daily basis.
Is there a more recent document classification dataset available which would be more relevant to the current format of these documents?
EDIT : Looking for a similar dataset to RVL-CDIP Dataset and tobacco3482 dataset but which is more recent created. Use case is to still distinguish between resumes, advertisements, passports, and emails.
I am not looking at detecting sensitive information within emails so my usecas is similar to RVL-CDIP Dataset  use-case


texts: Multiclass classification on imbalanced dataset : Accuracy or micro F1 or macro F1

texts:
I have a multiclass classification problem. Further, an instance can be assigned to exactly one class. My dataset is highly imbalanced. I know that accuracy is not a good metric to use in this case because one can simply predict the high frequent class and get a good score. I understand that micro F1 is better than macro F1 for multiclass classification problem but it turns out that the micro F1 score is the same as accuracy score. So, the whole idea of looking into alternative metric (i.e. micro F1) instead of accuracy has gone full circle.

Should I be using macro F1 instead?


texts: What is the best way to compare these small distributions?

texts:
I have one distribution of size 30.
This are results (ROC-AUC for example) from training a neural network for 30 times in a row with the same hyperparameters but since they are randomly initialized the result is always a little bit different.
Then I train the same network with other hyperparameters and only want to do that for fewer runs. Lets say for 5 runs.
My null hypothesis is that the smaller runs distribution is not smaller than the distribution with 30 runs (one side test).
What kind of statistical significance test would be the best to compare these small distributions?
PS: At the moment I am using Mann Whitney U Test. Is there anything better?


texts: Relation between convolution in math and CNN

texts: I've read explanation of convolution and understand it to some extent. Can somebody help me understand how this operation relates to convolution in Convolutional Neural Nets? Is filter like function g which applies weight?


texts: DCGAN: how do we construct the proposed CNN of the original DCGAN paper?

texts:
I am reading the original paper on the Deep Convolutional GAN (link: DCGAN paper) and on the fourth page the authors make a proposition on how to model the generator and discriminator as CNNs. However I have failed to find source code to their implementations of this construction. Their description of the layers of the CNNs are a little vague to me and not a definite recipe that I can follow.
In my research I am interested in finding an optimal construction for the DCGAN when we have images as data (maybe this goes without saying). Perhaps there are different approaches for different data sets? Where can I find a decent, or best, DCGAN available to us at this point that is already implemented (I am specifically looking for some Github repository)?
As reference, what I am looking at right now is this code: DCGAN-code.
EDIT
I found the code short while after I posted the question (typically). Here it is for anyone interested: DCGAN-sourcecode.


texts: Tensorboard reported accuracy does not match my own calculations

texts:
I'm using Google's AI Platform to train an image classifier with Tensorflow (resnet-50 model). I have only two classes. Training set is something like 4K images in size and validation set 1K images. After 20000 steps training is finished and Tensorboard reports top-1 accuracy above 80% (by the way I don't know how it's possible to have smoothed curve consistently below the non smoothed one):

When I try to do "manually" my own calculation (on the same validation set), I get much lower figures though (something like 60-65%). I did my own calculations after having saved the model locally. I've validated my method by deploying my model on AI Platform too (prediction results are stricly the same that those that I get locally).
Where is my error? Could it be that I have to preprocess my images before sending them to predict API? (something that would be done without my knowledge in the evaluation part while training).
EDIT: the training job that I submit to AI Platform includes the num_eval_images parameter. It is set to 0 since "If it is 0, all the images in validation_data_path will be used for evaluation" (see https://cloud.google.com/ai-platform/training/docs/algorithms/image-classification-start#gcloud)


texts: Using a decision tree with 3 dimensional input points

texts:
I implemented a normal classification tree (that uses the Gini index to look for a split). I am using it to predict the age of people. My input data was a series of points on 1 axis (Only X coordinate), and output was an age (9 years old, 10 years old....).

Example:

p1 = 1.3, p2 = 3.4, p3 = 2.1 ........ => Output = 9 years old

p1 = 1.4, p2 = 2.4, p3 = 2.6 ........ => Output = 10 years old

My tree looks something like (just an example):

if p1 > 1
    if p2 < 3
        output = 10 years old
    else
        output = 9 years old
else
    ...

This is working fine.

Now I want to modify it, to work with 3D points. I mean points that have X, Y, Z and not just X.

So my input data would be like:

p1 = [1.2, 1.5, 4.3], p2 = [4.2, 1.3, 5.2] .... => output = 9 years old

How do I approach this? should my tree use each coordinate as a seperate input (p1x, p1y, p1z, p2x, p2y....) or is there an approach that can be taken for my case?

Note: I am using classification trees but if you think there's another way of predicting in my case, I would love to know about it.

Also I'm using python code, but I'm looking for the logic not the programming part.

Thanks


texts:
I was trying Random Forest Algorithm on Boston dataset to predict the house prices medv with the help of sklearn's RandomForestRegressor.

Just to evaluate how good is the model performing I tried sklearn's learning curve with below code

 train_sizes = [1, 25, 50, 100, 200, 390] # 390 is 80% of shape(X)

    from sklearn.model_selection import learning_curve
    def learning_curves(estimator, X, y, train_sizes, cv):
        train_sizes, train_scores, validation_scores = learning_curve(
                                                     estimator, X, y, train_sizes = train_sizes,
                                                     cv = cv, scoring = 'neg_mean_squared_error')
        #print('Training scores:\n\n', train_scores)
        #print('\n', '-' * 70) # separator to make the output easy to read
        #print('\nValidation scores:\n\n', validation_scores)
        train_scores_mean = -train_scores.mean(axis = 1)
        print(train_scores_mean)
        validation_scores_mean = -validation_scores.mean(axis = 1)
        print(validation_scores_mean)

        plt.plot(train_sizes, train_scores_mean, label = 'Training error')
        plt.plot(train_sizes, validation_scores_mean, label = 'Validation error')

        plt.ylabel('MSE', fontsize = 14)
        plt.xlabel('Training set size', fontsize = 14)
        title = 'Learning curves for a ' + str(estimator).split('(')[0] + ' model'
        plt.title(title, fontsize = 18, y = 1.03)
        plt.legend()
        plt.ylim(0,40)

If you notice I have passed X, y and not X_train, y_train to learning_curve.

I am not understanding should I pass X, y or only the training subset X_train, y_train to learning_curve.

Update 1

Dimensions of my Train/Test split (75%:Train and 25%:Test)
        X.shape: (489, 11)
        X_train.shape: (366, 11)
        X_test.shape: (123, 11)
I had few additional questions regarding the working of learning_curve

Does the size of test data set varies according to the size of train dataset as mentioned in list train_sizes or it is always fixed (which would be 25% in my case according to train/test split which is 123 samples) for example

When train dataset size = 1 the will the test data size be 488 or will it be 123(the size of X_test)
When train dataset size = 25 the will the test data size be 464 or will it be 123(the size of X_test)
When train dataset size = 50 the will the test data size be 439 or will it be 123(the size of X_test)

Update 2

In the blog the dataset has 9568 observations and the blogger passes entire dataset X to learning_curve.

train_sizes = [1, 100, 500, 2000, 5000, 7654]

In first iteration when train_size is 1 then the test_size should be 9567 but why he say that

  But when tested on the validation set (which has 1914 instances), the MSE rockets up to roughly 423.4.

Shouldnt the test_size be 9567 instead of 1914 for first iteration

In second iteration when the train_size is 100 then the test_size should be 9468

What I meant to say is the test_size will be variable according to train_size correct me if I am wrong


texts: How to interpret the original time series approximated using principal component analysis?

texts:
I've read some posts about PCA applied on time series, but still a bit confused and I have the following questions(Suppose I am working with a time series of the return of 50 industries and I want to use a clustering algorithm to divide them into several group):

Say I have calculated the eigenvalue and eigenvector from the correlation matrix, and found that the first twenty eigenvalue account for 85% of the total, and I then use these twenty eigenvalue to approximated the original time series. I know if I choose all eigenvalues then I can get the identical original time series, but what information did I lose if I choose twenty of them specifically? What is the purpose to do so?

I found some post said we can always drop the first principle component(means we don't use it), why can we do that?

Can I interpret each eigenvalue as a trend in the market, for example the first principle component, can I derive whether the corresponding industry is in the same direction or different from the market trend based on the sign of its corresponding eigenvector, and if so, can I apply k-means to all industries by using the eigenvectors of the first few principle component to group them, is this make sense?

Welcome for any kind of hint or ideas, thanks.


texts:
I am not sure if the title accurately reflects my problem but I essentially would like to aggregate a set of metrics of similar nature that comes from different data sources into a single metric. Say we are measuring MetricA from SourceA, MetricB from SourceB,...and MetricN from SourceN sampled over the same general population. There is no guarantee that the union of samples will cover all in the population at least once. With varying sampling rates and frequencies across sources, I would like to come to some level of normalization to sum it up at the top and track that over time.

I am not sure what the best way is to come up with weights though. Has any of you come across with a similar problem or perhaps some papers that found reasonable approaches to avoid/minimize bias?

Many thanks for your suggestions!


texts: Negative examples and false positives in object detection using SSD model

texts:
I am training a model to detect some objects. I have 8 classes and there are other similar objects that I don't want the model to detect. I have tried to make an unknown class for them but they are so many and there are no pattern for the model to recognize.
How can I provide a negative examples for the model? Since that tfrecord can't accept images without annotation and making a new class doesn't work very well.


texts: Should I put time on my Vanilla ANN for classifying MNIST Dataset

texts:
I am building a Vanilla Neural Network in Python for my Final Year project, just using Numpy and Matplotlib, to classify the MNIST dataset. Here's the specifications of the model:

One Input Layer + One Hidden Layers + One Softmax Layer
Number Nodes in each layer :[784, 800, 10]
Activation function used: ReLU and Softmax.
Also Normalized the Train and Test set, by dividing it by 255
Have used Mini Batch Gradient Descent(Mini Batch Size=4096).
The Model shows very low accuracy on the Test set. Around 8% -11% accuracy.
And it shows an accuracy of around 66% - 72% for the Train set.

I haven't used Regularization until now. And I don't know if that will help.

Now fed up with this, I am thinking of just implementing the model in Tensorflow and see if that works. Or is it necessary to implement it from scratch? Because I think I know how every concept works(although not being able to find the loophole in my model). What do you have to say about it?

If you want to have a look at my code here's the link. It would be great if I can get any suggestions.

P.S.: Does it create any problem if a DL model is not written using OOP?


texts: Can I create a good Speech Recognition Engine while having millions of recorded conversations?

texts:
I have at my disposal millions of wav files containing recorded conversations between employees and clients, i'm doing some research on the possibility of creating a good speech recognition engine. I've tested Google's Speech-To-Text and it's great. Is it possible to create something similar? ( Of course, no one can beat the quality and quantity of the data Google has but how close one can get?). And of course, what are the technical limits ( like hardware needed for this kind of learning ) and how much time should it take to achieve it?

Note : i'm a beginner in ML, so far , i've done some binary and multiclass classification,I have an idea on Neural Networks but no work done on that. The simpler the answer the easier for me to understand , Thanks!


texts: Passing Dependency/Constituency trees to a Neural Machine Translator

texts:
I am working on a project on Neural Machine Translation in the English-Irish domain. I am not an  expert and have researched entirely on my own for a technology exhibition so apologies if my question is simple.
I am trying to parse all of my English corpus to constituency trees. Of course, the format of a sentence when using the Stanford Parser is something like:

(ROOT (S (NP (VBG cohabiting) (NNS partners)) (VP (MD can) (VP (VB make) (NP (NP (NNS wills)) (SBAR (WHNP (WDT that)) (S (VP (VBP favour) (NP (DT each) (JJ other)))))))) (. .)))

Of course, when dealing with simple sequences, each word is used, it's not symbolism like NP or NNS in constituency trees.
Right now, I'm working with PyTorch and Fairseq to produce all my models and have gotten a working seq2seq model. But, can I simply just pass my English input like shown above to a model and expect it to train? Do I need to make a new model from scratch that can deal with tree structures? I've tried very hard to research this, reading papers, books and playing around with tools but since I'm not in a class for this and since it's not really documented, I'm finding it hard to find this on my own.
Any help would be greatly appreciated


texts: How can l get 50 % examples in training set and 50% in test set for each class when splitting data?

texts:
l have a dataset of 200 examples with 10 classes. l would like to split the dataset into training set 50% and test set 50%.

for each class, l have 20 examples. Hence, l would like to get for each class : 10 training examples and 10 test examples.

Here are my classes :

classes=['BenchPress', 'ApplyLipstick', 'BabyCrawling', 'BandMarching', 'Archery', 'Basketball', 'ApplyEyeMakeup', 'BalanceBeam', 'BaseballPitch', 'BasketballDunk']

l tried the following :

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(final_data, true_label, test_size=0.50, random_state=42)

However it returns a 50% training set and 50 % test set, without respecting the proportion for each class (l would like to get 10 examples in test set and 10 examples in training set for each class).
Here is the resulted splitting :


texts: Checking if english sentences have impact catpured in them using NLP

texts:
I am looking for high level approach to identify if a given sentence has impact captured in it. For example in below two examples, sentence 2 has impact captured.

Example 1

Sentence 1: I learnt speaking spanish this year.

Sentence 2: I learnt speaking spanish this year which helped expand my business to Mexico.

Example 2

Sentence 1 : I taught machine learning to my students this year.

Sentence 2 : I taught machine learning to my students earlier this year which resulted in increasing base salary of students by 20,000 Dollars.

My initial thought is below approach.

think of all the english words like helped, resulted etc.,
Apply these words as filters on a public domain data set to get sentences
Curate the sentences to see if they fall in the "Sentence 2" category.
Apply classification/modelling to categorize into sentences with impact and without impact

Question:

Is the above approach good or there are better approaches?
What is the best publicly available dateset to tackle this problem.

Any inputs are greatly appreciated. Thanks alot in advance!


texts: Estimating minPts in DBSCAN for document layout clustering

texts:
I am trying to choose parameters for DBSCAN clustering algorithm, in particular minPts.

The Wikipedia article suggests a rule of thumb to derive minPts from the number of dimensions D in the data set. minPts >= D + 1. For larger datasets, with much noise, it suggests minPts = 2 * dim. I guess D and dim stands for the same, do they?

I wonder what is the value of D in my case?

I use DBSCAN for clustering 2-dimensional black-and-white scans of business documents based on their layout. First, each scan gets segmented into black-and-white boxes and then turned into 0-1 1-dimensional arrays (strings). Here's an example of a segmented document scan.

I use Levenstein distance in clustering to measure similarity between scans.

So I guess my D = 1 and I should start with minPts = 2?


texts: How to calculate mean and standard deviation of all features in a class identified by k-nearest neighbors?

texts:
I have classified my data into several neighborhoods using k nearest neighbors. I need to efficiently calculate the mean and standard deviation for all features of data points belonging to a particular neighborhood.
I am using sklearn.kneighbors.


texts: I build a classification model on keras (Vanilla MLP),It went quite well by using a tutorial.But what are the default weights of layers because I didn't mention any in the code.It has a feature of intializing weights, but what if we don't initialize.


texts: Text Similarities: which nlp methods to use?

texts:
I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text.

Question 1: Which NLP method should I start with?

I have tried cosine similarity, I know its a great method but it's very slow in computing similarities.

I have thought about extracting important information from text (for example: Term Extraction, I don't want to use LDA) and computing similarities between that information.

Question 2: Any suggestions?


texts: Resources on Data Science for Football / Soccer?

texts: I am searching for resources on data science projects which used football/soccer data.


texts:
What does "full connection table" mean in Yan LeCuns comment on 1x1 convolutions?

  In Convolutional Nets, there is no such thing as "fully-connected layers". There are only convolution layers with 1x1 convolution
  kernels and a full connection table.

  It's a too-rarely-understood fact that ConvNets don't need to have a fixed-size input. You can train them on inputs that happen to
  produce a single output vector (with no spatial extent), and then
  apply them to larger images. Instead of a single output vector, you
  then get a spatial map of output vectors. Each vector sees input
  windows at different locations on the input. In that scenario, the
  "fully connected layers" really act as 1x1 convolutions.


texts: How to choose most appropriate Machine Learning model?

texts:
I need some help from anyone who are familiar with machine learning area.

ID        | Mach_1  | Mach_2 | Mach_3  | Mach_4 | Mach_5 | Rejected Unit (%)
127189.11     1         0        1         1        1           0.23
178390.11     0         0        0         1        0           0.10
902817.11     1         0        1         0        1           0.60

Above is the example of my data, for each ID there are several mach that are available and if that particular mach is used for that ID , the value will be 1 and if the mach is not used it is 0. And the rejected unit is the percentage value of rejected unit for that ID.

What I want to know is which mach is the most affected to the rejected unit. And what is the percentage for each mach that is affected to rejected unit.

Can anyone help me to advise on what machine learning algorithm/model that I can use to analyse this case study ?

EDIT:

I have done the linear regression and below is my code, however the output has show two warnings as shown in the screenshot below.

import statsmodels.api as sm
#create model
mod = sm.OLS(y_train,X_train)
res = mod.fit()
print(res.summary())


texts: Where can I find a dataset that contains criminal case sentencing data?

texts:
I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as:

Type of crime
Defendant Age/Sex/Race
Plea
Verdict
Sentence

Is there a dataset like this?


texts: Why TensorFlow can't fit simple linear model if I am minimizing absolute mean error instead of the mean squared error?

texts:
In Introduction I have just changed

loss = tf.reduce_mean(tf.square(y - y_data))

to

loss = tf.reduce_mean(tf.abs(y - y_data))

and model is unable to learn the loss just became bigger with time. Why?


texts: Which insights a data scientist could derive from text-analysis?

texts:
I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging.
Since it is my first time in text mining and manipulation, I would like to know which kind of insights I could get from this information and what the best approach to present this analysis would be.

For example: if I had many texts like the following:

Hemingway =["When spring came, even the false spring, there were no problems except where to be happiest. The only thing that could spoil a day was people and if you could keep from making engagements, each day had no limits. People were always the limiters of happiness except for the very few that were as good as spring itself.","Most people were heartless about turtles because a turtle’s heart will beat for hours after it has been cut up and butchered. But the old man thought, I have such a heart too.","Perhaps as you went along you did learn something. I did not care what it was all about. All I wanted to know was how to live in it. Maybe if you found out how to live in it you learned from that what it was all about.","The people that I liked and had not met went to the big cafes because they were lost in them and no one noticed them and they could be alone in them and be together."]

Shakespeare=["These violent delights have violent ends And in their triump die, like fire and powder Which, as they kiss, consume","Let me not to the marriage of true minds Admit impediments. Love is not love Which alters when it alteration finds, Or bends with the remover to remove. O no, it is an ever-fixed mark That looks on tempests and is never shaken; It is the star to every wand'ring barque, Whose worth's unknown, although his height be taken. Love's not Time's fool, though rosy lips and cheeks Within his bending sickle's compass come; Love alters not with his brief hours and weeks, But bears it out even to the edge of doom. If this be error and upon me proved, I never writ, nor no man ever loved.","O serpent heart hid with a flowering face!Did ever a dragon keep so fair a cave? Beautiful tyrant, feind angelical, dove feather raven, wolvish-ravening lamb! Despised substance of devinest show, just opposite to what thou justly seemest - A dammed saint,honourable villain!","Lord Polonius: What do you read, my lord? Hamlet: Words, words, words. Lord Polonius: What is the matter, my lord? Hamlet: Between who?  Lord Polonius: I mean, the matter that you read, my lord."]

Would it be useful a word frequency distribution, POS tags distribution? I have never presented any results, so I would like to know more on how a data scientist/analyst may 'tell' and look at this data.


texts:
After many research, I still can't find a neat answer about this question:

When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other output nodes we didn't took an action from with loss = 0. is this right?
I'm trying to understand which activation I should use in my hidden layers. I'm inputing numbers between 0-10 and outputting a raw value where no activation function should be applied (this is because we calculate the loss with raw values).

Thnx for helping out!


texts: Cross-Validation for Unsupervised Anomaly Detection with Isolation Forest

texts:
I am wondering whether I can perform any kind of Cross-Validation or GridSearchCV for unsupervised learning. The thing is that I have the ground truth labels (but since it is unsupervised I just drop them for training and then reuse them for measuring accuracy, auc, aucpr, f1-score over the test set).
Is there any way to do this?


texts: Self-attention mechanism did not improve the LSTM classification model

texts:
I am doing an 8-class classification using time series data.
It appears that the implementation of the self-attention mechanism has no effect on the model so I think my implementations have some problem. However, I don't know how to use the keras_self_attention module and how the parameters should be set.
The question is how to utilize keras_self_attention module for such a classifier.
The first confusion matrix is 2 layers of LSTM.

    lstm_unit = 256

    model = tf.keras.models.Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(keras.layers.Flatten())
    model.add(Dense(num_classes, activation='softmax'))

The second confusion matrix is 2 lSTM + 2 self-attention.
    lstm_unit = 256

    model = tf.keras.models.Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(SeqSelfAttention(attention_type=SeqSelfAttention.ATTENTION_TYPE_MUL,
attention_activation='sigmoid'))

    model.add(keras.layers.Flatten())
    model.add(Dense(num_classes, activation='softmax'))

I have further tried different functions from the module, such as
1.MultiHead
    model.add(MultiHead(Bidirectional(LSTM(units=32)), layer_num=10, name='Multi-LSTMs'))

Residual connection

 inputs = Input(shape=(X_train.shape[1],X_train.shape[2]))
    x = Masking(mask_value=0.0)(inputs)

    x2 = SeqSelfAttention(attention_type=SeqSelfAttention.ATTENTION_TYPE_MUL,
    attention_activation='sigmoid')(x)

    x = x + x2

    x = Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True))(x)
    x = Flatten()(x)

    output = Dense(num_classes, activation='softmax')(x)

    model = Model(inputs=inputs, outputs=output)

But they are more or less the same, no much effect on the MAR MAP and ACC.
I have 2 Titan Xp so computation power is less problem for me, is there a way to make the model more accurate?


texts: Stateful LSTM : Using different training window

texts:
Would it make sense for stateful LSTM (or LSTM in general) if in one epoch I feed [0-9],[10-19],[20-29],[30-39]...[990-999] (with corresponding labels/Y data) from my dataset. When I've presented all of the data for that epoch I then call model.reset_states()

After that epoch, could I then move the window forward by some arbitrary amount e.g. 2, so I then feed in [2-11],[12-21],[22-31],[32-41]...[982-991]

I would do that for 99 batches (the final sequence in the batch is now length 8 so I can't make a complete sequence).

Would doing that make sense?  That way the network learns sequences from different starting/end points, and has differing output/Y values to train against.


texts:
How should I go about solving the following problem?

If you randomly type a 6-digit number on a note, what is the
probability that you can see the same number if you flip your note
upside down? How would you explain your answer to a 6-year-old?

The only piece I have been able to calculate is that there are 9x10x10x10x10x10 numbers.
Is this correct? and/or what is the correct way to get the answer to this problem?
Thanks a lot for your help!


texts: Change one node type in Tensorflow graph

texts:
So I have a tensorflow graph saved in a .pb file, it works well on my machine, but when I run it on tensorflow v 1.3 it loads fine though it crashes with the following error

InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'Reshape' with these attrs.  Registered devices: [CPU], Registered kernels:
  device='CPU'; Tshape in [DT_INT32]
  device='GPU'; T in [DT_COMPLEX128]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_COMPLEX64]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_INT8]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_UINT8]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_INT16]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_UINT16]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_INT64]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_DOUBLE]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_FLOAT]; Tshape in [DT_INT32]
  device='GPU'; T in [DT_HALF]; Tshape in [DT_INT32]

           [[Node: Reshape_165 = Reshape[T=DT_FLOAT, Tshape=DT_INT64](transpose_145, add_31)]]

After investigating I found out that the Reshape_165 operation is of shape DT_INT64 and v1.3 doesn't supportDT_INT64 for Reshape as the message reads, so I was wondering if there's a way to change the Reshape_165 operation dtype to DT_INT32 mind you the whole graph gets executed but for this operation which is the one just before the outputs concat_52 and concat_53

add_31
Reshape_165
concat_52/axis
concat_52
concat_53/axis
concat_53


texts: How do I use the supervised learning classification in my project?

texts:
Just to give an idea of what I'm doing:

  I'm doing a project with financial data (tick data) and I'm trying to
  create a way make a model learn what happens before a breakout.

  Usually there is a disproportionate ratio between buyers and sellers
  and the interval between the orders usually is shorter (people are
  buying or selling faster so they don't miss out). This happens a
  couple of times every day in the asset that I trade in Brazil.

So my question is: How do I train "mini datasets" and then put all of them together so the model learns what a good breakout looks like. I want work initially with these two variables:

time interval between orders and
order volume

Any tips on which algorithms or subjects that I can study to do this?

Here is the dataset that I have. It also include the broker which the order came from. I have 2 years of data like that and I know when it breaks out. Any tips?


texts: Tensorflow version conflict between Spyder and Anaconda-Prompt

texts:
I downloaded the TensorFlow version 1.14.0 from Anaconda Prompt.
pip install tensorflow==1.14.0

When I check the version of Tensorflow from Spyder it shows like this
   import tensorflow as tf
   print("Tensorflow Version : ",tf.__version__)

Why Anaconda prompt and Spyder show the different versions of TensorFlow?


texts: Comparing TFIDF vectors of different shapes

texts:
I'm working on a project using TF-IDF vectors and agglomerative clustering -- the idea is that the corpus of documents increases over time, and when a new document is added, the mean cosine similarity with each cluster will be calculated to find the best match.
However, I'm under the impression that it is costly and inefficient to re-calculate the TF-IDF vectors of every document each time a new doc is added to the corpus.
I'm trying to discern if it's possible to generate the TF-IDF vectors on an individual basis for each document (see below), and then calculate the cosine similarity by temporarily reshaping/modifying the two vectors.
doc_1 = ['some', 'text', 'here']
doc_2 = ['other', 'text', 'here']

vectorizer = TfidfVectorizer()

vector_1 = vectorizer.fit_transform(doc_1)
vector_2 = vectorizer.fit_transform(doc_2)

# somehow add the features from vector_1 to vector_2 that do not exist there, and vice versa...

# this way vectors can be calculated on the fly without having to regenerate vectors with an ever-expanding corpus

It's my understanding that running the sklearn TF-IDF vectoriser returns not only the vectors but a mapping between the unique words in the corpus and the vectors they correspond to, so I think this approach would be possible, but I'm not sure?
Finally, as I'm new to data science, am I approaching this issue horribly? Is there a much cleaner or efficient way of solving the problem of non-parametrical clustering (i.e. no n_clusters) with an ever-expanding corpus.


texts: Network structure: k-cliques vs. p-cliques

texts:
In network structure, what is the difference between k-cliques and p-cliques, can anyone give a brief explaination with examples? Thanks in advanced!

============================
EDIT:
I found an online ppt while I am googling, please take a look on p.37 and p.39, can you comment on them?


texts: How does the "skip" method work for upsampling? (fully convolutional NN)

texts:
I'm studying fully convolutional neural networks for image segmentation, so far i've study and kind of understood the deconv network.
Following this tutorial (Upsampling) i can't really understand how the "skip" method works.

I understood how to apply max unpooling and the transpose convolution for upsampling,but how does the "skip" method "relate" to those 2 methods?

Why do we use this "skip" method? Max unpooling and transpose convolution are not enough to upsample and give back a better resolution map?


texts: Predicting object by features probabilities

texts:
I have the definition of an object provided as features probability.
Each object has it's own feature importance and probabilities.
For example for object "X", I have "color" feature (with the weight of 0.8) - the object can be blue in 80% of cases and black in 20% of cases. And "shape" feature (with the weight of 20%) - square in 30% and round in 70%.

I'm trying to create a "predictor", so if I'm observing something blue and round - (0.8 x 0.8) x (0.2 x 0.7) - probability for object X.

Does it make any sense mathematically?
If this method sounds reasonable enough, how should I handle really small numbers (I can have a really long vector of features, the final number will be really small)?


texts: Which machine learning methods can be used to address MonteCarlo sampling problems?

texts:
I would like to pick the brains of machine learning experts here to point me to a branch of machine learning technique that is best suited for my context. I would like to estimate the average value $\bar{f}$ of a function $f$ specified generally as

$$\bar{f} = \sum_{v\in\mathbb{Z}_2^n}p(v) f(v)$$

where $p(.)$ is a probability distribution over $n-$bit sequences,

$$p:\mathbb{Z}_2^n \rightarrow [0,1],$$

$$\sum_{v\in\mathbb{Z}_2^n}p(v) = 1.$$

The function $f$ maps sequences to real numbers,

$$f:\mathbb{Z}_2^n \rightarrow [0,1].$$

Furthermore, changes to $p(v)$ and $f(v)$ resulting from local changes to $v$, can be computed efficiently.

This is a candidate problem for Monte Carlo sampling, where samples from $\mathbb{Z}_{2}^n$ are drawn according to $p(v)$. The drawbacks of this method is that sometimes the estimated mean can suffer from large statistical in cases where f(.) is large for rare events.

I’m new to the field of machine learning and was wondering if there’s a branch of this field that addresses problems of this nature. Any references would be greatly appreciated.


texts: How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction?

texts:
In the section, "Off-Policy Prediction via Importance Sampling", found in the chapter on monte carlo methods of the second edition of Sutton and Barto's, "Reinforcement Learning: An Introduction", the importance-sampling ratio $\rho_{t:T-1}$ of a sequence of action-state transitions $S_t, A_t, S_{t+1}, A_{t+1}, ..., A_{T-1}, S_T$ (where $T$ is the final time step, that is, $S_T$ is the terminal state of the episodic task) is defined as the ratio of the probability of that particular sequence of state-action transitions occurring under the target policy $\pi$ to the probability of that sequence occurring under the behavior policy $b$:

where $p$ is the state-transition probability. The author introduces the importance-sampling ratio as a means of obtaining the state-value function of the target policy $v_{\pi}(s)$ from an episode generated according to $b$, as explained in the following excerpt from the text.

My question is this: how exactly does multiplying the return under the behavior policy by the importance-sampling ratio in equation 5.4 produce the correct expectation for the state value function of the target policy? It is unclear to me how the ratio "transforms the returns to have the right expected value".


texts: Format of data in SQL for machine learning

texts:
I am a beginner at Machine Learning and am starting out on a ML project. I have a large chunk of the source material and have started extracting the data from it to be stored in SQL (initial test with SQLite, but that is going to be insufficient for production).

The question that I am now facing that I can't find any kind of answer to is to what extend to preprocess the data that I store for best performance?

For example, ML methods are usually bad at handling categories and need them to be more like 0/1 values in a lot of columns showing the category rather than the category as a string in a single column. As I have many different such cases for a single row it would mean a lot of extra columns in SQL to achieve this preprocessing. I will also be using different ML methods like regression and classification on the data so exact preprocessing requirementsmight be hard to predict.

The data consists of %, times, categories, string labels and more. I will have to do some additional processing after retrieval from database regardless of how much preprocessing I am doing beforehand as some preprocessing is just not feasible (or even possible) to store completely prepared in SQL. % is of course easy, but when to do what for many of the other forms of data still eludes me.

Setup is single machine for daily data retrieval (small updates), data extraction and storage, modelling (unknown update interval) and predictions (multiple daily). Since I will be using many different models and aggregate prediction results I am very keen to have high performance without having to go nuts about it. I work in Python but can shift c/Java-like language if significant gains can be shown. Current estimate is for around 10 million data points, but that could easily be 10 times that number when broken down into categories.

As I am new at this I think it fair that you ask for clarifications if I have left out anything of relevance in determining what the best format for data in the SQL should be. I realise that performance is not as clear as desired, but I don't know what the bottleneck is going to be. Small investments like some additional RAM is not really a bottleneck compared to the need for an additional machine to run some portion of the process.


texts: Transform an Autoencoder to a Variational Autoencoder?

texts:
I would like to compare the training by an Autoencoder and a variational autoencoder. I have already run the traing using AE. I would like to know if it's possible to transform this AE into a VAE and maintain the same outputs and inputs.

Thank you.


texts: Pyspark coverting timestamps from UTC to many timezones

texts:
This is using python with Spark 1.6.1 and dataframes.

I have timestamps in UTC that I want to convert to local time, but a given row could be in any of several timezones. I have an 'offset' value (or alternately, the local timezone abbreviation. I can adjust all the timestamps to a single zone or with a single offset easily enough, but I can't figure out how to make the adjustment dependent on the 'offset' or 'tz' column.

There appear to be 2 main ways of adjusting a timestamp:
using the 'INTERVAL' method, or using pyspark.sql.from_utc_timestamp.

Here's an example:

data = [
  ("2015-01-01 23:59:59", "2015-01-02 00:01:02", 1, 300,"MST"),
  ("2015-01-02 23:00:00", "2015-01-02 23:59:59", 2, 60, "EST"),
  ("2015-01-02 22:59:58", "2015-01-02 23:59:59", 3, 120,"EST"),
  ("2015-03-02 15:59:58", "2015-01-02 23:59:59", 4, 120,"PST"),
  ("2015-03-16 15:15:58", "2015-01-02 23:59:59", 5, 120,"PST"),
  ("2015-10-02 18:59:58", "2015-01-02 23:59:59", 4, 120,"PST"),
  ("2015-11-16 18:58:58", "2015-01-02 23:59:59", 5, 120,"PST"),
  ("2015-03-02 15:59:58", "2015-01-02 23:59:59", 4, 120,"MST"),
  ("2015-03-16 15:15:58", "2015-01-02 23:59:59", 5, 120,"MST"),
  ("2015-10-02 18:59:58", "2015-01-02 23:59:59", 4, 120,"MST"),
  ("2015-11-16 18:58:58", "2015-01-02 23:59:59", 5, 120,"MST"),
  ...
]

(I realize the offset and tz columns aren't consistent - this isn't real data)

df = sqlCtx.createDataFrame(data, ["start_time", "end_time", "id","offset","tz"])
from pyspark.sql import functions as F

these two options both do what is expected:

df.withColumn('testthis', F.from_utc_timestamp(df.start_time, "PST")).show()
df.withColumn('testThat', df.start_time.cast("timestamp") - F.expr("INTERVAL 50 MINUTES")).show()

But if I try to replace the "PST" string with df.tz, or the " 50 " string with df.offset.cast('string'), I get a type error:

  TypeError: 'Column' object is not callable

I've tried variations on this, but to no avail.


texts: What is the feedforward network in a transformer trained on?

texts:
After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns.

What I learned about neural nets is that they learn based on a target variable, through back propagation according to a particular loss function.

Looking at the architecture of a Transformer, it is unclear to me what the target variables are in these feed forward nets. Can someone explain this to me?


texts: packages installed after activating conda environment

texts:
I want to know after creating and activating a conda environment in terminal. Say venv:

conda create -n venv
source activate venv

Then the prompt will come with the enviroment name (venv).
would the packages installed   (say conda install tensorflow without --name venv) after environment activation only effect inside the environment venv? Or would it affect outside venv?
I don't want to mess up my environments.


texts: What are some key difference in functionality between ProM and Apromore?

texts:
I would like to know what are some use cases where one would use ProM and where one would use Apromore.
I have traces and events from logs that I would like to use process mining on (mapping a business process from these informations, filter to isolate types of cases, estimating average time for each event, estimating the percentage of cases that go through a branch, visually, etc.), but I have a hard time choosing a tool. Also, the two seem able to interact; therefore, I have trouble understanding the key features of both.
ProM seems to be a more desktop application, while Apromore web-based, distributed via a Docker image, but functionality-wise, what are some differences?


texts: How are pictures pre processed before being used as ML data

texts:
So I was watching this YouTube video
So basically the professor used ML to generate random faces in order to create data for a Kaggle challenge.
When I looked into the data file, I was expecting to see *.jpg or png files, but all I found was CSV files and based on his video, his model generated some png files.
what happened? how did he turn jpg into CSV? can we not use jpg?


texts: Adding hour data to Date feature in time series prediction

texts:
I'm a noobie with regards to working with time series data. I have a time series data which has date column in YYYY-MM-DD and a hour column containing 0 to 23 representing hour of the day and the value containing a value between 70 to 100 which is what we need to predict.

Models I tried:
I checked for Stationarity of the data and I split the dataset based on time(oldest 80% train and newest 20% as a test) and tried two models: AR and ARIMA. AIC score for ARIMA was around 2500.0

My questions are: Earlier I discarded the hour feature and built models with only Date feature. So Will Adding hour data improve my model? If I'm adding hours, Should I keep it as a separate column or should I include the hour in the YYYY-MM-DD date feature?

Thanks in advance


texts: GridSearch returns worse results than default configuration

texts:
I'm using the following code for accuracy score calculation. Why is it so that the default configuration gives better result than GridSearch?

Default configuration

clf = svm.SVC(kernel='rbf', gamma='auto')
clf.fit(x_train, y_train.values.ravel())

y_train_pred = clf.predict(x_train)
y_test_pred = clf.predict(x_test)

print('Train set accuracy: '+'{}'.format(metrics.accuracy_score(y_train, y_train_pred)))
print('Test set accuracy: '+'{}'.format(metrics.accuracy_score(y_test, y_test_pred)))

Train set accuracy: 0.861101243339254
Test set accuracy: 0.8480113636363636

GridSearch configuration

param_grid = {'C': (0.001, 0.01, 0.1, 1, 10),
              'kernel': ('linear', 'poly', 'rbf', 'sigmoid'),
              'class_weight': ('balanced', None),
              'gamma' : ('scale', 'auto'),
              'shrinking': (True, False)}

grid_search = GridSearchCV(svm.SVC(gamma='scale'), param_grid, cv=5)
grid_results = grid_search.fit(x_train, y_train.values.ravel())

print(grid_results.best_score_)
print(grid_results.best_estimator_)
print(grid_results.best_params_)

0.8373001776198934
SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)
{'C': 1, 'class_weight': None, 'gamma': 'auto', 'kernel': 'rbf', 'shrinking': True}


texts: Tools to perform SQL analytics on 350TB of csv data

texts:
In short, what would be the best method/tricks/techniques/tools for performing ad hoc sql (style) queries on 350TB of csv data? Would there be other options, tool wise that would do it faster if we dropped the "sql" requirement?

Is my best option Hive and as many servers I can't get my hands on? Would spark be of any benefit since this size of dataset wouldn't come close to fitting into memory? I have other related questions /ideas but don't want to bog this down.

Background

The data is originally stored in a binary format that gets converted into an ASCII. The ASCII file is getting turned into CSV (actually tab separated but whatever). 1 binary file = 1 ASCII file = 1 CSV file. The data format is pretty simple, 200 to 500 header columns, each row is a sample of an attribute in 1-second intervals.

Example (first row is header):
t,attribute1,attribute2...<attributeX>
1,val1,val2...<valx>
2,val1,val2...<valx>
3,val1,val2...<valx>
...
X,val1,val2...<valx>

Example queries that could be ran.

1) Take t value (t is non-unique, as each file starts at 1 and ends at some random time) and compare different attribute values between t value of 100-1000 for all other attributes that were sampled at 100-1000.

2)There is UTC value as an attribute value (one of the columns), and same idea as #1 we might want to compare all attributes between X dates and compare those attributes to Y data that is from a different UTC range.

3)We might want to figure out a single particular attribute and trend that from the earliest time we saw that attribute for a different particular attribute.

These are my best examples as provided to me but other types of queries could exist. If you do have a question of why don't we cut out the middle man of the conversions of the binary data to CSV data, we could theoretically. However, the program that converts the files is quite old and is very involved.


texts: Modelling latent online shopping states from e-commerce website clickstream data - What type of method to use?

texts:
I am new to Data Science but really enthusiastic about it. Really appreciate communities like this one, so thanks in anticipation.

I entered a competition to solve a data mining challenge for an e-commerce website. The challenge is as follows:

  Build a model based on clickstream data, to detect the phase of the
  customer in the customer journey (or related intent of customer).

We are not given specific definitions of the customer phases, instead we are supposed to come up with them ourselves based on observable behaviours of users (and maybe academic literature). I believe there are three main shopping states:

Directed buyers - Goal oriented, specific product in mind.
Deliberate searchers - Know the basic category, but not a specific product in mind.
Hedonic browsers - Similar to window shoppers in real life.

After cleaning up the data, I have a table with session-level summary variables such as:

Category breadth: How many unique categories a user visited in a given session
Product to category ratio: Time spent in category pages vs product pages
Session source: (Direct, Search Engine, Social media, Referral)
Added item to cart: Yes/No
Ordered item: Yes/No
etc

I have the raw analytics export data (you can see all variables in the IBM source bellow), so can basically make any other variables I want. But thought these are good enough to explain myself.

My question is:

What method would you use in order to come up with Latent Shopping States for these users? Preferably I would like to obtain the probability that each individual user is in a specific shopping state. As well as some sort of weight for the factors that compose each state (e.g. Directed shoppers have a very high Category Breadth)

Initially I thought some sort of clustering algorithm such as K-Means, but that does not give me probabilities. I am now leaning towards Latent Class Analysis, but its still a bit abstract to me.

If I can provide you with any additional information let me know!

Thanks,

Sources:

Ramachandran, Vandana; Viswanathan, Siva; and Lucas, Hank, "STATE OF
SHOPPING AND THE VALUE OF INFORMATION: INSIGHTS FROM THE CLICKSTREAM"
(2010). ICIS 2010 - http://aisel.aisnet.org/icis2010_submissions/254/
IBM Digital Analytics Export info - http://www.ibm.com/support/knowledgecenter/SSPG9M/Export/Export/digitaldatafeed.html


texts: Beginner in programming and data science with 100 hours to spend learning the basics

texts:
I started using the below link to teach myself data science, with some mathematical knowledge (including calculus, linear algebra, stats/probability) but very little programming experience:
Quora - Roman Trusov's answer on how to learn Data Science in 100 hours.
The above link has the following starting advice:

You will need an RDBMS to handle the data, so the first day would look like this:

Install and configure MySQL. Import the dump into the database
Read SQL basics. Spend some time doing simple exercises to get the hang of manipulating the data...

After a few days, I'm still stuck at this stage. I'd like to learn how to do data science, and was hoping that following the plan in the above link would achieve that quickly. But I'm starting to feel it is not working. Is debugging MySQL installations an important part of becoming a data scientist - so I should just get advice on that and try harder? If not, what can I do differently to achieve my goals of learning data science in similar time frame?


texts: Why does cross validation have a pessimistic bias?

texts:
My course notes list two reasons why cross-validation has a pessimistic bias. The first one is that the accuracy is measured for models that are trained on less data, which I understand. However, the second reason I don't understand. Supposedly, when we do cross validation and divide our data D into training sets D_i and test sets T_i, then the D_i and T_i are not independent (and even complementary) given D.
However, I don't see why this is different from the situation where we use a fixed testset: if we have a training set D and test set T, than T and D are also not independent given the union of D and T. In this case there is no bias, so I would expect there to be no bias for cross-validation either (apart from the fact that the model is trained on less data). Of course, since the different models that we train for cross validation use overlapping data, I would expect their accuracy to be correlated, which could lead to a higher variance, but I don't see how this could give a bias.