texts: What is an efficient clustering approach for grouping multivariate timeseries data into optimally persistent states?

texts:
I have a length 200000 time series of six dimensional data describing the evolution of a chaotic dynamical system. I am exploring ways to partition the state space of the data into minimally interacting regions; I want the least transitions from cluster to another in my time series. Other than not being interested in the trivial case of only 1 cluster I have no restrictions on cluster number though I expect it to be small (K<6 say).
The purpose of this is to try and identify quasi-stable basins of attraction in the chaotic system.

I have tried Hidden Markov Models with both Gaussian and Gaussian Mixture emissions, but they do not well capture the very non-gaussian structure of my data in phase space. I have also tried using DBSCAN with some success but since it didnt treat data as a time series, the parameters had to be very carefully tuned to get good results, which left me dissatisfied.

Do any other approaches leap to mind?


texts: Best technique for testing if an event causes a trend?

texts:
Example - A major holiday is coming up and the frequency of a product being purchased starts increasing significantly as it gets closer to the day of the holiday. Then after the holiday, the frequency starts decreasing to "normal" non-holiday frequencies.

For that example, I would be trying to find the best technique to test if the holiday is causing any trends/significant increases in frequency compared to the normal/average purchase frequency when there isn't a holiday.

Techniques I've tested -
Linear Regression
Theil-Sen Regression
RANSAC Regression
Mann-Kendall test
Skew
Pearson Correlation

All of the techniques listed above seem to work for this to a certain extent. However, I'm not sure if its better to use a combination of these or stick with one.

Note - This would be used for descriptive purposes, not predictive. The goal is to confirm if an event creates a trend or not.

Let me know if I need to add more information or clarify anything.

Thanks!


texts: Adding batch normalization layer to VGG16 network


texts: Whether AUC can be calculated as average of sensitivity and specificity or not?

texts:
I have gone through few of the paper for AUC calculation, following paper has mentioned AUC can be calculated as

AUC = (Sensitivity +Specificity)/2.

A systematic analysis of performance measures for classification tasks

Please let me know that whether it is correct or not.


texts:
I've generated two clouds of 3d points from multivariate_normal

data = np.random.multivariate_normal([2,2,2],[[1,0,0],[0,5,0],[0,0,10]],
size=500)
data = np.vstack((data, np.random.multivariate_normal([-2,-2,-2], [[1,0,0],[0,5,0],[0,0,10]], size=500)))
data = data - data.mean(axis=0)

And try to do PCA like this

covmat = np.cov(data.T)
v, W = np.linalg.eig(covmat)

And draw:

def get_vec(eig_v, eig_vec):
    t = np.linspace(0, eig_v)
    return np.array([np.array(v * eig_vec) for v in t])

def ang(v1, v2):
    return np.rad2deg(np.arccos(np.dot(v1,v2)/np.linalg.norm(v1)/np.linalg.norm(v2)))

l1 = get_vec(v[0], W[:,0])
l2 = get_vec(v[1], W[:,1])
l3 = get_vec(v[2], W[:,2])
x = data[:,0]
y = data[:,1]
z = data[:,2]
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(l1[:,0],l1[:,1],l1[:,2], c='r')
ax.plot(l2[:,0],l2[:,1],l2[:,2], c='b')
ax.plot(l3[:,0],l3[:,1],l3[:,2], c='y')
ax.scatter(x,y,z,c='g')
plt.show()

This is what I get:

It's clearly visible that axes are not orthogonal. I've checked it and they seem to be orthogonal with regard to numbers:

print(ang(W[:,0], W[:,1]))
print(ang(W[:,0], W[:,2]))
print(ang(W[:,1], W[:,2]))

90.00000000000003
89.99999999999999
90.0
Could it be that such a tiny error makes that much visual difference?


texts: Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

texts:
Which is better for accuracy or are they the same?
Of course, if you use categorical_crossentropy you use one hot encoding, and if you use sparse_categorical_crossentropy you encode as normal integers.
Additionally, when is one better than the other?


texts:
I'm currently trying to train a model on a large csv file (>70GB with more than 60 million rows). To do so I'm using tf.contrib.learn.read_batch_examples.
I'm struggling in understanding how this function actually reads the data. If I'm using a batch size of e.g. 50.000, does it read the first 50.000 lines of the file? If I want to loop over the whole file (1 epoch) do I have to use num_rows/batch_size = 1.200 number of steps for the estimator.fit method?

Here is the input function im currently using:

def input_fn(file_names, batch_size):
    # Read csv files and create examples dict
    examples_dict = read_csv_examples(file_names, batch_size)

    # Continuous features
    feature_cols = {k: tf.string_to_number(examples_dict[k],
                                           out_type=tf.float32) for k in CONTINUOUS_COLUMNS}

    # Categorical features
    feature_cols.update({
                            k: tf.SparseTensor(
                                indices=[[i, 0] for i in range(examples_dict[k].get_shape()[0])],
                                values=examples_dict[k],
                                shape=[int(examples_dict[k].get_shape()[0]), 1])
                            for k in CATEGORICAL_COLUMNS})

    label = tf.string_to_number(examples_dict[LABEL_COLUMN], out_type=tf.int32)

    return feature_cols, label

def read_csv_examples(file_names, batch_size):
    def parse_fn(record):
        record_defaults = [tf.constant([''], dtype=tf.string)] * len(COLUMNS)

        return tf.decode_csv(record, record_defaults)

    examples_op = tf.contrib.learn.read_batch_examples(
        file_names,
        batch_size=batch_size,
        queue_capacity=batch_size*2.5,
        reader=tf.TextLineReader,
        parse_fn=parse_fn,
        #read_batch_size= batch_size,
        #randomize_input=True,
        num_threads=8
    )

    # Important: convert examples to dict for ease of use in `input_fn`
    # Map each header to its respective column (COLUMNS order
    # matters!
    examples_dict_op = {}
    for i, header in enumerate(COLUMNS):
        examples_dict_op[header] = examples_op[:, i]

    return examples_dict_op

Here is the code im using to train the model:

def train_and_eval():
"""Train and evaluate the model."""

m = build_estimator(model_dir)
m.fit(input_fn=lambda: input_fn(train_file_name, batch_size), steps=steps)

What would happen if I would call the fit function again with the same input_fn. Does it start at the beginning of the file again, or will it remember the line where it has stopped last time?


texts:
Both PyTorch and Tensorflow Fold are deep learning frameworks meant to deal with situations where the input data has non-uniform length or dimensions (that is, situations where dynamic graphs are useful or needed).

I would like to know how they compare, in the sense of paradigms they rely on (e.g. dynamic batching) and their implications, things that can/cannot be implemented in each one, weaknesses/strengths, etc.

I intend to use this info to choose one of them to start exploring dynamic computation graphs, but I have no specific task in mind.

Note 1: other dynamic computation graph frameworks like DyNet or Chainer are also welcome in the comparison, but I'd like to focus on PyTorch and Tensorflow Fold because I think they are/will be the most used ones.

Note 2: I have found this hackernews thread on PyTorch with some sparse info, but  not much.

Note 3: Another relevant hackernews thread, about Tensorflow Fold, that contains some info about how they compare.

Note 4: relevant Reddit thread.

Note 5: relevant bug in Tensorflow Fold's github that identifies an important limitation: impossibility to do conditional branching during evaluation.

Note 6: discussion on pytorch forum about variable length inputs in relation to the algorithms used (e.g. dynamic batching).


texts:
I am given a data set with 500 features and I am told to apply PCA and find most important 2 features. I wrote some MATLAB code but I am very confused with all this PCA thing.
mean(projectData)
projectData = bsxfun(@minus,projectData,mean(projectData));

[coeff,score,latent] = pca(projectData);

This what I wrote but I do not know what to do next. I think I must print latent and take 2 values which are highest and assume their index to be feature number. This is what I understand. Please help me.


texts: Is Clustering used in real world systems/products involving large amounts of data? How are the nuances taken care of?

texts:
I am working with a real dataset of around 2.5 million short text data (posts, around 10 words in an average data element). I wanted to group similar posts together. So, I applied KMeans clustering on the dataset after doing the standard pre processing. After vectorization (I did TfIdf), the matrix was huge (2.5 million rows and around 60 k columns). I did an SVD + LSA transformation which made it smaller (2.5 million x 150 ). I gave 200 as the number of clusters and applied K Means. I must say that the results are not very bad. There are clusters which have similar posts (not the semantics, but the words similar), but there are those clusters which are randomly placed too.

Because of the large size of the data, I am not able to effectively visualize the results or measure the goodness of the clusters. Though I tried implementing silhouette coefficient, elbow methods for finding the optimal number of clusters, I was getting continuous memory errors. I don't know if 150 is the right number to use during dimensionality reduction. Could you please suggest any other methods for solving these issues?

When I reached this point, I started doubting the effectiveness of clustering as a method. I am curious to know if anybody is using clustering in large production scale systems, if so, what kind of techniques do people use?

Please find the code below:

`

 tfidf_vectorizer = TfidfVectorizer(max_df=0.5, max_features=200000,
                                 min_df=0.001,decode_error='ignore', stop_words='english',strip_accents='ascii',use_idf=True, ngram_range=(1,2),tokenizer=tokenizeAndFilter,lowercase=True)`

`    tfidf_matrix = tfidf_vectorizer.fit_transform(tokens.values()) #fit the vectorizer to synopses

    svd = TruncatedSVD(150)
    normalizer = Normalizer(copy=False)
    lsa = make_pipeline(svd, normalizer)

    X = lsa.fit_transform(tfidf_matrix)
    feature_names = tfidf_vectorizer.get_feature_names()

    with open(path+'terms.csv','w') as f:
       for col in tfidf_matrix.nonzero()[1]:
          f.write(feature_names[col]+','+str( tfidf_matrix[0, col]))
          f.write('\n')

        sil_dict={}
        for n_clusters in range(50,500,10):
            km=KMeans(n_clusters=n_clusters,init='k-means++',max_iter=500,n_init=1)
        km.fit(X)
            label = km.labels_
            sil_coeff = silhouette_score(X, label, metric='cosine',n_jobs=10)
    #        print("For n_clusters={}, The Silhouette Coefficient is {}".format(n_cluster, sil_coeff))
        sil_dict[n_clusters]=sil_coeff
            with open('kmeans_model'+str(n_clusters)+'.pkl','wb') as f:
               cPickle.dump(km,f)

    original_centroids=svd.inverse_transform(km.cluster_centers_)
    order_centroids=original_centroids.argsort()[:,::-1]

    terms=tfidf_vectorizer.get_feature_names()
    clusters=collections.defaultdict(list)
    k=0
    for i in km.labels_:
        clusters[i].append(tokens.values()[k])
        k+=1
        for i in range(10):
            print i
            for j in order_centroids[i, :10]:
                  print(terms[j] )
        for key,valueList in clusters.items():
            with open(path+'\\'+OUTPUTFILE+'_'+str(key)+'.csv','w') as f:
                 for value in valueList:
                    f.write(value+'\n')

`


texts: Is it a good idea to tune the number of folds for cross validation when tuning hyperparameters of RF

texts:
I'm new to data science. I'm trying to get the best model for Random Forest. Unfortunately, I'm not sure if my idea can produce a good generalized model.

1) I have split data to TrainingSet (70%) and TestSet (30%)

2) Then randomly selected hyperparameters for RandomForest and a number of folds for CrossValidation between (2-15)

3) Then I fetch the TraingSet data to the RandomForest learner

4) Then do CrossValidation of the model
- from CrossValidation I'm getting array with predictions

5) Measure Accuracy of prediction from CrossValidation against targets from the TrainSet

6) Repeat all steps and try minimize the AccuracyError

Is this a good way to get best generalized model?

Do I need to split data into TrainSet and TestSet?

OR I should I search for the optimal hyperparameters and number of folds with all data? I have feeling I don't need to split data when using k-fold CrossValidation during Hyperparameters tunning.


texts:
With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a mix of negative percentage in there which is undesirable.
Is there a way to handle  unwanted negative numbers in pandas python before running it through a machine learning model.


texts: Orange 3 - Same prediction for all of my data when using impute widget

texts:
When I use a linear or random forest regression to train from data, I can make predictions with test data. I have a problem when I introduce the "impute" data widget in my pipeline to fill in the missing values from the training data. My predictions then are all the same for all rows, even though the data table is correct. I tried to swap the impute widget for the pre-processor and I have the same behaviour. Below is my graph:

This works correctly:

Adding an impute widget and all my predictions become the same values:


texts: intent detection and slot filling using tensorflow.js

texts:
I'm still getting up to speed with machine learning, but I'm aware of the papers on joint intent detection and slot filling by Bing Liu & Ian Lane, and another by Xiaodong Zhang and Houfeng Wang - and I'm sure there would be others.

There are several implementations available on GitHub:

liu/lane by brightmart
liu/lane by HadoopIt
liu/lane by applenob

I'm hoping to be able to train a model using python and TensorFlow on the back-end, and then use the model in Node and browsers using TensforFlow.js

If I feed the training process with a limited set of patterns of the form:

flights from @City:origin to @City:destination @Date:date
flights from @City:origin to @City:destination @DatePeriod:datePeriod
airfares from @City:origin to @City:destination
...

...then the vocabulary would be fairly limited:

flights
from
to
airfares

Plus known entity values:

Boston, New York...
tomorrow
next week
saturday...
first...
january...

I have a few noob questions:

Is this strategy valid?
If my vocabulary is, say, 100 words then how big is the model that would have to be downloaded by the web app?
How difficult would it be to implement the .predict() side in TensorFlow.js?
Would there be any noticeable performance difference between the TensorFlow.js .predict() implementation and iterating through a list of regular expressions?


texts: what do we actually study in machine learning?

texts:
I just started learning machine learning and learned a few basic algorithms and there is one stupid doubt in my mind and I am unable to find the answer of it. What do we actually study in machine learning algorithms that solves kinds of problems based on the data given to the algorithms?

Take an example of linear regression:

We give data.
We tell machine how to calculate error.
Using gradient descent also we are telling machine how to optimize and based on all above some optimized parameters come out.

Why there is actually a "learning" word in this algorithm? Although we are the telling how to perform steps and we have provided all the mathematics in this algorithm.


texts: Normalization in Neural Networks Regression in MATLAB

texts:
I have 20 samples for a regression fitting purpose in Neural Network Toolbox (nnstart, nftool) in MATLAB . I have 3 inputs of 20 samples each (3 X 20 ) and 1 output (3 X 20, as the output number is more because of result with each input parameter), 1 hidden layer with 4-5 hidden neurons.
I am getting worse results without normalization in 70% training, 15% validation and 15% testing.
But even using min-max normalization isn't somehow helping it to get better. I used log normalization on output (target) data along with min-max (not on inputs). The results are now accurate. But I believe that this is not the right way.
Therefore, if I use log normalization on output and min-max on input only, is that correct way?
Is using MATLAB GUI Neural Network toolbar (nnstart or nftool) good for the purpose?
What are the alternatives in this case?


texts: Why does Ensemble Averaging actually improve results?

texts:
Why does ensemble averaging work for neural networks? This is the main idea behind things like dropout.

Consider an example of a hypersurface defined by the following image (white means lowest Cost). We have two networks: yellow and red, each network has 2 weights, and adjusts them to end up in the white portion.

Clearly, if we average them after they were trained we will end up in the middle of the space, where the error is very high.


texts:
I have some problems understanding/interpreting the C-Index cluster quality measure. So, if we have

$c(x_i, x_j) = 1 $ if $ x_i, x_j $ in the same cluster; $0$ else

$\Gamma = \sum_ {i=1}^{n-1}\sum_ {j=i+1}^n d(x_i,x_j)*c(x_i,x_j)$

$\alpha = \sum_ {i=1}^{n-1}\sum_ {j=i+1}^n c(x_i,x_j)$

$min=$ sum of the $\alpha$ smallest $d(x_i,x_j)$ of distinct pairs $x_i,x_j$ where $x_i \neq x_j$

$max=$ sum of the $\alpha$ largest $d(x_i,x_j)$ of distinct pairs $x_i,x_j$ where $x_i \neq x_j$

then the C-Index is defined as $C=\frac{\Gamma - min}{max - min}$

The result is a value in $[0, 1]$, where lower values indicate a better cluster quality.

So, here are some things I get from this value:

if all elements in a cluster are close together and all clusters are far apart, we can get  $\Gamma=min$, which means $C=0$
Analogously, in a worst case scenario, all the observations that are furthers apart might be in the same cluster, so we would get $\Gamma=max$, which means $C=1$

Now, these are the things I'm unsure about:

First: If we only have a single cluster in our clustering (e.g. k-Means for $k=1$), then $\alpha$ is equal to the number of distinct pairs of observations, so $max=min$, which means $C=\frac{\Gamma - min}{max - min} = \frac{\Gamma - min}{min - min} = \frac{\Gamma - min}{0}$ and we get a division by 0. A similar problem occurs if we have $N$ observations in $N$ different clusters, since $c$ is always $0$ in that case. So, is it correct to say that the C-Index can only be used for clusterings with $k$ many clusters where $1 < k < N$, for $N$ observations?

Second: Is it reasonable to say that the C-Index is agnostic to the number of clusters (e.g. the value of $k$ in k-Means)? For instance, we might have 5 observations $x_1...x_5$ close to each other, but each put into a separate cluster $C_1...C_5$. Then, we might have a clusters $C_6=\{x_6, x_7\}$ where $x_6, x_7$ are very close to each other but far apart from all other observations. In that case, $\Gamma=d(x_6,x_7)$, $\alpha=1$, $\min=d(x_6,x_7)$, so $\Gamma=min$, which means $C=0$. That is, we have the best possible C-Index-Value, even though, intuitively, it might have been better to put $x_1...x_5$ into a single cluster.

Lastly, this is more about k-Means: if we use normal k-Means (not global k-Means), are we always guaranteed to reach $C=0$, for an unbounded number of iterations? I can't seem to find an example that wouldn't result in this.


texts: Identify patterns leading to error in structured log files

texts:
I am currently working on parsed log data from IoT devices (log entries are batched every 15 minutes). All info_* / err_* are counts of the related log events:
| machine_id | timestamp           | os_ver | days_since_reboot | cpu_load | info_featA | info_featB | err_badrequests | err_timeoutrequests |
|------------|---------------------|--------|-------------------|----------|------------|------------|-----------------|---------------------|
| 1          | 2020-09-18 12:25:00 | 0.01   | 12                | 0.59     | 12         | 543        | 0               | 0                   |
| 1          | 2020-09-18 12:40:00 | 0.01   | 12                | 0.83     | 1          | 23         | 1               | 0                   |
| ...        | ...                 | ...    | ...               | ...      | ...        | ...        | ...             | ...                 |
| 2          | 2020-10-28 11:40:00 | 0.02   | 2                 | 0.12     | 43         | 54         | 0               | 1                   |
| 2          | 2020-10-28 11:55:00 | 0.02   | 2                 | 0.43     | 32         | 54         | 3               | 5                   |
| 2          | 2020-10-28 12:10:00 | 0.02   | 2                 | 0.23     | 0          | 4          | 0               | 23                  |

I am tasked to find some common patterns that might lead to an error event (e.g an increase in info_featA entries precedes a positive count in err_badrequests). For this reason I've thought to re-encode error columns into a boolean (True if err_* > 0) but I am fairly lost on the right approach to go about sequence analysis or finding the conditions that can lead to an error.
Also, error type is not very important for my problem as I just care about the presence of a positive error count (regardless of its type):
| machine_id | timestamp           | os_ver | days_since_reboot | cpu_load | info_featA | info_featB | has_err |
|------------|---------------------|--------|-------------------|----------|------------|------------|---------|
| 1          | 2020-09-18 12:25:00 | 0.01   | 12                | 0.59     | 12         | 543        | 0       |
| 1          | 2020-09-18 12:40:00 | 0.01   | 12                | 0.83     | 1          | 23         | 1       |
| ...        | ...                 | ...    | ...               | ...      | ...        | ...        | ...     |
| 2          | 2020-10-28 11:40:00 | 0.02   | 2                 | 0.12     | 43         | 54         | 0       |
| 2          | 2020-10-28 11:55:00 | 0.02   | 2                 | 0.43     | 32         | 54         | 1       |
| 2          | 2020-10-28 12:10:00 | 0.02   | 2                 | 0.23     | 0          | 4          | 0       |

At the moment I have created lagged variables (up to 2 hours before) to check if any lagged variable is somewhat correlated with error counts through a correlation matrix. I am pretty sure this is not the way to approach such problem.

The main goal is to identify (if any) some common patterns in the observed metrics that lead to an error.


texts: text classification : comparing classification reports

texts:
I have a 4-leabelled text classification problem.

Could someone help me choose among the below text classifiers ?

I was advised to select the second one ( the one which uses both unigrams and bigrams ) but I cannot really see why.


texts: Why not train the final model on the entire data after doing hyper-paramaeter tuning basis test data and model selection basis validation data?

texts:
By entire data I mean train + test + validation

Once I have fixed my hyperparameter using the validation data, and choose the model using the test data, won't it be better to have a model trained on the entire data so that the parameters are better trained rather than having the model trained on just train data


texts:
What is the GPU ram required to train YOLO v4? More generally what is the ram requirements to work with the more interesting architectures and bigger datasets?
I see Nvidia has released the RTX 3090 with 24GB ram but the RTX 3080-3070 still maintain 10GB and 8GB and I wonder if that is sufficient for academic and hobby workloads or if a 3090 will be required.


texts: Features from images using opencv in Python

texts:
I want to start   with  image processing (image classification).
I installed  opencv  and plan  to use it    with Python.
Being starter  at   this, I am looking   for references   regarding  trivial  issues like:
I just  saved some images  and  want to  get their pixel  values  as  features ,  hoewever   when I try loading it  in opencv,  I am    getting differently  shaped   3-dimmensional  objects.
Some   references   or suggestion regarding this   issue  will be highly  appreciated.


texts: How to implement loss function for CNN-SVM implementation

texts:
I am using Matlab R2018b and am trying to infuse svm classifier within CNN. My plan is to use CNN only as a feature extractor and use SVM as the classifier. I know people have already implemented it a few years back either in tensorflow or in other platforms. In implementing this I got stuck at a point during backward propagation. I got puzzled about which loss function I need to implement to upgrade the gradients and the parameters.

Few points came up during this:

I got a feeling to implement the hinge loss here. But which form of hinge loss should I implement? Should I move on to the second form of hinge loss implementation for calculating loss during backward propagation?
Besides, calculating the backward loss, should I calculate the forward loss as well to find out the loss occurred in the model?

Any form of advice doing this CNN-svm infusion will be appreciated as I am unable to find any such material implemented in Matlab to get help.

Thanks.


texts: I just stumbled across Kur. At first look, it seems to be making deep learning easy in almost similar lines like Keras on a high level. What are the pros and cons of each and what is suggested to be used for beginners?


texts: What are some ways to transition from additive to multiplicative(or vice-versa) changes in time-series quickly?

texts:
Let's say I have a variable x and as time progresses, I want to update the value of x in additive/multiplicative manner using some attributes.
At t=t, new_x = x * some_percentage or new_x = x + some_absolute_number.
Is there some way to transition from additive change to multiplicative change or vice-versa and the number by which we can? It's fine if we focus only on one of the additive or multiplicative by reactively changing the percentage_increase or absolute_change.

A toy example task (hypothetical)
For a salary of 100k, an increase of 50% is significant but for a salary of 100(could be pocket money of some child), a 50% increase might not be enough. When we are discussing a smaller number of salaries, we need to increase the salary by absolute number. For a salary of 100, we can increase it by 70 to make it 170)

Are there any ways or literature to handle this which can adapt quickly?


texts: How are samples selected from training data in Xgboost

texts:
In Random Forest, each tree is not fed with the full batch of training data, only a sample.

How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?


texts: setting class weights for imbalanced dataset, how using EarlyStopping?

texts:
I want to train a CNN with Early Stopping (Keras).
The data set is imbalanced, so I have set class_weights to 'balanced' like follows:

class_weights=class_weight.comput_class_weight('balanced', np.unique(y_train),y_train)

I would like also use Early Stopping, and here is the problem.
Which metric for monitor should I use?
Because I have still balanced the data set with the class weights, shouldn't it be ok when using val_loss?

Hoping for hints and thank you in advance


texts: is it possible (and/or logical) to set feature importance for xgboost?

texts:
If I understand tree based methods correctly, it would be better for more important features to be toward the top of the tree.

Is there a way I can dictate this in xgboost? Similar to how I can assign weights to each sample/row, can I somehow enforce certain features to be more likely to be put on top compared to other features? (or some other way that enforces certain columns to play a more important role in the model?)

I am aware that you would want the data to guide the model rather than enforcing it but 1 logical case for this is when you have limited data and would like to try helping the process through your subject matter expertise.


texts: What is the stochastic part in stochastic gradient descent?

texts:
From my understanding, a stochastic process whose value at a particular instant is dependent on the previous values taken and every time the process is run, the path chosen can be different. We can know only the limits and confines the process stays after certain intial seeding value.

The weight values for a neural networks are updated via the stochastic gradient descent method. What is the stochastic part about it? After some initialization of the variable, error function accumulated will be same if the same set of input data is provided after initialization in each test. Why is the relevance of the term stochastic? Where is the scope for the randomness and what is the source of this randomness?


texts: How to read specific column with specific row in x_test using python

texts:
I have a dataset with four input. here I am trying to predict X1 input in next time period using lstm model.

Here I put x_train 80% and x_test 20% .
In x_test first input some of rows are with 0 values.

What I want is I want to read that 0 value separately. I tried and it didn't work for me. Can anyone help me to solve this problem.
Here my code:

data=data.values
scaler_x = preprocessing.MinMaxScaler(feature_range =(0, 1))
x = np.array(x).reshape ((len(x),4 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(0, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)
train_end = 80
x_train=x[0: train_end ,]
x_test=x[train_end : ,]
y_train=y[0: train_end]
y_test=y[train_end :]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))

After writing code for lstm I tried to predict value for x_test. Before that I want to specify that rows with 0 values.

x_test_n = np.flip(x_test_n, axis=0)
curr_frame = x_test_n[0]

Then it just read first row with four input together.
What I want is to read first input with 0 values.

Image for what I got:

What I want to read

After  changing the code:

pred=[]
for index, row in x_test_n.iterrows():
  if index_row, index_col = np.where(x_test_n == 0)
    x_test_n.iloc[index] = pred[-1]
    row['X_test_n'] = pred[-1]
    pred1 = model.predict(x_test_n)
    pred.append(pred1)

It gave me an error:

  if index_row, index_col = np.where(x_test_n == 0)


texts: How to handle fixed values for variables in pre-processing

texts: I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are real-valued but all same values. When doing standardization of the variables so that each is a zero mean and variance 1, these variables give NaN values. Therefore, is it ok to exclude such variables (irrespective of being categorical or real-valued) that contain constant values from the normalization/standardization step? These variables are important as features hence I cannot delete them. Is there any other way to handle such variables?


texts: How to find the residuals of a classification tree in xgboost

texts:
So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on the residuals of the tree from the last round?

Say I have the following iris dataset:

And I get 3 error after running a classification tree. Then how do I fit another classification tree based on what I got wrong?

And after then, how would I weight them to form an ensemble?


texts: In a recommender system, how can you normalise the similarity between two arbitrary users?

texts:
Consider the following problem:

There are 1000 users, 100 items (movies, for example), and 10000 ratings. The probability of a user, $u$, rating a movie, $i$, is $\mathbb{P}(R_{u_i}=\text{yes})=\frac{1}{10}$, and the probability of any two users rating the same movie is $\mathbb{P}(R_{u_i}=R_{v_i}=\text{yes})=\frac{1}{100}$.

If we let a random variable $x_i=1$ if $i$ is rated by both $u$ and $v$, and $0$ otherwise, then we have $$E\Big[\sum_{i=1}^{100}x_i\Big]=100\times\big(1\frac{1}{100} + 0\frac{1}{100}+0\frac{1}{100}+ ... \big)=1.$$ The similarity between $u$ and $v$ is $$PC(u, v)=\frac{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})(R_{v_i}-\bar{R_v})}{\sqrt{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})\sum_{i\in T_{uv}}(R_{v_i}-\bar{R_v})}}=1,$$ where $T_{uv}$ is the set of all items rated by $u$ and $v$, and I used the fact that $R_{u_i}$ and $R_{v_i}$ become irrelevant in comparison to $\bar{R_u}$ and $\bar{R_v}$. This is an obvious problem because it means that in situations like this you will always have a perfect similarity between $u$ and $v$.

I believe that the normalisation solution is something like this:

$$PC(u, v)=\frac{\text{min}(|T_{uv}|, \beta)}{\beta}PC(u, v),$$ where $\beta\in[25, 50]$.

I don't understand the logic behind this normalisation. Could you explain please?


texts: How to interpreting the exponential coefficent in poisson regression with offset?

texts:
I am trying to find the village level risk factors for malaria. Therefore, I ran a poisson model in r with the prevalence of malaria(y) as dependent variable, altitude(x1) and Forestation(x2) as independent variable and log of Population(x3) as offset.

glmer(y~x1+x2+(1|cluster), family = poisson, offset = log(x3))

From what I have read, I understand that the interpretation of model with offset is different than a non-offset model.

It would be very helpful, If any one can clear the air on how to interpret the coefficients and exponential coefficient in the above-mentioned case.

Thanks


texts:
I have a scenario in which I'm required to run my analysis at the Account level. One of the features that I'd like to look at is the no. of subscriptions against an account. There can be multiple subscriptions against one account. I wonder how I can "aggregate" these multiple subscriptions and roll them up at the Account level, such that I have a single row for each account.

I could think of binary encoding but I have 5000 products and that would require creating these many features.


texts: 1D convolutional neural network validation improvement

texts:
I created 1D CNN in Keras, but I'm having issues with validation loss and accuracy.
I have 24k records, 22 features. Is my model overfitting or what is going on so validation loss and accuracy is inconsistent? They rise and drop over epochs.
Here is my model
#model
n_obs, feature, depth = X_train.shape
batch_size = 20
kernel_size = 2
pool_size = 2
filters = 16

inp = Input(shape=(feature, depth))
C = Conv1D(filters=filters, kernel_size=kernel_size, strides=1)(inp)

C11 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(C)
A11 = Activation("relu")(C11)
C12 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A11)
S11 = Add()([C12, C])
A12 = Activation("relu")(S11)
M11 = MaxPooling1D(pool_size=pool_size, strides=2)(A12)

C21 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(M11)
A21 = Activation("relu")(C21)
C22 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A21)
S21 = Add()([C22, M11])
A22 = Activation("relu")(S11)
M21 = MaxPooling1D(pool_size=pool_size, strides=2)(A22)

C31 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(M21)
A31 = Activation("relu")(C31)
C32 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A31)
S31 = Add()([C32, M21])
A32 = Activation("relu")(S31)
M31 = MaxPooling1D(pool_size=pool_size, strides=2)(A32)

C41 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(M31)
A41 = Activation("relu")(C41)
C42 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A41)
S41 = Add()([C42, M31])
A42 = Activation("relu")(S41)
M41 = MaxPooling1D(pool_size=pool_size, strides=2)(A42)

F1 = Flatten()(M41)

D1 = Dense(21)(F1)
A6 = Activation("relu")(D1)
D2 = Dense(16)(A6)
D22 = Dense(8)(D2)
D3 = Dense(2)(D22)
A7 = Softmax()(D3)

model = Model(inputs=inp, outputs=A7)

lrate = LearningRateScheduler(0.001)
adam = Adam(lr = 0.001, beta_1 = 0.9, beta_2 = 0.999)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
history = model.fit(X_train, y_train,
                    epochs=200,
                    batch_size=batch_size,
                    verbose=2,
                    validation_data=(X_valid, y_valid),
                    callbacks=[lrate])

Training set is about 15k records, valid 1.5k and test 6k records. Although test set achieve 96% correct classification I'm wondering if this validation inconsistent behaviour is something I should fix.
What can I do about it?
EDIT: example data


texts: Backpropagation of convolutional neural network - confusion

texts:
I've already seen many articles about this topic and Backpropagation In Convolutional Neural Networks by Jefkine seems to be the best. Although, as author said,

  For the purposes of simplicity we shall use the case where the input image is grayscale i.e single channel C = 1.

Also, he uses stride = 1 and assumes only 1 filter, for the same purpose.

These are the final equations for backpropagation (taken from the article):

The autor's notation explained:

I figured out how to do the forward pass with stride, depth and more filters, but couldn't do the same with the backpropagation. Do you know where to put the 3rd dimension, stride and filters number in those equations?

Also, how to backpropagate the bias (assuming there's 1 bias per filter)?

Let me know if something is unclear.

Thanks in advance.


texts:
I was reading the FastText paper and I have a few questions about the model used for classification. Since I am not from NLP background, some I am unfamiliar with the jargon.
In the figure, what exactly is are the $x_i$? I am not sure what $N$ ngram features mean. If my document has total $L$ words, then how can I represent the entire document using $N$ variables ($x_1$,..,$x_n$)? What exactly is $N$?

$$-\frac{1}{N}\sum_{n=1}^Ny_n\log(f(BAx_n)) $$
If $y_n$ is the label, then what sense does it make to multiply it with the output vector after softmax (lables would be like 0,1,2,3,.. )? Does the author mean we take the $y_n$-th component of the output vector in loss calculation?


texts: Different available packages in TensorFlow virtualenv?

texts:
I have installed TensorFlow on Linux (Anaconda) by following the documentation which states that one should create and activate a virtual environment tensorflow. So far, so good (albeit it is not entirely clear why is this virtual environment is necessary when I want to incorporate TF to my existing env).

But when I activate the tensorflow environment I observe that several packages are unavailable in the new environment while available outside the environment at the same time:

$source activate tensorflow
$python

>>>import h5py

... No module named h5py

$source deactivate tensorflow
$python

>>>import h5py
>>>

No problem in this case.

I guessed that I should install the missing package in the tensorflow environment as well, but when I try to, I get informed that the package in question has already been installed and nothing happens:

$source activate tensorflow
$pip install h5py

Requirement already statisfied ....

The same inconsistency occurs with several other packages. What is the problem here?


texts:
I am trying to fit a regression model on a non linear data.
The features I have are around 12 and around 800 samples.
With the help of PyCaret, i tried to fit the data on to around 22 model, and then selected the best one (Ada Boost) and then tried further to tune it to get better result.
However, none of the models gave a positive R2 score, Ada Boost was the least worst performing algorithm.
This is the test (red) and predicted test output (green) from the selected algorithm.

After trying all various techniques and still not getting a decent result, can we infer that the features are not enough to account for the variation of the target variable ?
In other words the provided features don't explain best the target variable.
It may sound silly but am a beginner in Data Sciences, so please dont mind.


texts: The tag preprocessing says it is a data-mining technique ! Is it a correct definition?

texts: This site appears to give Wrong definition of preprocessing of data.Data-mining has no connection with preprocessing of data. The data preprocessing focuses on data preparation for developing software and execution. Is it desirable to rewrite the definition and its scope? How to restructure it ?


texts:
Recently I had a doubt as to what is the real purpose of pooling layers in neural networks is? The most common answer is

To select the most important feature
To increase the receptive field of the network

I feel that these are not real reasons for using a pooling layer because

There is no real need to select important features because the fully connected layer at the very end could be used to identify the most important features
The receptive field could be increased by increasing the kernel size in the successive layers.

So the only real reason for using pooling is to reduce to the size of the feature representation thus leading to smaller memory and computational footprint as the networks deeper.

Do you agree with the analogy? Do you feel there is any other reason as well?


texts:
What I tried:

# -*- coding: utf-8 -*-

from nltk.stem.snowball import GermanStemmer
st = GermanStemmer()

token_groups = [(["experte", "Experte", "Experten", "Expertin", "Expertinnen"], []),
                (["geh", "gehe", "gehst", "geht", "gehen", "gehend"], []),
                (["gebäude", "Gebäude", "Gebäudes"], []),
                (["schön", "schöner", "schönsten"], ["schon"])]
header = "{:<15} [best expected: n/n| best variants: 1/n | overlap: m]: ...".format("name")
print(header)
print('-' * len(header))
for token_group, different_tokens in token_groups:
    stemmed_tokens = [st.stem(token) for token in token_group]
    different_tokens = [st.stem(token) for token in different_tokens]
    nb_expected = sum(1 for token in stemmed_tokens if token == token_group[0])
    nb_variants = len(set(stemmed_tokens))
    overlap = set(stemmed_tokens).intersection(set(different_tokens))
    print("{:<15} [as expected: {}/{}| variants: {}/{} | overlap: {}]: {}".format(token_group[0], nb_expected, len(token_group), nb_variants, len(token_group), len(overlap), stemmed_tokens))

what I get:

experte  [as expected: 0/5| variants: 3/5 | overlap: 0]: ['expert', 'expert', 'expert', 'expertin', 'expertinn']
geh      [as expected: 3/6| variants: 4/6 | overlap: 0]: ['geh', 'geh', 'gehst', 'geht', 'geh', 'gehend']
gebäude  [as expected: 0/3| variants: 1/3 | overlap: 0]: ['gebaud', 'gebaud', 'gebaud']
schön    [as expected: 0/3| variants: 1/3 | overlap: 1]: ['schon', 'schon', 'schon']

The two main problems are:

Overlaps: schön != schon
Non-working stemming, e.g. [experte, expertin, expertinnen], [ich gehe, du gehst, er geht]

A not so serious side problem is matching my expectations. So if the stemmer could actually bring the word in a basic form (not simply the stem), then it would be easier to analyze.

More Examples

Clashes

Input -> Output != clash
mittels -> mittel != "Das Mittel"

Unmatched expectations

Input -> Output / expected
Mädchen -> madch / Mädchen
Behaarung -> behaar / Behaarung


texts: Expected Lifetime: Churn Formula vs. Experience Data

texts:
I am analyzing data for a subscription based company. I.e they sell a service in exchange for monthly payment. I would like to conduct an analysis and come up with an estimate of the average lifetime (in months) of a customer. I have approximately 6 years of data including date of enrollment and date of cancellation. N is fairly large 70k, 80k, 100k, 115k, 135k, 161k enrolled clients in the 6 years, respectively.
I have seen articles such as this one that describe how to calculate average lifetime by taking $$\frac{1}{\text{churn rate}} = \frac{1}{\text{1-retention rate}}.$$
I don't entirely understand why this formula holds,but my understanding is that this is a problistic calculation. Someone on my team calculated an average lifetime of 71 months using this methodology.
Out of curiosity, I wanted to compare this to our experience, so I computed the following in R
data %>%
  filter(Cancelled == "TRUE") %>%  #--ignore active policies
  mutate(duration = 12*(as.yearmon(CancelRequestDate) - as.yearmon(EnrollDate))) %>%
  pull(duration) %>%
  mean()

> dataAvgPetMo
  24.52419

These two numbers are very different. Could anyone enlighten me as to why, and perhaps offer some guidance for how to interpret or refine this study to come up with a reasonable estimate of average lifetime.

new to this community. edits and advice welcome :)


texts: Python: convert variables into correct format for DataFrame

texts:
I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!!
The 3 variables I have are:
print(newspaper)

['Bolero']
['Schweizer Illustrierte Style']
['Bolero']
print(title)

['Schönheit und Tragik']
['magie pur']
['Das sind unsere Favoriten']
print(pubDate)

['2007-01-01']
['2007-01-01']
['2007-01-01']
It seems to like all variables are a list of lists, but I'm not quite sure. However, since the data is scrapped from a private website, I can't post the entire code here, but I hope this is already enough for you to access what the problem is with that variable format.
What I would like to have is a dataset of this format:

Newspaper
Title
PubDate

Bolero
Schönheit und Tragik
2007-01-01

Schweizer Illustrierte Style
magie pur
2007-01-01

Bolero.
Das sind unsere Favoriten
2007-01-01


texts: How to get sensitivity and specificity for multi-class classification for each fold of cross validation?

texts:
I am working on a multi-class classification consisting of 4 classes.
I am applying 5-fold cross-validation on it and would like to get the sensitivity (recall) and specificity score for each of those folds.

I found out that using cross_validate function, I can provide it with a list of scoring parameters for each fold.

    scoring = {'accuracy' : make_scorer(accuracy),
               'precision' : make_scorer(precision_score),
               'recall' : make_scorer(recall_score),
               'f1_score' : make_scorer(f1_score)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

However, this generates an error because these functions (except accuracy) are only for binary classification and not for multi-class.

So, I decided to make my own functions for sensitivity score and specificity score that returns the mean of the 4 individual values (1 for each class). I return the mean of them, and not the individual 4 values because scorer functions that return multiple values are not permitted. That is fine with me though as I want their mean only.

Here's what I tried:

    def sensitivity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Sensitivity = TP/(TP+FN)
        return np.mean(Sensitivity)

    def specificity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Specificity = TN/(TN+FP)
        return np.mean(Specificity)

    scoring = {'sensitivity' : make_scorer(sensitivity),
               'specificity' : make_scorer(specificity)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

But it still is throwing the same error:

ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets

I don't know what's not working here. I just want the mean of sensitivity for each class and mean of specificity for each class, for each of the 5 folds.

What is wrong with my approach and also is there a simpler way to do this ?


texts: Extract imperative sentences from a document(English) using NLP in python

texts:
I am very new to NLP, hence require some help on extracting imperative sentences from a document. I am working on a project where I need to get all the imperative sentences from the entire document(English documents). I understand I need to use POS tagging. But how do I proceed further.

Thanks.


texts: Should I use harmonic mean for averaging metrics in repeat runs of classifier evaluation?

texts:
I am testing an LSA classifier with a randomised set of test-cases, picked from a larger set using under-sampling (to balance the classes). I am performing parametric sweeps to examine the behaviour of the classifier under different hyperparameters. I noticed early on that the results have a margin of error, making the performance fine-tuning difficult. So I implemented an averaging technique which simply runs the same parametric scenario N times with different random test-cases and averages the results - using arithmetic mean. (I believe this qualifies as cross-validation.)

This works well and certainly smooths out the performance charts, so I can see the parameter change effects more clearly.

However, I recently learned about the harmonic mean, and now I suspect I should be using it for this purpose - because I am averaging a rate?

I think it probably doesn't matter for looking at one parameter sweep chart (it's relative to itself), but it probably will matter when I come to compare different classifiers (absolute comparison of scores).

Note: I am not talking about using harmonic mean to produce F1 scores from TPR, FPR etc. I am only talking about using it to average these performance metrics over multiple repeats of the entire test process.

e.g.

These charts show "confusion matrix" scores when sweeping a param.

The first chart shows a single run. It looks like there might be a performance sweet-spot at k=70 but I'm pretty sure this is just noise due to the random selection of the test-cases (and the limited number I have).

This is the same test run 10 times on different test-cases. (You can see the apparent minimum has gone, the results are smoother and even the ACd trend line is flat. This shows this range of k param isn't affecting performance.)

This is averaged by arithmetic mean, I'm wondering if harmonic mean is more correct.


texts: Why is it hard to grant efficiency while using libraries?

texts:
Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend to reach out for C/C++/low-level languages. The possibility of tailoring the code to the needs seems to be what makes these languages so appealing for BigData -- be it concerning memory management, parallelism, disk access, or even low-level optimizations (via assembly constructs at C/C++ level).

Of course such set of benefits would not come without a cost: writing the code, and sometimes even reinventing the wheel, can be quite expensive/tiresome. Although there are lots of libraries available, people are inclined to write the code by themselves whenever they need to grant performance. What disables performance assertions from using libraries while processing large databases?

For example, consider an entreprise that continuously crawls webpages and parses the data collected. For each sliding-window, different data mining algorithms are run upon the data extracted. Why would the developers ditch off using available libraries/frameworks (be it for crawling, text processing, and data mining)? Using stuff already implemented would not only ease the burden of coding the whole process, but also would save a lot of time.

In a single shot:

what makes writing the code by oneself a guarantee of performance?
why is it risky to rely on a frameworks/libraries when you must assure high performance?


texts: What is the difference between "equivariant to translation" and "invariant to translation"

texts:
I'm having trouble understanding the difference between equivariant to translation and invariant to translation.

In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio), one can find on the convolutional networks:

[...] the particular form of parameter sharing causes the layer to have a property called equivariance to translation
[...] pooling helps to make the representation become approximately invariant to small translations of the input

Is there any difference between them or are the terms interchangeably used?


texts:
Is there any example of how Keras Dense layer handles 3D input.
The documentation explains the following:
If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot).
But I could not understand the internal matrix calculation
For example:
import tensorflow as tf
from tensorflow.keras.layers import Dense
sample_3d_input = tf.constant(tf.random.normal(shape=(4,3,2)))
dense_layer  = Dense(5)
op = dense_layer(sample_3d_input)

based on the documentation for a 3D input of shape (m,d0,d1), the shape of Layer's weight_matrix (or) kernel will have the shape (d1,units) which is (2,5) in this case. But I dont understand how the op is calculated to have the shape (m,d0,units)


texts: Propensity Modeling, still use Test/Train Split?

texts:
I'm using Sklearn to build a classifier in which my client wants a predicted probability for each row of data.  The default of let's say Random Forest is if > 50%, then classifies as TRUE, but using the predict_proba function I'm able to get the probability.

The data I'm given has 10k rows, which all are labeled TRUE or FALSE.  If it's my job to provide a predicted probability by row, should I still use a 70/30 Train/Test split.  In which I create my best model on the Train 70% using a 10-fold CV.  But then when I need to output for all 10k rows, I would actually be make a prediction on the training data and the test data, which doesn't seem correct to predict on training data.  Any thoughts on how to approach this?


texts: How to apply supervised machine learning when the target label depends on multiple input rows?

texts:
The problem is a multi-label classification problem.

Now, I know how to train and classify using single row with several attributes. For example, if the dataset looks like the first table from the attached file. Here, each row is associated with a single label. Thus, I can train and test after separating the dataset into training and testing sets.
But the problem occurs when classification label / target label depends on multiple rows such as the second table from the attached file. Consecutive N rows makes one category.
Can you please guide me towards a solution?

Is it possible to fit this problem in any existing tool? For example, WEKA or Neural network using Keras.
Or do I have to change the algorithm in order to fit the problem! Is there any existing solution?
Or do I need to modify the rows in such a way that it transforms into one?


texts: what to do if the missing data in one column is based on some value/condition in another column in r?

texts:
I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever there is a 'U' in the gender column, there is an NA in two of the other columns namely Age and Tenure. This could basically be interpreted as a person who is not ready to disclose their Gender is not ready to disclose their age and tenure. How do I deal with such a situation? Apart from these three columns there are other 16 columns in the dataset that have got meaningful data in them. Would the normal imputation techniques out there like a KNN Imputation help me out in such a case?

Here is my reproducible example that I have tried my best with:

x<-data.frame(gender=c('M','M','F','F','U','F','M','U'),age=c(21,24,20,34,NA,40,56,NA),tenure=c(7,4,5,3,NA,2,4,NA),job=c('Doctor','IT','Banking','Truck Driver','Finance','Agriculture','Electrician','Teacher'),country=c('Australia','America','New Zealand','Sweden','England','France','Denmark','Norway'))

The Dataframe:

 gender age  tenure      job      country
1      M   21     7       Doctor    Australia
2      M   24     4        IT       America
3      F   20     5      Banking    New Zealand
4      F   34     3    Truck Driver Sweden
5      U   NA     NA      Finance    England
6      F   40      2    Agriculture  France
7      M   56      4    Electrician  Denmark
8      U   NA     NA      Teacher    Norway

As you can see from the example above whenever the gender is undefined, there are missing values in both age and tenure and this is the case overall in the entire dataset. What would be the best way to deal with such a situation? And this what is called a Missing at Random data, is that right? Any suggestions would be extremely helpful. Thanks a lot.


texts: Forecasting vs non-forecasting predition for time series anomaly detection

texts:
I have got the objective of implementing a uni/multivariate online anomaly detection system.

After multiple days of research, I could collect many ways to achieve this (Eg. moving average solutions such as ARIMA, Space state solutions as Kalman filters, Holt-Winters double/triple exponential smoothing, CUSUM, one-class SVM, deep learning sliding-windows autoencoding approaches, deep learning using autoregressive neural networks, etc).

In general, anomaly detection on time series works with a threshold on the deviation originated from the difference between a predicted point or group of points of the original timeseries and the predicted one.

Attending to this prediction, this can happen in:

a forecasting way (such as ARIMA would do, or you could achieve this result also by using a LSTM deep learning model),
or in a non-forecasting way (eg. denoising with an autoencoder would do, or analyzing fragments STL+ESD used by Twitter).

Which are the (dis)advantages of each one, attending to the objective I mentioned?


texts: How to determine window size for modeling user behavior

texts:
I am working in mining time series data for modeling user behavior. I collected the data from 37 smartphone users during 42 days(6 weeks). I want to use data during 3 weeks as model and the remaining as test (test performance of my model). The visualization of our data is following below:

In here, I want to ask what is the good window size for modeling user behavior, is one hour, one day, one week etc? or may be is there any good methods for determining window size?


texts: Imputing missing values by mean by id column in R

texts:
This is fairly straight forward but i am unable to do it. My data frame has a id variable which is repeating. For same id I want to replace the NAs in other continuous variable(rating and sur) with their corresponding mean. can anyone pls suggest

ID rating Sur
101 60 0.7687
101 78 NA
101 NA 0.765
102 60 NA
102 NA 0.654
102 75 0.435
103 NA 0.576
103 68 0.875
103 70 NA


texts: How does the supposed "Unified Architecture for NLP" from Collobert and Weston 2008 really works?

texts:
In this paper (here)  they suppose a "unified architecture for NLP" with deep neural networks with multitask learning
My problem is to understand the layered architecture in figure 1, see below:

Is someone able to give me a concrete, reproducible example of how this architecture processing 3 sentences through their layers?
What are the outputs after each layer?
Why they choose which layer?


texts: Suppose i want to have 80% training data and 20% testing data. How do i choose which 80% of the data to use for training? Should it be completely random? Like what if there is a class label with 2 clusters and the 80% training data we randomly choose only captures one of the clusters?


texts: How to calculate irreducible error using Bias and Variance for a given machine learning Model?

texts:
I am trying to calculate the Bias and Variance for a ML Model.

$$
Err(x)=E[(Y−\hat f(x))^2]
\\Err(x)=Bias^2+Variance+Irreducible\ Error
$$
$\hat f(x)$ is our model

$Y$ is the variable we are trying to predict

$Err(x)$ is the overall error (MSE).

I am using the mlxtend library for bias variance decomposition.

Steps I followed:

Generate training data set using the function $Y = f(x) + \epsilon$

$f(x) = a + bx + cx^2$

$\epsilon ∼N(0,σ^2) .$ is the normally distributed noise with mean $0$ and variance $\sigma^2$
Generate test data set using $f(x) = a + bx + cx^2$. Here I create X_test and y_test. y_test contains the true value (without noise), as Bias is calculated using the true function.
Use the mlxtend library function to calculate bias and variance. Here I am passing the Linear Regression estimator to the function.

My problem is even though the formula for MSE here is $Err(x)=Bias^2+Variance+Irreducible\ Error$ and I have also read that if our model is trained on a data which contain noise than it's impossible to eliminate that nose from the estimator. Still, upon decomposition I get $Irreducible\ Error = 0$.
Even though I am using the true function ($f(x)$) for calculating the $Bias$ still $Irreducible\ Error$ is $0$.

What am I doing wrong?

According to my understanding if I calculate $Err(x)$, $Bias^2$ and $Variance$ I should be able to get the $Irreducible\ Error$ from the above equation.


texts: How to decide which images to label next?

texts:
We have a custom dataset of 20 thousand images with two pixel-wise labeled classes.
However we have 1 million more raw images, which we would like to label.

We want to label the most important new images first. Importance is defined as:

Images with the more new information
Images helping our deep learning bounding box classifier to improve

So instead of labeling images, where we already have about thousand similar ones, we first want to label the images which are quite different from the already-labeled ones and help more to improve our classifier.

How can we assign priorities and decide which images to label first?


texts:
I have a dataset in which each feature is either 0 or 1 (like BBOW). I want to cluster the data such that one point can belong to more than one cluster(soft assignment).
I searched about this and I found that fuzzy k-modes can be applied for this problem. Since I am new to ML coding, Is there any implementation available online for fuzzy k-modes or any other similar algorithm?


texts: Is it possible to identify different queries/questions in sentence?

texts:
I want to identifies different queries in sentences.

Like - Who is Bill Gates and where he was born? or Who is Bill Gates, where he was born? contains two queries

Who is Bill Gates?
Where Bill Gates was born

I worked on Coreference resolution, so I can identify that he points to Bill Gates so resolved sentence is "Who is Bill Gates, where Bill Gates was born"

Like wise

MGandhi is good guys, Where he was born?
single query
who is MGandhi and where was he born?
2 queries
who is MGandhi, where he was born and died?
3 quries
India won world cup against Australia, when?
1 query (when India won WC against Auz)

I can perform Coreference resolution (Identifying and converting he to Gandhi) but not getting how can I distinguish queries in it.

How to do this?

I checked various sentence parser, but as this is pure nlp stuff, sentence parser does not identify it.

I tried to find "Sentence disambiguation" like "word sense disambiguation", but nothing exist like that.

Any help or suggestion would be much appreciable.


texts: Starting my career as Data Scientist, is Software Engineering experience required?

texts:
I am an MSc student at the University of Edinburgh, specialized in machine learning and natural language processing. I had some practical courses focused on data mining, and others dealing with machine learning, bayesian statistics and graphical models. My background is a BSc in Computer Science.

I did some software engineering and I learnt the basic concepts, such as design patterns, but I have never been involved in a large software development project. However, I had a data mining project in my MSc. My question is, if I want to go for a career as Data Scientist, should I apply for a graduate data scientist position first, or should I get a position as graduate software engineer first, maybe something related to data science, such as big data infrastructure or machine learning software development?

My concern is that I might need good software engineering skills for data science, and I am not sure if these can be obtained by working as a graduate data scientist directly.

Moreover, at the moment I like Data Mining, but what if I want to change my career to software engineering in the future? It might be difficult if I specialised so much in data science.

I have not been employed yet, so my knowledge is still limited. Any clarification or advice are welcome, as I am about to finish my MSc and I want to start applying for graduate positions in early October.


texts: How to extract keywords from a list of URLs?

texts:
I have a bunch of URLs in a text file like-

https://www.mycustomer.com/marketing/technology/how-marketers-are-over-simplifying-b2b-buyer-behaviour
https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts
https://www.forbes.com/sites/forbesagencycouncil/2017/02/03/video-marketing-the-future-of-content-marketing
https://www.searchenginejournal.com/seo-content-marketing-strategy/258253
...

What's the best way to go about extracting top 10 keywords from just these URLs (no parsing the webpages)?

I'm aware of TF-IDF but that usually requires a title and a body, can I still use it here? Or are there any other approaches (e.g. TextRank) that would work better here?


texts: Training data set for food image recognition

texts: There are many excellent posts and answers referencing data sources. However I can't seem to find many for food image/photograph recognition. I would have thought, with so many people taking and uploading pictures of their food, that this should be readily available. Can anyone help? Food101 and UEC are good but limited. Thank you.


texts: Horizontal Bar Char Issues in Tableau with Excel Spreadsheet Beginner Visualization Question

texts:
Hello i did a good graphic vertical chart with a dark theme.
but now i want to do it in horizontal chart and i have some issues.
i had used too much drag and drop and move option in Tableau maybe this is my beginner problem.
I only need join the data in a unique line visual
I did this:

I want to do this:

I created a calculated field like some 2nd explanation here but still nothing...


texts: Q-Learning: Target Network vs Double DQN

texts:
I am having a hard time understanding difference between Target Network and Double DQN
From this blog:
Target Network generates the target-Q values that will be used to compute the loss for every action during training. The target network’s weights are fixed, and are frequently but by small amounts updated towards the primary Q-networks values.
Double DQN: instead of taking the max over Q-values when computing the target-Q value for our training step, we use our primary network to chose an action, and our target network to generate the target Q-value for that action
That sounds very similar to me, and the equation makes me lost too :(
qTarget = r + γQ(  s’,  argmax(Q(s’,a,ϴ),ϴ’)  )

From whaat I know, Target Network Approach allows us to do the
following:

using Online Network, get scores for each action as the response for the current state. Pick the most appropriate action $a$ and take a
note of its score $q$
using that action, "travel" to the next state and get scores from Target Net for even further actions.
Select the most appropriate action from step 2) and call it $A$. also, take a note at its score $Q$
Punish OnlineNetwork for any difference between $q$ and $Q$.

Note: the gradient flows only through $a$ component of the "output
vector" $\{a, b, c, d ...\}$

Please correct me if these steps are wrong


texts: What is required in Affinity Propagation

texts:
I am hoping to use affinity propagation to cluster my data using sklearn. But I came across a question whether to use a distance matrix or similarity matrix in the fit method.

Please let me know what is suitable to use?


texts: How do GBM algorithms handle missing data?

texts:
How do algorithms GBM algorithms, such as XGBoost or LightGBM handle NaN values?
I know that they learn how to replace NaN values with other values but my question is: How do they do it exactly?


texts:
I am working on a CNN to detect body parts (Right Shoulder, Left Shoulder), something more like Facial Keypoints Detection. I tried solutions from Kaggle as I think it's almost the same problem but it did not work very well.
More details:

I collected and labeled my own data,
I am using generators to augment the images by only resizing the image and place it in 90x30px frame.
I normalized the input images and the output arrays to get values between 0 and 1.
I tried MSE loss and a RMSE with different CNN architectures.

Full notebook available here
[Note] I have bigger dataset than the one shown in notebook, it's 10K samples
so could it be the data? the architecture? or simply the optimization setup?


texts: Learning the uncertainty of a ML algorithm

texts:
I have a regression GAM (General Additive Model) and I want to learn its epistemic uncertainty( the variance of my residuals or predictions as a function of my input).
I have already used a bayesian approach to turn my GAM into a gaussian process so I can construct a covariance matrix but this approach is not scalable due to the high dimension of my problem.
I am trying to use an approach that uses the current model as a black-box and observe only the input and the residual, the closest thing that I found is quantile regression but I was wondering if there is any deep learning approach that learn the variance from the input.
Most of deep learning approach that I found estimate the mean and the variance simultaneously (Deep bayes, MVE, MC dropout ...)
A naive approach that I am implementing currently is a neural network that learn the variance as function of my input by minimizing the likelihood of my residuals as a centred gaussian but I didn't find any paper or ressources on this approach.
Do you have any idea on the problem, any possible ressources or an opinion on my current approach ?
Thanks


texts: Intuition behind Occam's Learner Algorithm using VC-Dimension

texts:
So I'm learning about Occam's Learning algorithm and PAC-Learning where for a given hypothesis space $H$, if we want to have a model/hypothesis $h$ that has an True error of $error_D \leq \epsilon$, with a probability of $(1-\delta)$ for a given probability $\delta$, we need to train it on $m$ examples with $m$ being defined as:

$$ m > \frac{1}{2\epsilon^2}\{\log(|H|)+log(\frac{1}{\delta})\}$$

Now, I'm looking for some way to explain the terms of the equation in very simple terms to gain some intuition into why this equation is the way it is, without reverting to a complicated mathematical proof. And so, while being introduced to the concept of VC-dimensionality which is, as I understand it, a measure of how complicated the example space $X$ can be while still being linearly separable within $H$.

Substituting $\log(H)$ for $ VC(H)$ (and adding a few constants) I found the following equation:

$$ m > \frac{1}{\epsilon}\{8VC(H)\log(\frac{13}{\epsilon})+4\log\frac{2}{\delta}\}$$

Which then made sense to me because basically if the size of $H$,i.e. the dimensionality $|H|$, then, to shatter $m$ examples we must have that: $|H|>2^m$, or in other words $log(|H|) \geq VC(H)$.

So far I've just been restating things I've learned online in my own words.

Now, if you will allow me to try to very hand-wavingly reprove the first equation above, here is my restatement of Occam's Learning Algorithm in simple terms:

In order to have an algorithm that performs perfectly over hypothesis space $H$, we must train it on at least every possible partition of $H$, which is defined as: $2^m$ where $2^m > |H|$

Question 1: Are all my assumptions so far correct?

Question 2: Does this mean my intuition is correct?

If so, then the rest is just algebraic manipulation where we first add some probability that $|H|$ is not expressive enough by some constant, we call this $\delta$:

$$ 2^m > \frac{|H|}\delta $$

Isolate $m$, split up the $\log$ terms using logarithmic identities:

$$ m > \log(|H|) + \log(\frac{1}{\delta}) $$

Adding some margin of error $\epsilon$ to the size of $H$, square it, and add a few constants, we get:

$$ m > \frac{1}{2\epsilon^2} \{\log(|H|) + \log(\frac{1}{\delta}) \} $$

Question 3: Did I just understand this in simple terms or am I way off?


texts: Interpretation of tuning parameters (shrinkage and nrounds) in XGBoost

texts:
I'm using XGBoost for a multiclass classification problem in R. I'm trying different combinations of the shrinkage parameter and number of iterations to try and settle on optimal values for these parameters (using cross-validation). However, I'm having trouble with exactly interpreting the significance of the parameters. The XGBoost algorithm isn't completely a black box for me - I know how boosting works at a high-level.

The questions I have though are as follows:

Does a low shrinkage (e.g. 0.01) and high nrounds (e.g. 500) yield the same results as a high shrinkage (e.g. 0.3) and low nrounds (e.g. 50)?
Does nrounds correspond to the number of weak classifiers being fit?
The xgb.cv function in R returns test mean errors and standard deviations for each of the iterations. For the nth iteration, is this the mean over all iterations less than equal to n over all the folds? Apologies if I'm interpreting this incorrectly. Also, what is the significance of the standard deviation here?
Is there some blog post/link I can refer to for a comprehensive guide on tuning XGBoost parameters?


texts: Problem of finding best combination of features when desired feature is feature some_feature_A/some_feature_B

texts: Problem is stated: we have giant csv file with one target column and rest are inputs, we don't know these features impact target but we would like to use algorithm that besides using linear and non-linear transformations will also take into account that right solution would be some_feature_A/some_feature_B. Is there algorithm that will allow to take this case into account? One way is to craft these feature columns yourself but is there better way?


texts:
I want to perform clustering on a dataset which has a few thousand stock prices time series, each one with daily prices for previous 5 years (a couple thousand days).
What would be the best approach for this?


texts: Keras: Prediction performance does not match accuracy

texts:
I am using Keras/CNN to identify plankton images collected with an in situ camera.  When making confusion matrices on the test sets following training I am finding that the accuracy from the predictions is quite poor.

I have a large number of files and have been using flow_from_directory and generators, I suspect that there might be something happening with the indexing of the predictions (e.g. this post), but as near as I can tell the indexing of the filenames/labels is matching up.

I worked up a quick example that is similar to what I am doing with the mnist_png dataset:

import numpy as np
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.layers.advanced_activations import LeakyReLU
from keras import backend as K
from keras.preprocessing.image import ImageDataGenerator

img_width, img_height = 28, 28

train_data_dir = 'S:/mnist_png/training'

num_epochs = 100
batch_size = 128
num_test_samples=10000

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()

model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))

model.add(Conv2D(16, (3, 3)))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.5))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.5))

model.add(Flatten())
model.add(Dense(512,activation='linear'))
model.add(LeakyReLU(alpha=.3))
model.add(Dropout(0.5))

model.add(Dense(512,activation='linear'))
model.add(LeakyReLU(alpha=.3))

model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='Adam',
              metrics=['accuracy'])

# augmentation for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip=True,
    rotation_range=90,
    validation_split=0.1)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical')

model.fit_generator(
    train_generator,
    steps_per_epoch=batch_size,
    epochs=num_epochs)

...after 100 epochs I'm getting loss: 0.7517 - acc: 0.7482.

I then evaluate the test set thusly:

test_data_dir = 'S:/mnist_png/testing'

test_datagen = ImageDataGenerator(
    rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    test_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    shuffle='False',
    class_mode='categorical')

#Evaluate model on test set
scores = model.evaluate_generator(test_generator,workers=12)

...the scores for that were 0.6184 and 0.8168, so in the same ballpark.

But it gets weird when I look at the predictions, e.g.:

test_generator.reset()# Necessary to force it to start from beginning
Y_pred = model.predict_generator(test_generator)
y_pred = np.argmax(Y_pred, axis=-1)
sum(y_pred==test_generator.classes)/10000

The proportion of predictions that are actually correct (calculated in the last line) is around 0.1; when I look at a confusion matrix it's all over the place and the diagonal is a lot of zeros.
I have verified that the test_generator.classes match up with the directories in test_generator.filenames, and shuffle is off.  Per this post calling test_generator.reset() should force it to take the files in order, but I'm wondering if it is not.

Does anyone have any thoughts on why this is happening or further steps to troubleshoot it?


texts: Xgboost multiple class predictive performance beats one versus rest

texts:
I have an NLP task I'm tackling with xgboost (R implementation).

Before describing my doubt I'll give you some background:

I have a corpus of documents for which I did topic discovery, using a term x term matrix clustering approach. For each document, I get a topic score computed using the terms in the document (with a TfIdf score). Then for each document, I pick up the topic with the highest score.

The following step is to create a model that given the term x document score matrix and the best topic per document, predicts the best topic.

I tried two different approaches:

a multiple class model, where a topic is associated with each document;
a one versus rest series of models, one per topic, where each document is labeled as belonging or not to a topic.

Here are the results of the two approaches, using AUC:

    i                 topic    single     multi
1   1             Topic.nv1 0.9564445 0.9880821
2   2  Topic.nv10_Topic.wv9 0.9848492 0.9969546
3   3            Topic.nv11 0.9174293 0.9741100
4   4 Topic.nv12_Topic.wv11 0.9874073 0.9967725
5   5 Topic.nv13_Topic.wv10 0.9509909 0.9916768
6   6 Topic.nv14_Topic.wv12 0.9864622 0.9959161
7   7            Topic.nv15 0.7333333 0.9333333
8   8   Topic.nv2_Topic.wv3 0.9590279 0.9877953
9   9   Topic.nv3_Topic.wv5 0.9448966 0.9879057
10 10   Topic.nv4_Topic.wv2 0.9521490 0.9908656
11 11   Topic.nv5_Topic.wv6 0.9761665 0.9946294
12 12             Topic.nv6 0.9439377 0.9889028
13 13   Topic.nv7_Topic.wv4 0.9656248 0.9926163
14 14             Topic.nv8 0.9673726 0.9944970
15 15   Topic.nv9_Topic.wv8 0.9716538 0.9929586
16 16             Topic.wv1 0.9610704 0.9925414
17 17             Topic.wv7 0.9765398 0.9904255

It is visible that the multiclass approach systematically outperforms the one vs rest one. NB: These are training set performances.

Is there a clear theoretical reason for this?


texts: Origin of the Boolean Model of Information Retrieval

texts:
Simple question, but I can't really find the answer to that: Who "invented" Boolean Retrieval? Of course, I assume that the concept grew over time, but is there a paper or publication that mentions/defines the Boolean Model as a whole for the first time?
On Wikipedia, the book by Lancaster and Fayen (1973) is cited, but I couldn't find any definition in there, either.


texts: Deep learning facial recognition research project

texts:
I'm a second year pure maths, applied maths and computer science student. I have taken up a research course and was given the topic to focus on facial recognition using deep learning.

I have done a fair deal of reading on the topic (looking at older methods such as eigenfaces and fisherfaces all the way to newer methods such as deepface). I also feel comfortable with the maths and algorithms underlying ANNs.

I was wondering whether someone in the community could maybe point me in the direction in what I (as an undergraduate) could take a deeper look at in the category of facial recognition using deep learning?

I thought it might be interesting to pursue recognising faces given a few images of a face i.e. < 5. Maybe someone has a better suggestion?


texts: Follow up question regarding Upsampling for Imbalanced Data and the use of ADASYN instead of SMOTE

texts:
I have a follow-up question regarding this topic.
I have been working on a project predicting success(1) or failure(0) for organizations by using the Decision Tree and Random Forest algorithms.
My dataset has a minority class of successes which I would like to upsample using SMOTE or ADASYN.
I understand that the reasoning mentioned in this post applies to SMOTE and random upsampling by duplicating but does this also apply to upsampling via ADASYN? As I under ADASYN introduces even more randomness to the synthetic new observations so perhaps the correlation might be lower? In other words, does the use of ADASYN justify upsampling before the split or even upsampling the training and testing data separately?
I have seen a research paper that first applied the train test split and then upsampled the minority class using ADASYN in the testing and training dataset separately. This approach made better sense to me since as compared to upsampling before the train test split which introduces the possibility of leakage from the training to the test data this approach instead removes that possibility of leakage by separately upsampling the training and testing dataset. I have heard that this approach is also not fully correct since the testing dataset is supposed to replicate the real world and hence we are not supposed to change it in any way.
On the other hand instead of upsampling the minority class in the test dataset, I can even downsample the majority class which might be a better approach since the testing dataset remains still has only observations from the real world. Here we just gave the algorithm a fair chance (50:50) to pick between each class (1 or 0). Although once again the real world most likely will not have the 1s and 0s in equal proportions.
Additionally, some places also suggest doing the train test split proportionally so the training and testing datasets have the minority and majority class in equal proportions. As I understand this can be done using stratify=y when running the code. Please let me know if I need to do this and why?


texts: Extracting words belonging to a key from the text

texts:
I have a description of a product say mobile in this form -

"5.5" (13.97cm) full hd 1920 x 1080 pixels ips screen gorilla glass display with 401ppi resolution - 450 nits brightness & contrast ratio of 1000:1first phone with theatermax technology and dolby atmos surround sound over speakers13mp f2.2 primary camera with 5p lens element, dual led dual tone flash, 5mp front camera with 4p lens elementandroid v5.1 os with 1.3 ghz mediatek 6753 64-bit octa core processor, arm mali t720 gpu, 3 gb ddr3 ram, 16gb internal memory (expandable up to 128gb) and dual micro sim dual standby (4g+4g)3300mah lithium-ion battery with fast charger 2.0 providing talk-time of 29 hours"

I have to extract the important elements of the description and map it to keys like -

Display size - 5.5
Display Resolution - 1920 x 1080
Sim type - Dual Sim
Ram - 3gb
Storage - 16gb.

I have a dictionary that maps key values pairs.

Example - {Ram:["3gb","3 GB","4 gb"],"Sim type":["Dual","Micro","Nano"]}

Can anyone suggest how to do this. I am coding in python. How to proceed it with NLTK. Should I use ngrams??? Any useful examples that uses self tagging will be helpful. All examples i see uses already defined tags like noun,person,organisation etc. I want to give my own dictionary as corpus and train the model so that it extracts the relevant key values pairs from the text.


texts: calibrating classifier probabilities for unbalanced data when class ratios are unknown

texts:
I've built a binary classification convolutional neutral network, trained on simulated data with equal numbers of simulations for each class. I've obtained good results for a validation set with equal classes and am using beta regression for calibrating the output probabilities [1]. The classifier will now be applied to an empirical dataset, where the classes are likely very unbalanced. If I knew the true class proportions in the empirical dataset, my approach would be to fit the calibration regression to simulations with the true class proportions (probably resampled from the training set, to avoid the burden of additional simulations). But the relative proportions of each class in the empirical dataset is unknown.
I also recognise that my validation dataset should match the class ratios in the empirical dataset. Based on knowledge of the empirical dataset, I suspect the class ratios will be something like 100:1 (or perhaps even more skewed). However, I would also like to apply the same CNN architecture (trained on different simulations) to other empirical datasets, where I have no knowledge of the ratios, other than that they will be highly skewed.
So I'm interested in general strategies for dealing with this problem. References to relevant papers would be greatly appreciated. My searches on this topic so far have mostly uncovered blog posts about how to learn from imbalanced datasets, which are usually quite introductory and not relevant for me anyway, as I'm training from simulations such that the classes can be chosen to have arbitrary proportions.
[1] http://proceedings.mlr.press/v54/kull17a.html


texts: How to draw neural network diagrams with this particular style?

texts: I would like to draw a neural network architecture with the follow style. Do you know which tool can be used to do this? The paper is Operation-aware Neural Networks for User Response Prediction.


texts: Is there an advantage to use decision trees in OneVsAll than using "classical multiclass trees"?

texts:
I wonder if the is cases where it's better to use OneVsAll decision trees for multiclass classification ?
I think that maybe it could be better for explainability of the model, but I didn't see anybody saying that.


texts: Feature selection for Support Vector Machines

texts:
My question is three-fold

In the context of "Kernelized" support vector machines

Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and the main motive behind introducing kernels to a SVM is to increase the dimensionality of the problem, in such a case reducing the dimensions by parameter reduction seems counter-intuitive
If the answer to the 1st question is "NO", then, On what conditions would the answer change that one should keep in mind ?
Are there any good methods that have been tried to bring about feature reduction for SVMs in scikit-learn library of python - I have tried the SelectFpr method and am looking for people with experiences with different methods.


texts: What is the difference between BERT architecture and vanilla Transformer architecture

texts: I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional encoder, So, I think this is what made BERT different from the vanilla Transformer model. But as far as I know, the Transformer reads the entire sequence of words at once, therefore it is considered bidirectional too. Can someone point out what I'm missing?


texts:
I am running the following code:
import pandas as pd
import matplotlib.pyplot as plt
from pandas import read_csv

df = read_csv('demo.csv')
df
plt.hist(df.values)
plt.show()

lower_bound = 0.1
upper_bound = 0.95
res = df.Values.quantile([lower_bound,upper_bound])
res

and then I get the following error:
AttributeError: 'DataFrame' object has no attribute 'Values'


texts: I created a XGBRegressor model with certain encoded 'object' dtypes in the data. Now if I want to run the model with new set of data which is freshly encoded it's giving wrong predictions. How to ensure that the new dataset is encoded in the same  way as was the train data? Or any other solution to this problem?


texts:
I have two input arrays that include both historical and forecasted data, and one input array that is only historical. I'm trying to predict (or "forecast") the latter array given the forecasts of the first two arrays.
Inputs:

Rainfall (historical and forecast)
Air temp (historical and forecast)
Water volume (historical)

Training target:

Water volume (historical)

Desired model output:

Water volume (forecast)

I'm still a bit of a noob, and lost as to how this would work. None of the examples I've found take the target's previous state as input. The crux of the problem is that I could do a vanilla multivariate LSTM with only rainfall and air temp as inputs, but it would not take into account the starting condition of water in the container.


texts:
I'm a little bit lost about the rescale parameter in the ImageDatagenerator function.
I know that the rescale argument by itself does not augment my data and that by doing rescale=1./255 it will convert the pixels in range [0,255] to range [0,1].
Currently, I'm only using random flips(vertical and horizontal) and 90º degrees rotations but I'm pondering whether or not I should add the rescale parameter.
My question is what is the advantage in doing that?


texts: Which decision tree algorithm does H2O use?

texts: Does H2O's plain random forest use CART, C4.5, 5.0, or something else? I cannot find this information. sklearn's docs say they use a modified version of CART, and I assume H2O also uses something like CART.


texts: I am have a data set with 52 variables. Most of them have zeros, it resembles a sparse matrix. How can I cluster this kind of data and are there any special types of clustering? I am attaching pca plot


texts: Gradients for bias terms in backpropagation

texts:
I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation  when we take derivative with respect to bias) and I derived all the equations used in backpropagation. Now every equation is matching with the code for neural network except for that the derivative with respect to biases.

z1=x.dot(theta1)+b1

h1=1/(1+np.exp(-z1))
z2=h1.dot(theta2)+b2
h2=1/(1+np.exp(-z2))

dh2=h2-y
#back prop

dz2=dh2*(1-dh2)
H1=np.transpose(h1)
dw2=np.dot(H1,dz2)
db2=np.sum(dz2,axis=0,keepdims=True)

I looked up online for the code, and i want to know
why do we add up the matrix and then the scalar db2=np.sum(dz2,axis=0,keepdims=True) is subtracted from the original bias, why not the matrix as a whole is subtracted. Can anyone help me to give some intuion behind it. If i take partial derivative of loss with respect to bias it will give me upper gradient only which is dz2 because  z2=h1.dot(theta2)+b2 h1 and theta will be 0 and b2 will be 1. So the upper term will be left.

b2+=-alpha*db2


texts: Workaround / fallback value for tfp.distributions.Categorical.log_prob in tensorflow graph mode

texts:
Is there a way to avoid tfp.distributions.Categorical.log_probraising an error if the input is a label out of range?
I am passing a batch of samples to the log_prob method, some of them have the value n_categories + 1, which is what you get as fallback value when you sample from a probability distribution off all zeros. Some of the probability distributions in my probs batch are all zeros**.
dec_output, h_state, c_state = self.decoder(dec_inp, [h_state, c_state])
probs = self.attention(enc_output, dec_output, pointer_mask, len_mask)
distr = tfp.distributions.Categorical(probs=probs)
pointer = distr.sample()
log_prob = distr.log_prob(pointer) # log of the probability of choosing that action

I don't care what value I get from log_probin those cases because later I will mask it and not use it. Not sure if a fallback value can be implemented somehow. If not, is there any workaround to avoid an error to be raised while I execute it in graph mode (with @tf.function)?
**This is because I am doing stochastic decoding with an RNN of batches of sequences of variable length, a seq to seq task.


texts: I am working on a two different architecture based on LSTM model to predict the users next action based on the previous actions. I am wondering, what is the best way to present the result? Is it okay to present only the prediction accuracy? Or Should I use other metrics? I found a paper using top_K_accuracy where on a different paper I found AUC of ROC. Overall, I want to know what is the state of art of presenting prediction accuracy based on LSTM model.


texts:
My goals is to create a classifier from list of N vectors as input.
My vectors came from embedding model of image.
Is There paper which propose method to create classifier from embedding vectors ?
Edit:
As  Shubham Panchal  advices me. I try to process the data as 1D vector.
Each embedding from image generates me a 1280 float vector.
I decided to take 10 images took at consecutive times. Here is how I capture images. In my example, I want to classify blinking leds from a device. So I use opencv to diff images and generates me 10 images when differences between frames are important. I checked in my dataset, the result is ok. For each blinking type, the goog image is extracted.
Then for each image of my behavior, I use a VGG-16 to generate a 1280 embedding vector, so I get a 10x1280 array for each behavior. I find the max and min value of the dataset and I normalize it.
Finally, I create a classifier to classify the behavior. I get poor result, lower than 0.4 in accuracy on my train dataset and my test dataset.
Here is the models, I used
def create_model1D(input_shape, output_label_size):
    model = Sequential()
    model.add(Conv1D(16, 5 , activation='relu', input_shape=input_shape))
    model.add(Dropout(0.1))

    model.add(Conv1D(32, 5 , activation='relu', input_shape=input_shape))
    model.add(Dropout(0.1))

    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))

    model.add(Dense(6, activation='softmax'))

    model.compile(optimizer=Adam(learning_rate=0.000001), loss = "sparse_categorical_crossentropy", metrics= ['accuracy'])
    model.summary()
    # plot_model(model, to_file='model.png')

    return model

def create_model2D(input_shape, output_label_size):
    model = Sequential()
    model.add(Conv2D(16, (2, 2), activation='relu', input_shape=input_shape))
    model.add(MaxPool2D())
    model.add(Dropout(0.1))

    model.add(Conv2D(32, (2, 2), activation='relu'))
    model.add(MaxPool2D())
    model.add(Dropout(0.2))

    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))

    model.add(Dense(6, activation='softmax'))

    model.compile(optimizer=Adam(learning_rate=0.000001), loss = "sparse_categorical_crossentropy", metrics= ['accuracy'])

    model.summary()
    # plot_model(model, to_file='model.png')

    return model


texts: How to filter in pandas based on string value?

texts:
I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format:
 ID | A | B

And the id values looks like TCGA-A6-6653-01Z-00-DX1.e130666d-2681-4382-9e7a-4a4d27cb77a4 but multiple repeating ones. I want to group the unique ids and do a mean operation on A and B.
Can anyone guide me?   Thanks


texts: How to set up multi cluster spark without hadoop on Google Compute engine

texts:
I'm new to apache spark.

Is it possible to configure multi cluster spark without hadoop?

If so, can you please provide the steps.
I would like to create clusters on Google Compute Engine  (1-master, 1-worker)


texts: what is sentence embeding and how to do sentence embedding for a sentence and how to use word embedding to create a sentence embedding?

texts: What is sentence embedding? How would you do sentence embedding for a sentence like: "How old are you?" How do you use word embedding to create a sentence embedding?


texts: In linear regression model, how can we define cost function. also after defining cost function how to minimize the error term?