texts: Debugging a simple 1-D CNN for solving a simple classification problem

texts:
I have a rather simple classification problem that I am trying to solve. Each instance in my problem is a list of 1024 bytes (each byte is represented by a digit between 0 and 255). There are 2 classes (say, Class 'A' and 'B'). In class 'A', all instances have a common feature. There exists a particular 2-byte pattern say: "200 180" that is present in high frequency among all instances in Class 'A'. However, in class 'B', this is not the case. In class 'B', the presence of the byte string '200 180' is more of an artefact of randomness rather than anything else.
I would like my CNN to differentiate between A and B classes without explicitly feeding it the byte pattern. I am using the following code to try to do that:
model = models.Sequential()
model.add(layers.Conv1D(filters=1, kernel_size=2, activation='relu', input_shape=(1024, 1)))
model.add(layers.MaxPooling1D(pool_size=64))
model.add(layers.Flatten())
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
          loss='binary_crossentropy',
          metrics=['accuracy'])

When I use some simple heuristics to gauge the "distinguishability" between the two classes, I can see that it's possible to easily get more than 90% accuracy rates by simply looking for the frequency of the byte pattern. However, my above CNN fails miserably at doing this. For example, here is a 5-epoch training attempt result for my CNN:
Epoch 1/5
1563/1563 [==============================] - 65s 41ms/step - loss: 0.5297 - accuracy: 0.8468
Epoch 2/5
1563/1563 [==============================] - 61s 39ms/step - loss: 0.7534 - accuracy: 0.5000
Epoch 3/5
1563/1563 [==============================] - 61s 39ms/step - loss: 0.7100 - accuracy: 0.5029
Epoch 4/5
1563/1563 [==============================] - 61s 39ms/step - loss: 0.6977 - accuracy: 0.5216
Epoch 5/5
1563/1563 [==============================] - 62s 40ms/step - loss: 0.6934 - accuracy: 0.5303

As you can see, the result is more like a random guess than anything else. I wanted to know if you have any advice on how you think this task can be handled better.


texts:
I am trying to pick a technique for classifying conversational text.  I am concerned about treating the problem at a level of fidelity of each individual message because people often say things like, "ok" or short responses that have no inferable meaning.   How does conversation classification typical handle these types of problems?

To elaborate, a conversation might be:

P1: Hi I want to buy a car?
P2: Ok. Great!
P1: What cars do you have?
P2: A large variety!

The topic is cars, but this can not be inferred by anything P2 says, nor should it be.  So would you break a conversation into blocks of time, or is there a technique for partitioning?


texts:
I have the following time serie data and I need to detect the following pattern:
- a 1st spike above a given value
- a 2nd spike close to the given value

What defines "a spike" is that its value is x% above the surrounding spikes.

I would like to use python (numpy and pandas modules) to detect these.

As a background, the goal is to trigger an action when the second spike is detected (outlined in red in the attached image).


texts: Use Machine Learning/Artificial Intelligence to predict next number (n+1) in a given sequence of random increasing integers

texts:
The AI must predict the next number in a given sequence of incremental integers (with no obvious pattern) using Python but so far I don't get the intended result!
I tried changing the learning rate and iterations but so far no luck!

Example sequence: [1, 3, 7, 8, 21, 49, 76, 224]

Expected result: 467

Result found : 2,795.5

Cost: 504579.43

PS. The same thread exists on AI Stackexchange & Stackoverflow and I've been advised to post it here!

This is what I've done so far:

import numpy as np

# Init sequence
data =\
    [
        [0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
        [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]
    ]

X = np.matrix(data)[:, 0]
y = np.matrix(data)[:, 1]

def J(X, y, theta):
    theta = np.matrix(theta).T
    m = len(y)
    predictions = X * theta
    sqError = np.power((predictions-y), [2])
    return 1/(2*m) * sum(sqError)

dataX = np.matrix(data)[:, 0:1]
X = np.ones((len(dataX), 2))
X[:, 1:] = dataX

# gradient descent function
def gradient(X, y, alpha, theta, iters):
    J_history = np.zeros(iters)
    m = len(y)
    theta = np.matrix(theta).T
    for i in range(iters):
        h0 = X * theta
        delta = (1 / m) * (X.T * h0 - X.T * y)
        theta = theta - alpha * delta
        J_history[i] = J(X, y, theta.T)
     return J_history, theta
print('\n'+40*'=')

# Theta initialization
theta = np.matrix([np.random.random(), np.random.random()])

# Learning rate
alpha = 0.02

# Iterations
iters = 1000000

print('\n== Model summary ==\nLearning rate: {}\nIterations: {}\nInitial
theta: {}\nInitial J: {:.2f}\n'
  .format(alpha, iters, theta, J(X, y, theta).item()))
print('Training model... ')

# Train model and find optimal Theta value
J_history, theta_min = gradient(X, y, alpha, theta, iters)
print('Done, Model is trained')
print('\nModelled prediction function is:\ny = {:.2f} * x + {:.2f}'
  .format(theta_min[1].item(), theta_min[0].item()))
print('Cost is: {:.2f}'.format(J(X, y, theta_min.T).item()))

# Calculate the predicted profit
def predict(pop):
    return [1, pop] * theta_min

# Now
p = len(data)
print('\n'+40*'=')
print('Initial sequence was:\n', *np.array(data)[:, 1])
print('\nNext numbers should be: {:,.1f}'
  .format(predict(p).item()))

Another method I tried but still giving wrong results

import numpy as np
from sklearn import datasets, linear_model

# Define the problem
problem = [1, 3, 7, 8, 21, 49, 76, 224]

# create x and y for the problem

x = []
y = []

for (xi, yi) in enumerate(problem):
    x.append([xi])
    y.append(yi)

x = np.array(x)
y = np.array(y)
# Create linear regression object
regr = linear_model.LinearRegression()
regr.fit(x, y)

# create the testing set
x_test = [[i] for i in range(len(x), 3 + len(x))]

# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % np.mean((regr.predict(x) - y) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(x, y))

# Do predictions
y_predicted = regr.predict(x_test)

print("Next few numbers in the series are")
for pred in y_predicted:
    print(pred)


texts:
I'm trying to evaluate classifiers comparison by running the sample script that can be found here.
What I noticed is that in some cases the classifier is not reset.
In fact, duplicating some of those (with no parameter change) the score and the countour change between the two.
This can be seen simply replacing AdaBoostClassifier() in the classifier list with another MLPClassifier(alpha=1)
I guess that at every cicle of the for loop the classifier should be reset in order to make a fair comparison among the different models, and this case should behave the same I think.
In particular, differences are noticed duplicating the MPL (Neural Net) and the Random Forest, while there is no change duplicating KNN or RBF SVM.
I also tried to clone the classifier, and even del clf in the loop, but the behaviour stays the same.
How can I make the evaluation replicable and not influenced by the previous run? I want to be sure that when I use the same model, and only change the parameters the result is correct, and this will be possible only if two identical model yield the same result.


texts: Perform k-means clustering over multiple columns

texts:
I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not satisfactory.
I know how to perform the algorithm on two columns, but I'm finding it quite difficult to apply the same algorithm on 4 numerical columns.
I am not really interested in visualizing the data for now, but in having the clusters displayed in the table.The picture shows that the first row belongs to cluster number 2, and so on. That is exactly what I need to achieve, but using 4 numerical columns, therefore each row must belong to a certain cluster.
Do you have any idea on how to achieve this? Any idea would be of great help. Thanks in advance! :


texts: Doing predictive modeling on predicted value

texts:
It's a project that I'm working on. Here are the steps I took:

I want to make a recommendation service based on the customer data. I first used a collaborative filtering method to get the recommended products. Then I am trying to use that result as a target variable of my random forest model.

The specifics:

My data: Columns A, B, C, D, E, F. Each row indicates each customer.
Do unsupervised learning(collaborative filtering) to make a recommendation. Only used column A (which has information about the products they have bought)
Choose one recommendation product for each customer, and make that into a categorical column
Merge that column with the original data, and take out column A
My data: Columns B, C, D, E, F, NEW_COLUMN
Use supervised learning to get important features that explains the NEW_COLUMN
I am trying to use supervised learning on the NEW_COLUMN because I want to get some useful predictions that could be used to explain the reason for the NEW_COLUMN(the recommended products)

This is the path that I'm taking. However, is it okay to do predictive modeling with the data that I made using predictive modeling?

Also, if there are any references, it would be a great help!


texts: Is statistics a pre-requisite to learn deep learning?

texts:
I am very interested in deep learning, and I was recommended to read The Elements of Statistical Learning, but it is very hard to get into. So my question is :

How important is learning statistics and reading books like elements of statistical learning, if my end goal is to learn deep learning?

I should have mentioned that I have already done Andrew Ng's Machine Learning course.


texts: How to apply EOT algorithm to 3d model /xpost from AI beta

texts:
Many of you have probably seen the turtle from LabSix that gets mistaken for a rifle in Google's InceptionV3 image classifier. I read the paper and I understand how they apply EOT to 2d images and on the individual pixel values, but I am still unsure how they implement the EOT algorithm to the 3d model.

Are they using EOT to perturb the individual coordinates in the 3d model's mesh? Or are they perturbing images of a turtle and then printing the turtle from the images?
How do they check the InceptionV3 output iteratively without having to 3d print the object each time and check the probabilities given?

Any examples that someone can point to would also be very helpful.


texts: Machine learning for Point Clouds Lidar data

texts:
Our main use case is object detection in 3d lidar point clouds i.e. data is not in RGB-D format. We are planning to use CNN for this purpose using theano. Hardware limitations are CPU: 32 GB RAM Intel 47XX 4th Gen core i7 and GPU: Nvidia quadro k1100M 2GB. Kindly help me with recommendation for architecture.

I am thinking in the lines of 27000 input neurons on basis of 30x30x30 voxel grid but can't tell in advance if this is a good option.

Additional Note: Dataset has 4500 points on average per view per point cloud


texts: Column header in XTS and Data frame in R


texts: understanding the use of additive or multiplicative scale of timeseries?

texts:
In python, by using seasonal_decompose we find whether the given time-series is having multiplicative scale or additive scale data.
But, what is the use of finding it. I haven't seen it's use in building models like SARIMAX or ARIMA or any such models. Can anyone explain its use and where it is used? I'm new to time-series concepts.


texts: How can I parallelize GloVe reverse lookups in PyTorch?

texts:
I feel like I'm missing something obvious here because I can't find any discussion of this.  I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a word generation network.  I'm currently just iterating through the vocabulary on the cpu. I've sped it up a bit using a process pool, as shown in the snippet below, but it's still very slow for large vocabs.

Is there a way to move this to to the GPU using cuda?  I've also read that there is a way to turn this sort of thing into one big matrix operation...  Any references would be appreciated.  Thanks!

glove = torchtext.vocab.GloVe(name='6B', dim=wordDim)

def closest(vec):
    dists = [(w, torch.dist(vec, glove.vectors[glove.stoi[w]] for w in glove.itos]
    return sorted(dists, key=lambda t: t[1])[0]

output = # word vectors…
# using a process pool to parallelize the lookup
pool=ProcessPoolExecutor(max_workers=8)
predictedWords = [w for w in list(pool.map(closestWord, output)]


texts: What is the branch of data science that deals with generating data? Looking for tutorials

texts:
So far all the tutorials (Python/Tensorflow) I have been following dealt with processing the data, they are classification problems. Things that give Yes or No answers. Now I want to find tutorials on actually creating things using ML from large data sets.

For example lets say the task is to generate a face using millions of face photos. This is not a problem that produces statistics from data, but rather a problem of using statistics to generate data. My understanding is, the algorithm would generate random images and run them through the classifier until they closely resemble the face. Is this AI? I wouldn't consider it to be intelligent anything. I understand there is also a ton of creativity that goes into generating the images. But let's say I just wanted to use a +-RNG on every pixel. Is there an adaptive process where it would start matching at higher convolutional levels, where the image is just blobs of information and then use that information, adding or subtracting color, edges, etc, to move to lower levels that more closely resemble a face.

What is this type of problem called and what should I be looking for when searching a tutorial. If there is one that shows you how to do something similar, can you point me to this?

Thank you


texts:
My data is a group of 10 thousand points (each having an node location (x,y)) that are spread across a plane. They are also chromatically-colored based on their weight.
I need to finalize a bayesian nonparametric clustering method that groups points on mainly weight, but also on distance: that is, clusters are by defintion have some relevance to distance, but there are clear topological distinguishing factors between the first quarter and the last quarter of data (I say quarter as an arbitrary amount; in reality, the exact number and topology of the cluster changes through iterations).

As you can see in this picture above, I’ve tried to use notability to create a crude chromatically-colored image of data with varying cluster topology types; over each iteration of my algorithm, the clusters, as mentioned, change location (based on their weight) and their shape, and some overlap (and the possiblility of new clusters (or a possible decrease in the total number) is very high per iteration, where this image represents one iteration of x points)
Additionally, since I am doing this analysis with data via python, i was thinking of using the T-SNE machine learning package as a substitute for a generic clustering method, but I only have a limited knowledge on its functionality. Also, since my data is based on the same weighted scale, it may be overkill.
EDIT: I changed the picture to show overlapping clusters, so it is clearer what I mean. However, keep in mind even these visible clusters are not homogenous in weight (they still vary but in a small threshold). Sure there is noise, but I really want to treat each cluster independently to see each cluster’s behavior over time (as well as clusters that are newly formed, hence the nonparametric method)


texts:
I am working on a project where I have twitter user profiles and their tweets. The users are divided based on their number of followers in two groups (g1 and g2). Then with each user in g1, one user from g2 were matched based on their profile and activity using nearest neighbor (not propensity score). Now I want to do some statistical tests, for example, how differently the sentiment of the tweets changes for these two groups before and after some events. So I have lets say tweets posted within 7 days before and after some date and estimated the mean sentiment scores of all tweets posted by each user in each group. For two groups sample sizes are different (even though they were matched) since not every one posted any tweets within the date range. Now if I want to do a t-test to see if people in g1 has larger positive change in sentiment than g2 after the reference date. I have the following questions:

I am doing an independent sample t-test, by treating each user in each group independent. For each person, take the difference (before and after) in mean sentiment scores for both groups, then testing to see if there is significant different across the groups in terms of change in sentiment scores. Is this appropriate or I should do the matched pairs test? I have gone through other posts here but did not find and definitive answer.
The users who did not have and tweet for the time range, is this okay to assign difference in mean zero, or I should exclude them from the samples?

Thanks in advance. Cheers!


texts: NLP to recognize the meaning of a paragraph

texts:
How can I apply NLP to extract the summary of a paragraph,I found this but was wondering if there is a better and easy way before implementing it, as the ans seems to be an old one.
Basically what I need is, I have 2 paragraph and wanted to know how related they were(maybe in percentage) in context of their meaning.

PS:I am new to both NLP and this forum.


texts:
I'm working on a project in R where I have roughly 1200 emails from a company, most of which are labeled class$_{1}$ or class$_{2}$, which are the types of requests. Roughly 1000 emails are labeled class$_{1}$, and 200 are labeled class$_{2}$.My goal is to used supervised learning to build a model that will classify new emails.

But, after a lot of pre-processing (parsing, removing stopwords, etc.), and trying typical algorithms (SVM, decision trees, etc.) on a document term matrix, my confusion matrix contained many false positives and false negatives, but only a few false negatives with SVM.

I'm wondering how could I improve my results? Do I need to use oversampling,  or bi-gram feature representation? I guess the problem is that the topics of the two categories are really close.


texts: Can i expect good results having low correlation attributes?

texts:
This was a question i saw in an interview for a data scientist position:
"Here is the following correlation heatmap that i got from my attributes. Regarding the correlation of each feature with the dependent variable (target/class), it is noticeable that correlations are not very expressive.

Yet, i would like to know if can i expect good results from a classification model using this dataset. Also, what further investigation can i do (if i shouldn't look after correlation only)?"


texts: How to find a similarity value between cars

texts:
So I have a database of web-scraped cars, and I want to find the similarity between cars based on the km driven (e.g 69000), the model year (e.g 2012), and the trim of the car which will be one of three categories: base, mid, top.

What similarity measure can I use that will give me a decently accurate similarity ratio


texts: Can autoencoder latent variables to be used as features for classification?

texts:
I did some experiments on convolutional autoencoder by increasing the size of latent variables from 64 to 128. I used 4 covolutional layers for the encoder and 4 transposed convolutional layers as the decoder. When I increase the size of latent variables from 64 to 128, the autoencoder will give result in good reconstructed images.
However, when I tried to use the features from the latent variables for classification task, it turns out increasing the size of the latent variables will decrease the performance of classification task.
On the other hand, I also built autoencoder using resnet-34 (34 layers depth). It turns out deeper layer gives something what I want, for example increasing the size of latent variables from 64 to 128 will increase the quality of the reconstructed images as well as increase the classification task.
My question is: How can the features of latent variables from resnet-34 give better performance to the reconstructed images and classification task? and How can also the features of latent variables from 4 layers convolutional autoencoder will give bad result for the classification task but good result for the reconstructed images?


texts: Can I use depth prediction map to infer horizontal distances?

texts:
I have a hardware + software setup that uses a sensor to give good estimates of depth, onto a pixel map - think Kinect or similar. Example below for context:

Now assume I can access individual pixel values that give an estimate of depth distance from the camera source.
My question is, can I use this information to infer/estimate horizontal distances? For example, I want to know how wide an aisle is, or the approximately horizontal distance between two points.

For example, I would want to estimate the distance of the horizontal arrow line.
Hope this question is appropriate for here, if not let me know a better stackexchange. Thanks!


texts: What kind of data is not appropriate using CF to do recommendation?

texts:
I am currently working on a recommendation system for daily news. At first, I evaluated all the recommender algorithms and their corresponding settings (e.g., similarities, factorizers, ...etc) implemented in Mahout. Since we want to recommend daily news for users, we use the reading behavior of each user collected two days ago as training set, data of the next day as the testing set. The evaluated RMSE is good, the best recommender is SVD+SGD, so we implemented the recommender on our system for several days of trial run.

However, the result, the actually recommended news, seems to be not very attractive for real users ("not attractive" here means, the users feel like "why you recommend this to me?"). So we decided another approach: use the tags and categories and their relationship to do the main job of recommendation, the result from CF is for just supporting.

This makes me wonder if CF if not appropriate for some kind of content. Because I also worked on movie and music recommendation, CF is a good tool. But for news, it seems not the case.

Can anyone explain why this happening, and also give some guideline about how to choose appropriate recommendation methods? Thanks:)


texts: What is the role of logistic function in logistic regression?

texts: After training Logistic regression on training data, getting each test sample to algorithm computes weighted sum of sample features. If this is greater than zero, we know the sample is from negative class and otherwise, sample is positive. I want to know why we need to calculate the logistic function that map our weighted sum to range zero and one. One reason can be that we want to interpret the results as probability. are there any other reasons?


texts: Incorrectly applying random forest model?

texts:
I am fairly new to random forest models (and data science in general), and was wondering if I am operationalizing the model I created correctly.

Context: I am creating a random forest model to predict employee voluntary turnover.

Problem: Test set has an accuracy of 97% (AUC .992[this seems too high], Precision: 88%, Recall: 99%) vs train set accuracy of 96%, and yet as we keep getting new voluntary terminations, their probabilities based on the model tend to be less than .1. I.E., indicating that they are not going to leave.

I guess that makes sense considering that as of the time when the model was created, those employees were still within the organization and therefore the model accurately classified them as with the company, but accurately classifying whether an employee is still with the organization or not is of no help, I need to be able to identify those employees that have a higher probability of leaving, which is what my understanding of what the RF model would do (I have done this before with logistic regression models).

Possible Explanations for this that I can think of:

1) Data set doesn't seem too imbalanced:  8054 non terms vs 2158 terms

2) Overfitting? But the test set doesn't drop substantially in accuracy

3) high correlated predictor variables?

Code Attached.

Thanks!


texts:
I have a data-set with 2 target classes.
In training dataset, the ratio of the 2 classes are 1:93

With my neural network, the current accuracy is 63%.
I tried undersampling, oversampling, equal sampling but not improvements.


texts: How to calculate NDCG in recommendation system

texts:
This is a question about NDCG, which is a recommendation evaluation metric.

The following are being used as evaluation indicators for recommendations.

$$DCG = r_1 + \sum\limits_{i=2}^{N}\frac{r_i}{log_2i}$$
$$nDCG = \frac{DCG}{DCG_{perfect}}$$

The code is as follows:

def dcg_score (y_true, y_score, k = 20, gains = "exponential"):
    """Discounted cumulative gain (DCG) at rank k
    Parameters
    ----------
    y_true: array-like, shape = [n_samples]
        Ground truth (true relevance labels).
    y_score: array-like, shape = [n_samples]
        Predicted scores.
    k: int
        Rank.
    gains: str
        Whether gains should be "exponential" (default) or "linear".
    Returns
    -------
    DCG @k: float
    """
    order = np.argsort (y_score) [::-1]
    y_true = np.take (y_true, order [: k])

    if gains == "exponential":
        gains = 2 ** y_true-1
    elif gains == "linear":
        gains = y_true
    else:
        raise ValueError ("Invalid gains option.")

    # highest rank is 1 so +2 instead of +1
    discounts = np.log2 (np.arange (len (y_true)) + 2)
    return np.sum (gains / discounts)

def ndcg_score (y_true, y_score, k = 20, gains = "exponential"):
    """Normalized discounted cumulative gain (NDCG) at rank k
    Parameters
    ----------
    y_true: array-like, shape = [n_samples]
        Ground truth (true relevance labels).
    y_score: array-like, shape = [n_samples]
        Predicted scores.
    k: int
        Rank.
    gains: str
        Whether gains should be "exponential" (default) or "linear".
    Returns
    -------
    NDCG @k: float
    """
    best = dcg_score (y_true, y_true, k, gains)
    actual = dcg_score (y_true, y_score, k, gains)
    return actual / best

Assumes k = 5.

At this time, how should NDCG calculate for items that could not be recommended within kth?

For example,

y_true = [5,4,3,2,1]

y_score = [0,0,0,0,0] # 0 means we could not recommend within the top 5

At this time,

>>> np.argsort ([0,0,0,0]) [::-1]
array ([3, 2, 1, 0])

So, following the above code,

NDCG @ 5 = 1.0

This looks strange.

In such a case, should the score be 0 and not be included in the NDCG score calculation?

If you have any references, just showing them is fine with me.

Thank you.


texts:
I am new to prolog and need help on how to write a prolog program. Here is what i am trying to do. I have downloaded the dataset from this link - https://archive.ics.uci.edu/ml/datasets/Blood+Transfusion+Service+Center
My goal is

Use Any of the decision tree algorithms to build a decision tree for the given data
Create rules from the decision tree.
Code the rules into a Prolog Knowledge base
Get the data about the blood donor from the user and predict if the donor has donated in March 2007.

I need some guidance on how to start the work. I have developed algorithms in python and understand #1 and #2 but have no idea on work with #3 and #4.


texts:
I am running a RandomForestClassifier on my data but my jupyter notebook is very slow. It took almost 2 hours to run the below code:

rf = RandomForestClassifier()
rf_random = RandomizedSearchCV(estimator = rf, param_distributions =
   random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
rf_random.fit(X_train, y_train)

My dataset has 30K rows and 300 features.

I am not sure if something is wrong with my code or jupyter notebook configuration.
I am using a remote desktop windows machine.

I would really appreciate any help!
Thanks in adnvace


texts: Building Recommender for book paragraphs

texts:
I have some application which are offering a book to read. Users normally read some paragraphs of it only (it contains +6000 paragraphs).
Looking at scatter for users vs paragraphs:

Which you can see is semi equal distributed. Using SVD algorithm for matrix factoize gives a semi random predictions. I have total of 18k records of user read paragraphs. Looking for users, it seems that a user is reading semi random set (i.e. it is hard to specify common topics for a single user readings)
Can you suggest me how to produce suggestions related to each user ?


texts: How does a Recommender System recommend movies to a New User?

texts:
Consider a New user which has never rated any movie on the Website or the System has never seen the user.

How does the System recommend Movies to the User and based on what ?
How will we evaluate the Recommendation , is the recommendation accurate or appropriate and how confident the system is about the Recommendation .


texts: Custom conditional loss function in Keras

texts:
I'm looking for a way to create a conditional loss function that looks like this: there is a vector of labels, say l (l has the same length as the input x), then for a given input (y_true, y_pred, l) the loss should be:

def conditional_loss_function(y_true, y_pred, l):
    loss = if l is 0: loss_funtion1
           if l is 1: loss_funtion2
    return loss

It is like a kind of semi-supervised loss funtion.


texts:
I am building a document classifier using Naive Bayes. There are 10 classes. My question is that :

1 Should each class contain the same number of documents for training?
  What if the number of training example in each class is different?

2 Does the number of classes and classification algorithm have any relation? say is there any thump rule like if there are 100 classes shall algorithm 'X' has better performance than 'Y'


texts: TypeError: unhashable type: 'numpy.ndarray''

texts:
When I tried to execute the below code

for epoch in range(training_epochs):
    sess.run(training_step, feed_dict={x:train_x,y:train_y})

the following error is raised:

  TypeError: unhashable type: 'numpy.ndarray'

How to solve this problem?


texts: Colab can not connect to GPU from a python file

texts:
I am trying to run a github deep learning repository in Colab but I can not connect the python files to colab GPU.

I can connect to GPU when writing a script in the colab notebook e.g. when I run this cod from a notebook cell :

import os, torch
print('Torch', torch.__version__, 'CUDA', torch.version.cuda)
print('Device:', torch.device('cuda:0'))
print(torch.cuda.is_available())

I get:

Torch 1.4.0 CUDA 10.1
Device: cuda:0
True

but when I run it from a file called myExample.py  e.g. using

!python myExample.py

I get:

Torch 1.4.0 CUDA 10.1
Device: cuda:0
False

Is there any solution to this?


texts:
When training the generator, we want to minimize log(1 - D(G(z))) (this has a form of log (1-x)). Using graph calculator we plot and see that the slope of this function is very small at first, thus giving handicaps for G when training in early stages. To solve this, we have to equalize the "minimize log(1 - D(G(z)))" to minimize -log(D(G(z))) (-log x). Using the graph plotter again, we can see the gradient at the early steps is much more stronger.

However, in this graphs plotted, the x value is increasing, while our data is images. Can someone explain the intuition or clarify this point for me? I can still dont understand why inverting to -log(D(G(z))) helps Generator learn faster and better.

The plots of two functions:


texts: What kind of algorithm should I choose for this music classification system?

texts:
I have in mind a program for analyzing short fragments of music, categorizing them as "good" or "bad". This would be part of a larger program that searches for larger good phrases and whole pieces.

The idea now is to take a fragment of music, X, and compare it to known good sample fragments G_1, G_2, ... etc. and get a rank of similarity to each one. Then compare it to known bad fragments B_1, B_2, B_3, .. etc.

"Good" music is subjective of course, but this program would work with G and B samples that I hand-optimized according to my own tastes.

Good music is then music that resembles at least one of the G's, while resembling none of the B's. A fragment that has strong similarity to both G's and B's is probably bad: The B's have veto power.

So, how to determine similarity? Musical fragments can be represented by image-like matrices of pixels. The vertical axis is pitch and the horizontal axis is time. If a note of pitch P_1 occurs between times T_beg and T_end, then that's like drawing a little line between (T_beg, P_1) and (T_end, P_1).

An sample X to be classified can be convolved, in a sense, with a known sample K. It can be transposed up or down (vertical shifting) or moved left or right in time (or stretched in time, or other alterations) and each transposition would be superimposed on the G or B sample. I'm not too familiar with convolution but I think that overlapping pixels are multiplied and the sum of all is taken. The transposition with the brightest result pixel is a good indication of how similar X is to the K sample: it's magnitude becomes the measure of similarity.

Dark pixels don't matter much. A preponderance of dark pixels doesn't make music bad. It just means the real pattern isn't found in those locations. A bright match to a known bad fragment is what makes music bad.

I'd like to perform these computations with NumPy or a similar language optimized for matrix or image computations.

Can I get some idea whether there is a name for this kind of operation, and where to look for efficient implementations of it? Boosting speed with a GPU would be a bonus.


texts: deepAR RNN from AWS Sagemaker - should I clean the data first?

texts:
I test the Sagemaker AWS solution for RNN: deepAR.
Previously I used sklearn for this and obviously I cleaned the data to avoid highly correlated time series (KBest, PCA, etc.)
Do you think it is needed here, or RNN will handle internally correlated series.
PS. This is typically the case with blackbox commercial solutions. They look fantastic, but when it comes to details you have nowhere to consult.


texts: What should I observe when choosing which optimizer suits my Deep Neural Network model?

texts:
I have trained my neural network model with optimizers such as RMSProp, AdaGrad, Momentum, and Adam.

Currently, after running the code, I have printed out the Train and Test Accuracy of every epoch (50 in my case). However, I would like to know how should I determine which of these optimizers performs the best?

Does a higher train accuracy at the last epoch determine which is best or would a higher test accuracy do so? Also, I observed that when using the Momentum optimizer, the model train accuracy reached its' highest around 0.91 in the 16th epoch compared to the other optimizer.

Hence, would that conclude that the Momentum optimizer performs best in this case?


texts: What type of consideration can be made using clustering?

texts:
I am clustering my data to see how information look like and which group may be identified.
Since clustering is an unsupervised algorithm, I cannot test the accuracy of the classification.
So I was wondering what type of consideration I can make after using clustering.
For example, if I had many emails, with no flag or label for spam/not spam, how could I use clustering to group them into two groups and test the ‘accuracy’ of the clustering?
To give more context on what I am trying to do:  I have different files (csv) having fields like date, users, emails’ subjects and emails’ bodies.
I would like to run some analysis but, in order to do this, I would need to classify emails into spam/not spam.
I have 23000 emails so it is very difficult to do this manually. I already included in a list of words the common words used as flag for spam (ads, buy, offer, porn, promotion,...) but, since the most of emails has no these words in a title or  in the body, this first step can assign ‘spam’ flag to around 100 emails. Very low! I have tried with topic classification (lda) but it is not so accurate. I thought then to use k-means clustering to assign these labels, once labelled manually around 300 emails.
I do not know if this is the right way to proceed for assignignig labels, so comments and answers would be greatly appreciated.


texts:
I have to perform multi-step multivariate forecasting of time series, using keras. I found an example where LSTM is used. I could modify that example replacing LSTM with SimpleRNN.
Now I would like to use Elman RNN, which is different from SimpleRNN, as far as I know. How could I implement Elman RNN using keras?
I try to be more explicit, illustrating in detail what I would like to do.
I have three sequences, two of them representing the input, one the output:
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

The sequences are used to fill the data structures X and y such that the following loop:
for i in range(len(X)):
print(X[i], y[i])

gives:
[[10 15]
[20 25]
[30 35]] [65 85]
[[20 25]
[30 35]
[40 45]] [ 85 105]
...

This means that I associate the values of X related to 3 time steps, to the values of y related to 2 time steps.
n_steps_in, n_steps_out = 3, 2

The Elman model I am trying is:
model = Sequential()
model.add(SimpleRNN(100, activation='relu', return_sequences=True,
input_shape=(n_steps_in, n_features)))
model.add(TimeDistributed(Dense(n_steps_out, activation='relu')))
model.summary()
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=200, batch_size=n_steps_in, verbose=0)

where
n_features = X.shape[2]   # i.e., 2

Now, when I test the prediction with
x_input = array([[70, 75], [80, 85], [90, 95]])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

I get results like:
[[[195.89265 224.1713 ]
[162.78471 194.95282]
[139.1635  161.6026 ]]]

but also like:
[[[207.80466   0.     ]
[189.1255    0.     ]
[184.61163   0.     ]]]

or like:
[[[  0.      240.88077]
[  0.      218.17479]
[  0.      205.80429]]]

I would like to understand why sometimes there are 0. values in yhat (i.e., the predicted y) and why yhat has 6 values (3 pairs) instead of 2 (1 pair).


texts: R, caret: does caret automatically create dummy variable for random Forest?

texts:
I created a two class data set, dat. This data set only contains 25 variables. 24 are numeric and 1 is categorical (with 26 levels). The result of the auto-tuned model gives a mtry of 2, 14, 26, 38 and 50. So is this true that the caret created the dummy variable for the categorical variable? What other models does caret create dummy variable automatically?

Please see following example code.

pacman::p_load(R.utils, caret, doParallel)
cl <- makeCluster(8, outfile="")
registerDoParallel(cl)
set.seed(2)
dat <- twoClassSim(48842, noiseVars = 10)# 25 variables.

dat = cbind(dat,data.frame(categorical = sample(LETTERS,nrow(dat),replace = T)))

fit_on <-  list(rs1 = 1:1300)
pred_on <- list(rs1 = 1330:10000)
indexFinal = 1:1000

fitControl <- trainControl(method = "cv",number=1,repeats=1,
                           classProbs = TRUE,summaryFunction = twoClassSummary,
                           index= fit_on, indexOut = pred_on,
                           indexFinal = indexFinal,
                           verboseIter = TRUE,
                           savePredictions = TRUE
)

model = train(Class ~ ., data = dat,
              method = "rf",
              preProcess=c("center","scale"),
              trControl = fitControl,
              tuneLength = 5)

model

# mtry  ROC        Sens       Spec
# 2    0.9166711  0.9298806  0.6778085
# 14    0.9237881  0.8851236  0.7645137
# 26    0.9177710  0.8746803  0.7770797
# 38    0.9132661  0.8689258  0.7753204
# 50    0.9093577  0.8687127  0.7728072
# there are only 25 variables.

stopCluster(cl)


texts: What is the hypothesis space of decision tree learning?

texts:
Could you please explain what the hypothesis space for decision tree learning look like?

And what is the cardinality of this space?


texts: Underlying model for prediction using different prediction variables

texts:
I have time-series energy consumption data for a duration of one-month. The frequency of data is half-hourly. The features of dataset are

temperature - temperature value at particular time instant
humidity -    humidity value at particular time instant
timeofday -  this corresponds to 48 half-hourly durations of a day.
weekday -  day of the week (value between 1 - 7)
prevday1 -  energy consumption at the same time-instant on previous day
prevday2 - energy consumption at the same time-instant on a day before the  previous day
prev_instant1 - energy consumption at the previous time instant. For example, for 7 pm, previous_instant1 is 6:30 pm
prev_2_hour - average energy consumption during the last two hours

Using these variable I need to forecast power (energy) consumption. The pairs plot of these variables is

I started with a linear model, but it seems that the model is performing very badly. I want to know is there any automatic technique, which can check all possible combinations of predictor variables and output the best fitted model. I can check all models (linear combinations, non-linear, splines etc) manually, but it will take huge time.

UPDATE

I am forecasting in real-time at a half-hourly rate (very short term forecasting). Also, I use separate models corresponding to different half-hours of the day. For example, In a day we have 48 half-hours so corresponding to each half-hour I have a separate model. For the whole process, I use following approach:

Create a training data set with above mentioned features. This training data set is of one month duration (30 days)
In the next step, I forecast for the next day (testing day) using specific models for each of 48 half-hour durations
At the end of the day, I retrain all my 48 models with recently updated data (recent historical 30 days)

Steps 2 and 3 continue till stopping condition. Stopping condition used is the no. of testing days. A screenshot of my training data is as:

head(training_data)

tail(training_data)


texts: What happens to a machine learning technique (specifically Decision Tress and Logistic Regression) if the validation dataset has a new category?

texts:
Let's suppose I have a dataset which has a categorical variable and the problem I am solving is a classification one.

This categorical variable var has ['A','B','C'] as the possible set of data.

What happens to a decision tree if a new category 'D' is seen only in the validation data set (meaning: absolutely new data)? Supposing the variable var is a feature used in the tree.

With the decision tree:

Does it give an error? The decision tree stops the path and returns the not-final-node's probability?

With the Logistic Regression:

The dummy variables are zero for all the categories (I suppose), and then the model runs normally?


texts: Logistic Regression Multi-level Independent variables


texts: Get values from column that appear more than X times (without pandas or external libraries)

texts:
I have a .csv file that contains a row with various loan IDs. I would like to be able to output the IDs that occur 6 or more times.

CUST_ID,PROD,BAL,PER
1151612518-107,1151612518-107c,780.1,4
1151612518-1049,1151612518-1049a,344.72,2
1151612518-813,1151612518-813a,1376.1,6
1151612518-427,1151612518-427a,913.51,4
1151612518-1056,1151612518-1056a,1851.49,7

The data is in this format and I would like to have a way to analyze the first column and output every CUST_ID that occurs 6 or more times.


texts: Dimensionality reduction for feature extraction when missing some feature values

texts:
I have two questions:
1-Which method is appropriate for dimensionality reduction for feature extraction when missing some feature values?
2-Which textbook is the best source for the answer in (1)?


texts: Classification task - how to handle large data file?

texts:
I'm planning to construct a classification model for predicting New York taxi trip fare. The CSV-datafile for this is very large, containing 112 234 626 rows (ca 10 GB). I have managed to download this file to my computer's local disc but
the further memory problems arise when I'm starting to use this file for parsing and model construction.

So, my questions are:

What is the optimal number of rows that I should take from this datafile for use in my task?
How should I take these rows to prevent low memory problems?
How should I take these rows so that it contains optimal sample of all the months. The file contains data of year 2018 and if I take the rows so that it contains only pre-specified number of rows then for example the dataframe may contain only data for January (but I want all months).

2018 Yellow Taxi Trip Data


texts:
I'm currently working in a problem of Object Detection, more specifically we want to count and differentiate similar species of moths.

We are already testing some detection algorithms:

YOLOv3
RetinaNet
Mask R-CNN

For the datasets, first we'll scrape the web for those images to have a starting model.

Then we'll get plenty of pictures from our devices that will look like this:

Unfortunately it will take 6 months to get those pictures.

Currently we have some samples of moths, and we have the resources to take pictures of them in a studio. We want to simulate the field conditions as well as taking close ups of the moths in different angles and lighting conditions. This experiment would also allow for us to get the accuracy of classification from an expert entomologist.

Is the effort of taking studio photos worth it?

Would these pictures significantly increase the models accuracy?

Is there any opportunity for publication?

Is there any paper that uses studio photos as well as field photos for object detection?

Also, there's plenty of opportunities for publication here in the company, so they would go the extra mile if we publish something, or if these pictures would improve the final accuracy of the models.


texts: Intuition for Logistic Regression Performance

texts:
I have a bunch of time series data I'm doing classification on. I used TPOT with a custom cv (walkforward time series split) to find the best performing model. Logistic regression was selected. I'm a bit surprised by this, given that most kaggle competitions are won by ensembles of newer models like XGB, etc. In fact, not only was logistic regression the best model but it was actually the only one that learned something meaningful (outperformed the others like RF, XGB, etc. by > 5% where 60% is the best accuracy it achieved).

My expertise is mostly in deep networks, not as much the stuff dealt with by sklearn, but in this case the training data is very limited and quite noisy. What should I take away from logistic regression being the best classifier? Can I use this knowledge to inform my design of other classifiers that could perform better? Again, I have a lot less experience with this side of ML, so I'm really looking for an intuition as to how I can take this knowledge and use apply it to build better models or ensembles.


texts: max_iter hyper parameter in sklearn.cluster.MiniBatchKMeans

texts: What is the significance of max_iter in sklearn.cluster.MiniBatchKMeans? Is this the maximum number of times partial_fit() can be executed on batches of data?


texts: How can the process of hypertuning of XGBoost parameters be automated?

texts:
I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here.

After performing grid search and some manual settings, I found that the following parameters work the best for me:

weight <- as.numeric(labels) * nrow(test) / length(labels)
upscale <- sum(weight * (labels == 1.0))

xgb_params = list(
  objective = 'binary:logistic',
  eta = 0.1,
  max_depth = 4,
  eval_metric = 'auc',
  max_delta_step = 10,
  scale_pos_weight = upscale
)

How can the process of setting optimal hyperparameters for xgboost be automated for best AUC? Please note that some of these parameters aren't supported by the caret implementation of xgboost but are very important for the model I have to design.


texts: Tensorflow Calculate error for a single neuron

texts:
I'm required to be able to calculate the error on a given neuron in a neural network using Tensorflow.

Using this :

mse = tf.reduce_mean(tf.squared_difference(out, Y))
opt = tf.train.GradientDescentOptimizer(0.1).minimize(mse)

i just get the updated weight, how i can get or calculate the error for each neuron in my network.

Thanks for the replies.


texts: What's the best way to do classification basing on two given datasets (annual data and daily data)?


texts:
I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point traversing an iTree, and $n$ is the sample size used to grow the iTree.

  The difficulty in deriving such a score from $h(x)$
  is that while the maximum possible height of iTree grows
  in the order of $n$, the average height grows in the order of
  $log(n)$. Normalization of $h(x)$ by any of the above terms
  is either not bounded or cannot be directly compared.

So herein lies my first question. What do they mean that the normalization of $h(x)$ by any of the above terms is either not bounded or cannot be directly compared? The final score function in this paper is given by

$$s(x,n)=2^{-\frac{E(h(x))}{c(n)}}$$

where

$$c(n)=2H_{n-1}-2(n-1)/n$$

is the average path length of an unsuccessful search from BST Theory. Note how they are taking the expectation of the path. So in that case we are averaging all the values anyway, so why can we not use the growth of the average height of the tree? Additionally they don't even mention the analytical form of average height here, which from what I gather is $2\sqrt{\pi n}$ reference, though I have not read this reference thoroughly and may be mistaken.

Am I missing something here?


texts: How to quantify a tokenized user agent string for a neural network?

texts:
I am currently experimenting with user agent strings. My current plan is to tokenize the user agent string and split it in its important parts. Like OS, Browser Name, etc.

After that, does it make sense to just create some sort of map where iOS = 0.1 and Windows = 0.2 where i map the strings to numbers for the operating system feature of my network?

Is that the current best approach to work with such data, or should I try a different approach? The network should predict which category the user will click, and than move that category to a more prominent place.

For example, I expect that an iOS user prefers an iPhone mobile case more than an android one.

Thank you for helping with my experiments.


texts: Elastic Regression fitting good mean and bad variance

texts:
So, I'm kinda new to machine learning and I was trying to predict the monthly sales of a business using a set of features and using a sliding window of the past sales of 12 months.

I used some algorithms to do it, including linear/polynomial regression, lasso/elastic and SVR. I got the best results with elastic regression resulting in the following result:

As it shows, the model fit the mean of the curve somewhat well, but I would like it to fit the variance as well. So, I've been searching what technique or feature to use could better fit my data, but I still found nothing precise.

Would someone knows what could I do to to take the variance of the system into account?

Thanks in advance!


texts:
In the paper Large-Margin Softmax Loss for Convolutional Neural Networks the author has a figure as below:

He is claiming that he is using only 2d features to classify MNIST with a CNN. How is he doing that? Is he reducing the dimension of MNIST images to 2 and how? How do you visualize such figures in 2D if the classification is for 10 classes?


texts: Getting prediction labels from TensorFlow 2 ImageGenerator


texts: How to best summarize and visualize the outcome and performance of a multi-class problems?

texts: I estimate a multi-class origin-destination model with 45 classes. In particular the classes are geographical regions between which people can move. Currently I summarize the overall performance using Cohen's kappa to take the unequal class sizes into account. But I would love to provide deeper insights into the performance. Do you have any recommendations of how to proceed with so many classes?


texts: specifics of dilated convolutions in tensorflow

texts:
This ones a little esoteric. I'm developing a wave-net inspired model. First time needing dilated convolutions.

My question is which values is the convolution looking at relative to the output position? For example if I have a Conv1D with kernel_size=4, dilation_rate=2 and padding='same' are the inputs split evenly to both sides so relative positions of input data would be (-3,-1,1,3), or justified to one side giving relative positions like (-6,-4,-2,0) or something different? Is there an easy way to control this behavior?


texts:
I am trying to understand the input for the PrefixSpan model (pyspark). I thought in the input RDD, each row should be a list. However, as I looked into the source code, each row is actually a list of list. Why does it need a list of list instead of just list? Thanks!

https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala

 def trainPrefixSpanModel(
      data: JavaRDD[java.util.ArrayList[java.util.ArrayList[Any]]],
      minSupport: Double,
      maxPatternLength: Int,
      localProjDBSize: Int ): PrefixSpanModelWrapper = {...}


texts: Calculating error from accuracy score in Sklearn

texts:
I have one question thats maybe simple, but my brain overfited :)

I wrote a code for simple linear regression in Python and Scikit-Learn, and I calculated score of my model.

        accuracy_test = model.score(x_test, y_test)

I got the score of 0.97. Does that mean that my error is 0.03 (3%)?
For example, Is it correct to say, that If i predict some value with model.predict function, and I get result of 100 with that, that I can say that my result will be in range from 97 to 103 because of my 3% error (score of 0.97)?

Can you say this for accuracy_score also?


texts: Matching financial reconciliation data / matching multiple rows based on column values

texts:
I'm working with financial reconciliation data and the ask is to train the algorithm to match transactions (that are otherwise manually matched if the existing application didn't because not all the rules were met).

The Currency, Account, Client have to be the same, the Settle & Trade dates of each item can be the same or vary, the Net Settlement should ideally be 0 (when the sign for one item is flipped) or as close to 0.
The rest of the columns aren't as helpful because they can be blank or vary for the same match.
So if for 2 rows of data the currency, account & client are the same, one of the dates varies by 1 day & the settlement difference is 0.50 - they're reconciled (or at least tagged as an 80% match or a partial/possible match). In the picture for the 2nd set, 3 items are reconciled - the difference is -48 and the dates vary a bit.
There are also instances where it's 5+ (upto thousands) items of data all being part of the same transaction (this makes my head hurt and the math aspect of it makes this seem easy but it doesn't always become 0 and looking at the "training" data makes no sense  )
I was wondering if this is feasible (because there are so few unique identifiers and so many possible number of matches), what would be the best method(s) - algorithm, language, libraries etc., For a start, just being able to classify only 2 (not more) exact (same amount) matches would be great. I do have access to plenty of training data.
I have only worked with R & Python for analysis/simple predictions and this was over 1 year ago.
Since I started at my first job I've basically been doing data visualization and not much else and have completely lost touch.
If it was just reconciling matches, a macro or a large if-statements riddled program would do, but we want to know why items have been matched so the existing application can be bettered.


texts: How to Avoid rarely used discrete feature values in a dataset

texts:
On Google's ML crash course it states:

  Good feature values should appear more than 5 or so times in a data
  set. Doing so enables a model to learn how this feature value relates
  to the label. That is, having many examples with the same discrete
  value gives the model a chance to see the feature in different
  settings, and in turn, determine when it's a good predictor for the
  label.

This make sense, but if that happens what should we do? For example, consider a dataset with street_name as feature and out 2000 rows, only 4 of them have street_name equal to "Learning St.".

Should we remove those rows containing rare feature values? Or what?

Any insights would be greatly appreciated.


texts:
My goal is:

Calculate the probability of a client take a loan.

My problem is:

How to take in account the date of the loan in this process.

Certainly the date is important here since, let's assume my clients usually take loans closer to Christmas, for example.

My initial thought was to use the day of the year, since this would absorb those "special dates" like between days 340~365 they are more likely to take the loan since this is close to Christmas and this way I don't have to "manually consider" those special dates and can even discover new special periods.

Question:

Is this a sound approach?


texts: Computing derivatives for backpropagation across a convolution step

texts:
This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and intermediate outputs of convolution layers. Here's the simple setup I'm going with for now:
For practical purposes, I've heard we generally use cross-correlation, so my entire treatment of the problem is based around that. Let $I$ be an $m\times n$ image (or an intermediate conv layer output) on which a $p\times q$ filter $K$ is used to produce output $O$. Starting out, I'm only considering valid cross-correlation performed on $I$, which means no padding. We then have:
$$O(i,j)=(I\star K) (i,j)=\sum_{u=1}^p\sum_{v=1}^qI(i+u-1,j+v-1)K(u,v)$$
where $\star$ means cross-correlation. Let $dX$ generally denote the derivative of the loss function w.r.t. the quantity $X$. Then
$$dK(u',v') = \sum_{i=1}^{m-p+1}\sum_{j=1}^{n-q+1}dO_{ij}\frac{\partial O_{ij}}{\partial K(u',v')}
\\=\sum_{i=1}^{m-p+1}\sum_{j=1}^{n-q+1}dO(i,j)I(i+u'-1,j+v'-1)
\\=\sum_{i=1}^{m-p+1}\sum_{j=1}^{n-q+1}I(u'+i-1,v'+j-1)dO(i,j) = (I\star dO)(u',v')
\\\implies dK = I\star dO$$
To find $dI$, first I can apply a change of variables to the formula for $O(i,j)$. Let $i'=i+u-1$ and $j'=j+v-1$. Corresponding to the summation limits $u=1$ and $u=p$, we have $i'=i$ and $i'=i+p-1.$ Similarly for $v$ and $j'$. We get:
$$O(i,j)=\sum_{i'=i}^{i+p-1}\sum_{j'=j}^{j+q-1}K(i'-i+1,j'-j+1)I(i',j')$$
Then
$$dI(i'',j'')=\sum_{i=1}^{m-p+1}\sum_{j=1}^{n-q+1}dO_{ij}\frac{\partial O_{ij}}{\partial I(i'',j'')}$$
Note that there will be values of $i,j$ such that $i''< i$ and/or $j''<j$ in which case the derivative $O(i,j)$ w.r.t. $I(i'',j'')$ will be zero (since the summation indices $i',j'$ in the formula for $O(i,j)$ above will start from $i,j$ respectively and won't equal $i'',j''$).
So at this point I'm stuck. Can anyone help me out from this point on, to derive the expression for $dI$? I worked through the specific case where $m=n=4$ and $p=q=2$. In that case, it turns out that $dI=dO*K'$, where $*$ denotes full cross-correlation with zero-padding and $K'$ is the transposed (flipped) version of $K$. But how do I deal with the problem of indices $i'',j''$ going out of bounds in the above equations? Is there an alternative approach?

Secondly, how are the formulas for $dK$ and $dI$ generalized to the case when stride $=2$ ? Again, I worked out for the specific case $m=n=4$ and $p=q=2$ as follows:
Let the stride in the $x$ ($y$) direction be $s_x$ ($s_y$). Then
$$O(i,j)=\sum_{u,v=1}^2K(u,v)I((i-1)s_y+u,(j-1)s_x+v)$$
(You can check the above formula yourself) Then
$$dK(u',v')=\sum_{i,j=1}^2dO(i,j)I(u'+(i-1)s_y,v'+(j-1)s_x)$$
I don't see any easy way of writing this in the form of a cross-correlation formula. For example, if $s_x=s_y=2$,
$$dK(2,1)=dO(1,1)I(2,1)+dO(1,2)I(2,3)+dO(2,1)I(4,1)+dO(2,2)I(4,3)$$

How does one efficiently implement an operation like this? Not to mention I haven't even touched upon what $dI$ looks like with strides greater than $1$.


texts: Treating missing data in categorical features

texts:
I have a dataset with one of the  categorical columns having a considerable number of missing values. The interesting thing about this column is that it has values only for a particular category in "another" column .
For eg :
column 1                        column2
========================================
Google                             -
Google                             -
Google                             -
Google                             -
Facebook                        Image
Facebook                        Video
Facebook                        Image

My column of interest has values only for one category (Facebook) that is present in another column. Therefore, the missing values for google cannot be imputed with average, cannot be predicted and those rows cannot be ignored either.
In such a situation, is it wise to consider the missing values '-' as a separate category in one-hot encoding? Or will this affect my machine learning model badly?


texts: day is out of range for month import from csv file in python

texts:
Here I have one csv file with different values and date_time. I wrote the code and run it. Then it gave me an error  time data '8/6/2018 6:45' does not match format ' %d/%m/%Y %H:%M:%S'
I want to plot this different three values in one graph. Can anyone help me to solve this problem?
Here is my code:

condition = ""
date_time  = []
x1 = []
x2 = []
x3 = []
def convertTime(s):
  tm = time.strptime(s, " %d/%m/%Y %H:%M")
  return datetime.datetime(tm.tm_year,tm.tm_mon, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec)
with open('data43.csv','r') as csv_file:
csv_data = csv.reader(csv_file, delimiter=',')
 row_num = 0
for row in csv_data:
 if(row_num == 0):
  condition = row[0]
elif(row_num > 1): #Data starts here
  if(row[0] != ''):
    date_time.append(convertTime(row[0]))
  if(row[1] != ''):
    x1.append(int(row[1]))
  if(row[2] != ''):
    x2.append(int(row[2]))
  if(row[3] != ''):
    x3.append(int(row[3]))
row_num = row_num + 1
fig1 = plt.figure(1)
ax = fig1.add_subplot(2,1,1)
ax.plot(date_time,x1)
ax.stem(date_time,x2,'C1--','C1o',linefmt=None, markerfmt=None, basefmt=None)
ax.stem(date_time,x3,'C2--','C2o',linefmt=None, markerfmt=None, basefmt=None)
ax.legend()
ax.xaxis_date()
ax.get_xaxis().set_major_formatter(DateFormatter('%d/%m/%Y %H:%M:%S'))
plt.xlabel('t')
plt.ylabel('k')
leg = plt.legend( loc = 'upper right')
plt.draw() # Draw the figure so you can find the positon of the legend.
bb = leg.get_bbox_to_anchor().inverse_transformed(ax.transAxes)
xOffset = 0.3
bb.x0 += xOffset
bb.x1 += xOffset
leg.set_bbox_to_anchor(bb, transform = ax.transAxes)
plt.rcParams["figure.figsize"] = [20,20]
ax.plot(style='.-')
plt.show()

error :


texts: Clarification on K Means Clustering Algorithm and its Analysis

texts:
I fail to understand how the circled point is assigned to cluster c2 and not c1. From what I have understood, the points are assigned to the (nearest?) centroid in order to minimise the squared distances i.e. to minimise the assignment cost.

Visually too, it appears to me that the circled point belongs to the first cluster.

Please correct me if I'm wrong.

Source - http://www.cs.yale.edu/homes/el327/papers/OnlineKMeansAlenexEdoLiberty.pdf


texts: Elbow method on hundreds columns and rows

texts:
So I have these vectors called matrix_ after I applied TF-IDF (term frequency-inverse document frequency), and I also converted it to dataframe matrixDF_ .

I wanted to cluster these matrix_ vectors with MiniBatchKMeans from sklearn. But since the vectors are large (569 rows and 829 columns), then I need to find the optimum K-Cluster first.

That is why I use KElbowVisualizer from Yellowbrick. At first, it works fine and I got the optimum k-cluster. However, when rerunning it again, it showed me a different number of k-cluster. And that is where I am confused then.

Anyone can help me with this problem?

here is my code:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import MiniBatchKMeans
from scipy.spatial.distance import cdist
from sklearn.metrics import silhouette_score
from yellowbrick.cluster import KElbowVisualizer

""" vectorize the words. getting the tf-idf of each words """
vec = TfidfVectorizer(
     lowercase = True,
     stop_words = 'english',
     use_idf=True
)

matrix_ = vec.fit_transform(df)
matrixDF_ = pd.DataFrame(matrix_.toarray(), columns=vec.get_feature_names())

model = MiniBatchKMeans(MiniBatchKMeans(init_size=559, random_state=None))
visualizer = KElbowVisualizer(model,k=(2, 11), metric='distortion',
timings=True, size=(1080, 720))
visualizer.fit(matrix_)        # Fit the data to the visualizer
visualizer.show()

I also tried fit the dataframe matrixDF_. to the visualizer, but each output after every run has different output number.

Here are some of output examples. First output:

second output:


texts:
I am trying to create a comparison group. So far this group contains 45 data points and I need to populate the remaining 55 (for a total of 100 data points).

These remaining 55 need to be a randomized selection supplied from a larger data set. Any recommendations for R code that would create a randomization loop?


texts: Does white balance correction improve the performance of visual object detection?

texts:
Visual object detecion aims to locate and recognize objects of interest in images.

I wonder, does white balance correction contribute to better performance of visual object detection? Are there any related studies or papers?

Thanks in advance for your suggestions!


texts: How to recognize a two part term when the space is removed? ("bigdata" and "big data")

texts:
I'm not a NLP guy and I have this question.

I have a text dataset containing terms which go like, "big data" and "bigdata".

For my purpose both of them are the same.

How can I detect them in NLTK (Python)?

Or any other NLP module in Python?


texts: Negative correlation between OOB statistics and test set statistics during tuning of a RandomForest

texts:
I am tuning the parameters of a binary random forest classifier using a random search with a priority queue for training. After training with a fixed number of estimators (3000), the strategy is to reduce priority (and eventually pause) of configurations with performances lower than the median of all the other running configurations with the same number of estimators, while I keep adding new estimators to the best performing configurations, until a total number of estimators is reached.
The maximum number of estimators is used to constraint computational requirements and the scheduling is used as a means to invest computational resources in the most promising configurations earlier.
As the performance criterion I use the average precision score computed on the OOB predictions, I also check the average precision of the trained models on a held out test set.
I have run this setup several times, but every single time the OOB score and the test score are negatively correlated, meaning that the better a model performs on the OOB, the worse performs on a held out set. The thing is also the opposite is true!!!

In the picture, the color is according to the OOB average precision.
I thought that this phenomenon would be caused by an unfortunate train/test split, but even with a random split for each model this still happens.
I also run a smaller training experiment in which I used a 10-fold cross-validated average precision score instead of oob, but it is still negatively correlated to the test statistic.
What do you think could be the cause of this behaviour?


texts: How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

texts:
I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for calculation of cost function change?

I am grateful for every good reply I can get, have a nice day! :)


texts: How does Pytorch deal with non-differentiable activation functions during backprop?

texts:
I've read many posts on how Pytorch  deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation functions during backprop. However I was not able to come up with a full picture as to what exactly happens.
Most answers deal with ReLU $\max(0,1)$ and claims that the derivative at $0$ is either taken to be $0$ or $1$ by convention (not sure which one).
But there are many other activation functions with multiple points of non-differentiability.

2 points

4 points
How does Pytorch systematically deal with all these points during backprop? Does anyone have an authoritative answer?


texts: Does increasing kernel size in a CNN result in higher accuracy on the training set?

texts:
In a convolutional neural network, does increasing the size of kernel always result in better training set accuracy? For example, if I use 5x5 kernels in a CNN instead of 3x3 ones, will it always generate better training accuracy?

Increasing kernel size means effectively increasing the total number of parameters. So, it is expected that the model has a higher complexity to address a given problem. So it should perform better at least for a particular training set. Or will it be harder to learn a bigger kernel?

I am not concerned about the validation set accuracy here.


texts:
I'm going to use ( RNN+Logisitic Regression ) to make sentiment analysis.

Should I do preprocessing for text like remove stop words, punctuation and extract keywords by found nouns ?


texts: dealing with HuggingFace's model's tokens

texts:
I have a few questions regarding tokenizing word/characters/emojis for different huggingface models.
From my understanding, a model would only perform best during inference if the token of the input sentence are within the tokens that the model’s tokenizer was trained on.
My questions are:

is there a way to easily find out if a particular word/emoji is compatible (included during model training) with the model? (in huggingface context)

if this word/emoji is not was not included during model training, what are the best ways to deal with these words/emojis, such that model inference would give best possible output considering the inclusion of these word/emoji as input. (for 2. it would be nice if it could be answered in the context of my huggingface setup below, if possible)

My current setup is as follows:
from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
pre_trained_model = 'facebook/bart-large-mnli'
task = 'zero-shot-classification'
candidate_labels = ['happy', 'sad', 'angry', 'confused']
tokenizer = AutoTokenizer.from_pretrained(pre_trained_model)
model = AutoModelForSequenceClassification.from_pretrained(pre_trained_model)
zero_shot_classifier = pipeline(model=model, tokenizer=tokenizer, task=task)

zero_shot_classifier('today is a good day ', candidate_labels=candidate_labels)

Any help is appreciated


texts:
I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan.

Most often single models are visualized with a decision tree or transition state diagram. However, with multiple different models what are potential visualizations that could communicate the transition states that differentiate each model?

I have seen other people use a table to depict different models and the transition states.

For clarity, I am not referring to a transition probabilities chart but a method of communicating the differences between multiple models.


texts: Counting similarity between two GIT commits

texts: I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a similarities of changes made through two different commits. I can gather a commit message of these commits and differences in the source code. However, I don't know how good to count such similarity. I am considering to put message and text differences from source code into DBPedia Spotlight and count cosine similarity from the result annotations. Would you have some better idea of how to proceed it?


texts:
There is a two class dataset with 1121 values in total, having 230 from same class and 891 from the other class. The training set is choosen as 230+230=460 from both classes and the test set as the entire 1121 data.
1)Accuracy values are less than 0,50 even some are as low as 0,18 and 0,20. Does this make sense? For a two class outcome, there is more chance for an accurate prediction if I toss a coin. Can there be an accuracy of less than 0.50 for a two class prediction?
2)When both test-train set is choosen from the 460 class balanced rows and k-fold(1:10) is made, the accuracy levels are considerably higher, up to 0,90.
3)Can the difference between the results be because the test set is much larger than the train set?


texts: NLP: Mapping Penn treebank and Brown corpus, to Universal PoS Tags

texts:
I am experimenting with NLP and PoS tagging. I wish to build a large corpus, composed of Penn Treebank and Brown corpus, and possibly even more. Unfortunately, their PoS tags are not compatible.

Is there a way to map their tags to universal tagging? (Or any other tagging?)


texts: Experience replay in Reinforcement learning - Batch Size

texts:
I am starting to understand Reinforcement Learning playing with the GYM Cart-pole environment. I would like to ask if the experience replay can slow down the training and if there is a maximum of the batch size?

Maybe I misunderstood, and when the program starts the fit function on the replay I guess it is not sending the action to the agent, but the environment is still running. Not sure if I missed a point or if the fit function run so quickly that the agent will have plenty of time to catch up with the environment for the next action during training.


texts: can u please give an explanation for this ?


texts: What are real world applications of Doc2Vec?

texts:
I am new to Doc2Vec. As I understand Doc2Vec group similar documents based on the context of their words.

I have a set of newspaper documents and I want to identify what are the main topics of the newspapers (group 'Politics' news documents to one group, 'Sports' news documents to another group etc.) based on their content. Thus, I am interested in knowing;

What variant of Doc2Vec is more suitable for this (dbow, dm)?
What are the real world applications of Doc2Vec algorithm?


texts: Why is my Siamese network always predicting 1?

texts:
There is no change to loss and the accuracy stays the same.

left_input = Input((3,224,224))
right_input = Input((3,224,224))
encoded_l = encoder2(left_input)
encoded_r = encoder2(right_input)

# Getting the L1 Distance between the 2 encodings
L1_layer = L.Lambda(lambda tensor:K.abs(tensor[0] - tensor[1]))

L1_distance = L1_layer([encoded_l, encoded_r])
prediction = L.Dense(1,activation='sigmoid')(L1_distance)
siamese_net = Model(inputs=[left_input,right_input],outputs=prediction)
optimizer = Adam(0.0005, decay=2.5e-4)
siamese_net.compile(loss="binary_crossentropy",optimizer=optimizer,metrics=['accuracy'])

siamese_net.fit([Xl_train.reshape((-1,3,224,224)),Xr_train.reshape((-1,3,224,224))], to_categorical(y_train),
          batch_size=32,
          epochs=50,
          verbose=1,
          validation_data=([Xl_test.reshape((-1,3,224,224)),Xr_test.reshape((-1,3,224,224))],to_categorical(y_test)),
          shuffle=True)

I only have positive pairs set so I had to make the negative pairs set by using np.roll() on one set of members in the pair to produce a new combination. So it isn't predicting 1 because of the imbalance problem(since both positive and negative samples have the same number of pairs). I have tried shuffling the dataset, using softmax with 2 nodes in the final layer. I tried it on SVM with rbf kernel. While the accuracy was pretty bad, it was still at least predicting 0.
And here's the code for the encoder-

def encoder():
    model=Sequential()
    model.add(Convolution2D(128, (7, 7), activation='relu'))
    model.add(Dropout(0.2))
    model.add(Convolution2D(64, (5, 5), activation='relu'))
    model.add(Dropout(0.2))
    model.add(Convolution2D(64, (5, 5), activation='relu'))
    model.add(Dropout(0.2))
    model.add(Convolution2D(16, (5, 5), activation='relu'))
    model.add(Dropout(0.2))
    model.add(Convolution2D(1, (5, 5), activation='relu'))
    model.add(Flatten())
    model.build((None,3, 224, 224))

    #print(model.summary())
    return model

Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 128, 218, 218)     18944
_________________________________________________________________
dropout_1 (Dropout)          (None, 128, 218, 218)     0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 64, 214, 214)      204864
_________________________________________________________________
dropout_2 (Dropout)          (None, 64, 214, 214)      0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 64, 210, 210)      102464
_________________________________________________________________
dropout_3 (Dropout)          (None, 64, 210, 210)      0
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 16, 206, 206)      25616
_________________________________________________________________
dropout_4 (Dropout)          (None, 16, 206, 206)      0
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 1, 202, 202)       401
_________________________________________________________________
flatten_1 (Flatten)          (None, 40804)             0
=================================================================
Total params: 352,289
Trainable params: 352,289
Non-trainable params: 0

EDIT 1- I added-
kernel_initializer='random_uniform',bias_initializer='zeros'

to all the layers so that it would stop predicting everything as 0. The training losses are decreasing now but the validation loss and accuracy aren't being changed.

Edit 2- Resolved it. It was an issue with my dataset.


texts: Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

texts:
I have a sample data of around 10 ATM's Locations along with their Utilization Count (Deposits, Withdrawals and Others) for the past 3 months. I am planning to collect additional data such as nearby places of Commercial Interest and Others where there might be demand of Cash. The data is collected for approximately 300 meters of each ATM, i.e., places of Commercial Interest nearby the ATM.

I would like to build a 'Scoring/Rank Model' which can take all these inputs into consideration and Rank these 10 ATM based on Utilization and places of economic interest.

As much I have searched Google, I could not find any answers which can provide a solution. I am pretty new to Data Science and any inputs/suggestion will be really valuable.


texts: How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries

texts:
Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow)  30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i have circled in black where "malicious" pattern should be.
Interactions between the dimensions exist, therefore dimensions cannot be studied individually (or can they?)

I'm trying to build a supervised ML model using Scikit Learn which learns a normal rythm, and detects when symptoms might lead to a spike. I am lost for which direction to take. I have tried Anomaly detection, but it only works for on the spot detection, not prior.

How could I detect "malicious" patterns prior to those events (taking them as target variables) ?

I welcome any advice on which algorithms or data processing pipeline might help, thank you :)


texts: Time series analysis vs linear regression

texts:
I am working on developing an algorithm which will predict the future traffic for the restaurant. I am confuse that which of the two: Linear regression or time series analysis I should use as the base for my algorithm. The features I am using are: Day,whether there was festival,temperature,climatic condition , current rating,whether there was holiday,service rating,number of reviews etc.

Please guide me how should I proceed . Also how can I optimize my algorithm so that it can learn with time.


texts: How do I interpret my result of clustering?

texts:
I am working on a clustering problem. I have 11 features. My complete data frame has 70-80% zeros. The data had outliers that I capped at 0.5 and 0.95 percentile. However, I tried k-means (python) on data and received a very unusual cluster that looks like a cuboid. I am not sure if this result is really a cluster or has something gone wrong?

The main reason for my worry, why is it looking like a cuboid and why are the axes orthogonal?

one thing to notice is that: I first reduced the dimensionality using PCA to two dimensions and performed clustering on the same and the plot here is on the 2-dim PCA data

Edit : I chose k using silhouette index in python.


texts: Interpret clustering results after variable transformation

texts:
since some time I have a question to which I have not found the proper answer yet.

My doubt concerns the interpretation of the results of a clustering algorithm which was run on features to which a log-transformation was applied.

Specifically, let's assume we want to run a k-means algorithm on 3 interval variables. Unfortunately, these three interval variables are extremely bad distributed and the k-means gives the worst result we have ever seen.
However, let's imagine that by applying a log transformation to each variable, we obtain three incredibly perfect normal distribution.
Then, we run again the k-means and we obtain perfect clusters.

Now, my doubt concerns the interpretation of this cluster obtained by running a k-means on three log-transformed variables: it is not clear whether our interpretation of the clusters obtained should be made on the original variables or it should be made on the log-transformed variables?

Clearly, my example is related to log-transformation but we can talk about z-score or min-max normalization or any other kind of transformation that we apply in order to improve the quality distribution before running the clustering algorithm.

To clarify, what I mean by interpretation is the profiling of the cluster, which means try to describe which are the characteristics common to the individuals belonging to that cluster.


texts: How to analyze the abnormal trajectory of the vehicle?

texts:
I use this code to mark the latitude and longitude of the vehicle with red points on the map:
import pandas as pd
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt

def testPlot():
    df = pd.read_csv(r'device_data.csv', encoding='utf-8')
    gb = df.groupby(['device_id', 'upload_time_1'])
    sub_dataframe_list = []
    for i in gb.indices:
        sub_df = pd.DataFrame(gb.get_group(i))
        sub_dataframe_list.append(sub_df)
    for sub_dataframe in sub_dataframe_list:
        device_id = sub_dataframe['device_id'].iloc[0]
        upload_time_1 = sub_dataframe['upload_time_1'].iloc[0]
        sub_dataframe = sub_dataframe.sort_values(by=['upload_time'])
        fig = plt.figure(figsize=(20, 10))
        m = Basemap(llcrnrlon=77, llcrnrlat=14, urcrnrlon=140, urcrnrlat=51, projection='lcc', lat_1=33, lat_2=45,lon_0=100)
        m.readshapefile(r'E:/test_opencv/gadm36_CHN_shp/gadm36_CHN_1', 'states', drawbounds=True)
        x = sub_dataframe['longitude'].tolist()
        y = sub_dataframe['latitude'].tolist()
        lats = y
        lons = x
        m.drawcountries(color='#ffffff', linewidth=0.5)
        m.fillcontinents(color='#c0c0c0', lake_color='#ffffff')
        x, y = m(lons, lats)
        plt.plot(x, y, 'bo', color='r', markersize=1)
        plt.savefig(r'all_device_data_image/'+str(device_id)+'_'+str(upload_time_1)+'.png')
        plt.close()

if __name__ == '__main__':
    testPlot()

The normal vehicle trajectory is like this, the latitude and longitude are continuous：

But the abnormal vehicle trajectory is like this, the latitude and longitude are not continuous, it is very sparse:

If the abnormal trajectory is supplemented with green latitude and longitude, it is a normal trajectory, like this:

For my question, how can I judge the vehicle trajectory is abnormal? Statistics or machine learning or deep learning can be used, but I don't know which algorithm can do it. If I don’t use statistics or machine learning or deep learning, what should I do? Determine the average and standard deviation of latitude and longitude?
I think it is not a good idea to use the average and standard deviation of latitude and longitude.


texts: Implementing spatio-temporal convolutions in pytorch

texts:
I am trying to implement a layer to perform the (2+1)D convolutions described in this paper: https://arxiv.org/pdf/1711.11248.pdf

The basic idea is as follows: Let's say I have a 3D convolutional layer that takes in an input with $N_{i-1}$ channels, and performs $N_i$ 3D convolutions on a 3D tensor representing $T$ 2D frames taken over $T$ timesteps.

Instead of doing 3D convolutions over the tensor, the approach is to replace the 3D convolution with a 2D convolution followed by a 1D convolution along the temporal axis. In particular, if you wanted to perform $N_i$ 3D convolutions with kernels of size $N_{i-1} \times t \times d \times d$, you instead perform $M_i$ 2D convolutions with filters of size $N_{i-1} \times 1 \times d \times d$ followed by $N_i$ 1D convolutions along the temporal axis of size $M_i \times t \times 1 \times 1$.

Here is the tensorflow implementation of this type of layer: https://github.com/facebookresearch/R2Plus1D/blob/master/lib/models/video_model.py

I am having trouble implementing this type of layer in pytorch, however. This is what I have right now, and I'm clearly doing something wrong. I would appreciate it if someone could point me in the right direction as to how I would go about performing this type of convolution. I'm setting the intermediate channel dimension $M_i$ to 20 in this snippet:

class hybrid3d(nn.Module):

def __init__(self, n_classes):
    super(hybrid3d, self).__init__()
    t, d, d = 3, 3, 3
    self.conv1_1 = nn.Conv2d(3, 20, kernel_size=(1, d, d), stride=[1,1,1] , padding=[0,0,0])
    self.relu1_1 = nn.PReLU()
    self.conv1_2 = nn.Conv1d(20, 12, kernel_size=(t, 1, 1), stride=[1,1,1], padding=[0,0,0])
    self.relu1_2 = nn.PReLU()

def forward(self, x):
    x = self.relu1_1(self.conv1_1(x))
    x = self.relu1_2(self.conv1_2(x))
    return x


texts: What model should i use to extract relation between words

texts:
I want to create a ML model which would give a score from 0 to 1 which would signify the relation between them.
I know about Relationship Extraction(RE) but that's more related with sentences based relation. Instead i want to input two words and that should output the relation between them and a input dataset being a lot of sentences.


texts:
I'm trying to use LSTM (with Keras) for a time series problem. I would like predict the next value of the time series given its previous value. I'm using TimeseriesGenerator to create the training data as follows (setting length equal to 1 indicating the prediction is based on the previous value):

generator = TimeseriesGenerator(series, series, length=1, batch_size=10)

For the modeling I'm using the following:

model = Sequential()

model.add(LSTM(num_units, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')

model.fit_generator(generator, steps_per_epoch=len(generator)/n_batch, epochs=50)

for prediction:

yhat = model.predict(x_input, verbose=0)

when I set num_units = 1 (in the the first layer), the predicted values are much lower than the typical values in the time series (the typical values in the time series are 30-50 and the prediction is around 0.4). However, when I set num_units = 700, the predicted values become very close to the test values and the predictions seem to be overfitting. Why is that? What does the num_units in the first layer intuitively represent? If our input data is just one number, what does it mean to have a layer with 700 units? What is the intuition behind mapping a number to 700 neurons and what does one gain from it? In general, if in our time series, we're trying to make a prediction from the past n observations, how many units should be in the first LSTM layer?


texts:
I want to improve my current application. I am using redis using ElastiCache in AWS in order to store some user data from my website.

This solution is not scalable and I want to scale it using Amazon Kinesis Data Firehose for the autoscale streaming output, AWS Lambda to modify the output of my logs.

I want to visualise  the data in near real time and here is where my doubt is:

If I use s3 to store my straming data and I use aws athena and a use a web-based applications for visualise this data in near real time, aws is going to charge me every time that the application will update the new log data?

A best solution could be use AWS elastic search and do it with Kibana?


texts:
I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores).

Any suggestions for a good book/blog/any source where I could learn about this. The challenges I see are - the sample size is usually one or 2 stores for the test so how do you do statistically significant testing with such a small sample; test stores or control stores are matched 1-1 but often not selected randomly. Any guidance in learning about the data science in this area is greatly appreciated.

Thanks!!


texts: How to visualize segmentation result with CrossEntropyLoss

texts:
I know this question already has a clear answer about how to visualize segmentation result. My question is if I'm using the CrossEntropyLoss as the loss function, after training. The predicted classification (segmentation) result doesn't represent the class 0,1,2. It represents the unnormalized Log probability as told in cs231n standford class.
So if I want to visualize the segmentaion result, should I use the

- nn.LogSoftmax()

to get the classes and then plot them out?
or should I do a maximum function over the first dimension of my output.

For example, if the predicted output is size 3x1024x1024, how could I visualize the segmenation result?

Thank you in advance.


texts: What is the intuitive meaning of "leaf weight" in xgboost

texts:
I looked through Tianqi Chen's presentation, but I'm struggling to understand the details of what the leaf weights are,  and I would appreciate if someone could help clarify my understanding.

To put the equations into words on the slide "Put into context: Model and Parameters", the predicted value/score (denoted as yhat) is equal to a sum of the K trees of the model, which each maps the attributes to scores. So far so good, I think.

Then in this slide shown below it gives a mock example of a decision tree that calculates the amount that someone likes a computer game X.  (Aside: Isn't this a weird example? Who likes a computer game X by an amount 2? What does that even mean? Why not choose an example that has a more concrete, relatable meaning?)

Now this is where I start to get lost. I can tabulate mock data for the mock example, assuming that the model is perfect so that the weights (w1, w2, w3) are equal to the true value. But even that seems weird: what's the difference between the predicted value/score and the weights?

       x_i: "attributes"          y_i, the true score (not yhat_i, which is the predicted score)
                |                                 |
|¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯|¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯|
| Age <15 (x_0) | is male (x_1) | Amount likes the computer game X  |
| 1             | 1             | 2                                 | (Young boy)
| 1             | 0             | 0.1                               | (Young girl)
| 0             | 1             | -1                                | (Old man)
| 0             | 0             | -1                                | (Old woman)
| 0             | 0             | -1                                | (Young woman, older than 15)

My question is: Can someone please share what would be the ACTUAL function f? I assume it's a vector/matrix, but what are the actual numbers? Then my bonus follow up is how would you compute the f for this mock example? I feel it's such a simple question, but I can't figure out the answer. If someone could break this down for me in nitty gritty detail it would be a huge help. Thanks!


texts: Understanding ROCs in imbalanced data-sets

texts:
A response variable (label) $B$ can either be $0$ or $1$.

In the training set, $B_i = 1$ is an extremely rare event at only $0.26\%$ occurrences. Which makes the prediction of this label on a test data-set a difficult problem.

I used SMOTE to sample from the training-set of some $1.55 \times 10^5$ rows to obtain a completely balanced set of $620$ rows.

balanced.df <- SMOTE(B ~ ., df, perc.over = 100, perc.under = 200)
randomForest with $1000$ trees was used to fit the model, as shown:

randomForest.Fit <- randomForest(B ~ ., data = balanced.df, ntree = 1000)
For making a validation set, $2000$ rows were sampled at random without repetition from the data-set.

The actual frequencies of $B_i$ in the validation set are:

   0    1
 1998   2

And those in the predicted set are:

   0    1
 1836  164

The results seem promising, but perhaps a little too much. Also, it is essential that the percentage of False Positives are reduced.

My questions are:

How severely do you think the skew in data is affecting the validation results?
Is there a point in validating again by creating an arbitrary data-set with bias, for example by selecting more $B_i = 1$ in the validation set?
What other metrics/validation techniques which reflect the "accuracy*" of prediction?

*The term accuracy is used in a generic sense.


texts: What to do when there is more than one object in a cell in Yolo v1

texts:
I'm trying to implement Yolo v1 and I'm kinda stuck with its loss function.

I understand the need of multiple bounding boxes per cell although only one object is detected per cell. Each bounding box can be specialized in some aspect ratios and sizes.

I understand also that when you calculate the loss function, you choose the bounding box that best fits (higher iou) the ground truth object in that cell.

What I don't understand is how do you calculate the loss function when there is actually more than one object in a cell. The loss function doesn't specify it and assumes that one (and only one) object may or may not be in a specific cell.


texts: Building Timeseries models for stock trading having multiple stocks

texts: I have gone through some of the tutorials on the timeseries and all of them have taken one stock for the timeseries and tried to forecast it. My dataset contains many stocks for the time period(each day). Do i need to choose one stock for or some of them or all of them for training the model?


texts: When does decision tree perform better than the neural network?

texts:
I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree works the best with training accuracy of 1.0 and test accuracy of 0.5. The neural networks, which I believed would always perform the best no matter what has a training accuracy of 0.92 and test accuracy of 0.42 which is 8% less than the decision tree classifier.

Could someone please explain the circumstances/cases where neural networks could have low accuracy when compared to a modelling technique like the decision tree. I had tried my neural network with different configurations like:

1 hidden layer and 1 neuron : Train Accuracy 34% Test Accuracy 42%
7 hidden layers and 5 neurons in each layer: Train Accuracy 79% Test Accuracy 42%
1 hidden layer and 100 neurons: Train Accuracy 34% and Test Accuracy 35%

but not in a single case, I found the neural network to beat the decision tree test accuracy of 50%.


texts: What should be the input shape for convLSTM if ResNet-50 is applied?

texts:
I have a dataset 12 videos. Each video is comprised of 179 frames. On these frames, I have applied ResNet-50 to extract features, and I received (179,7,7,2048) features. As far I know,

179=Total number of frames
2048=total number of features generated from a frame
7*7=kernel size / Filter size

Now I have to train my model using convLSTM by passing the features extracted through ResNet-50. And I know that input shape for convLSTM is

batch_shape + (channels, conv_dim1, conv_dim2, conv_dim3)
OR
batch_shape + (conv_dim1, conv_dim2, conv_dim3, channels)

So what should be the input shape for convLSTM and how can I apply the output of ResNet-50 to the convLSTM?
Regard