texts: Necessity of balancing positive/negative examples in binary classification machine learning?

texts:
When using machine learning models like gradient boosted trees and CNN, is it required (or considered as an always-do good practice) to balance the amount of positive/negative examples when learning for binary classification?

Given P positive examples and N negative examples, where P << N, I can think of several choices: (Let's forget about validation set and test set)

Choice A) No balancing at all, put all examples (totally P+N) into the training set without weighting w.r.t. their ratio.

Choice B) Put all examples (totally P+N) into the training set, but weight all positive examples 1/2P and all negative examples 1/2N, so that total weight of positive examples and negative example equal.

Choice C) Take all P positive examples, then sample P negative examples (out of N), and train with these 2P examples with uniform weighting.

What are the pros/cons for each of the approach and which one(s) do we usually go with?


texts: How to benefit Data augmentation when it yields to different classes

texts:
I'm trying to classify rooftop sky images orientations, whether it is horizontal or vertical. Knowing that the most obvious feature here is known: orientation. I can simply augment each class by rotating it 90° so it belongs to the other class. It is like dynamic augmentation to different classes.

Using keras image classification capabilities, obviously defeats the purpose:

datagen = ImageDataGenerator(
        zoom_range=0.2, # randomly zoom into images
#       rotation_range=90,  # mistaken the model
        horizontal_flip=True,  # randomly flip images
        vertical_flip=False)  # randomly flip images

I am using keras for classification but open to other libraries, and techniques other than deep learning.


texts: Website navigation path using graph theory

texts:
In web analytics, one of the main issues is to find the optimal path of navigation. Question: Is there any literature in graph theory area, which is focused on studying optimal website navigation path?

My Comment to Tom was pretty large. So putting it here:

But in network theory there's a concept of main node (forgot the exact terminology). This node is most important because it has most connection to other pages. Home page is a very simple kind of main node. Also different set of visitors might have different main node. So, if we could capture what is the main node, then if we directly drop the visitor to this page or some other page, then the conversion might increase. Because the visitor who will come to this main page, will able to easily navigate to his required page. And then able to get what he is looking for. Another method people use for shortest path is MCMC methodology. But I'll prefer network method. Figuring out the entire method is pretty cumbersome and I don't have much time. So I was wondering if somebody wrote some journal paper on it, I can refer that.


texts: Application of validated logistic regression model on new data

texts: I have built a logistic regression algorithm in R and have trained it and tested it and now I want to apply it on new incoming data. My only question is that whether I always have to keep my training set data in the csv I want to get output of. If I am removing training set data, I am getting all values as NA


texts: How to get same accuracy with identical models in Keras and Tensorflow?

texts:
As we all know Keras backend uses Tensorflow and so it should give out same kind of results when we provide same parameters, hyper-parameters, weights and biases initialisation at each layer, but still the accuracy is different.

This maybe because of the batches of images which are fed at each step in both the models are not identical and as it gets shuffled randomly.

Is there any way in which we can make sure that the same batch of images are fed into the model while eliminating the randomness?

I have tried using all the same parameters, hyperparameters, same weights and biases initialization with seed values.

The accuracy of both the models are not same.


texts: What is the difference between an RMSE and RMSLE (logarithmic error)?

texts:
RMSE vs RMSLE

Root Mean Squared Error (RMSE) and Root Mean Squared Logarithmic Error (RMSLE) both are the techniques to find out the difference between the values predicted by the machine learning model and the actual values.

  But, what is the purpose for RMSLE( "logarithmic")
  Does a high RMSE imply low RMSLE?

Can somebody explain in-detailed differences between RMSE and RMSLE? And how the metric works under the hood?

  When would one use RMSE over RMSLE?
  What are the advantages/disadvantages of using RMSE over RMSLE?


texts:
Whether correct or not, I'm not able to judge being myself in the early days of the Data Science.

However, I have applied a Kmeans on a corpus where some random documents (very short sentences) have been added.
These have been vectiorized so to be suitable.

With clusterization results at hands, I was somehow expecting the vectors (keyword) to fall only in one cluster at a time (and no more than that).
This is not the case.

In some circumstances, I have a vector falling in two clusters and I wonder why this is the case.

Is this because of the inappropriate usage of Kmeans on vectors made from documents?
Is this normal as the way Kmeans works (moving the centroids, but de facto assigning objects to the nearest cluster by distance)?
Is this overlap due to the fact that in analysing my results I assess the whole group of items within a cluster and not just (say) the top X near to the center?

--
Example:

corpus = [
'The car is driven on the road.',
'The truck is driven on the highway.',
'The train run on the tracks.',
'The bycicle is run on the pavement.',
'The flight is conducted in the air.',
'The baloon is conducted in the air.',
'The bird is flying in the air.',
'The man is walking in the street.',
'The pedestrian is crossing the zebra.',
'The pilot flights the plane].',
'On the route, the car is driven.',
'On the road, the truck is moved.',
'The train is running on the tracks.',
'The bike is running on the pavement.',
'The flight takes place in the sky.',
'Birds don''t fly when is dark',
'The baloon is in the water.',
'The bird flies in the sky.',
'In the road, the guy walks.',
'The pedestrian is passing through the zebra.',
'The pilot is flying the plane.',
'This is a Japanese doll.',
'I really want to go to work, but I am too sick to drive.',
'Christmas is coming.',
'With the daylight saving time turned off it''s getting dark soon.',
'The body fat may compensates for the loss of nutrients.',
'Mary plays the piano.',
'She always speaks to him in a loud voice.',
'Wow, does that work?',
'I don''t like walking when it is dark',
'Last Friday in three week’s time I saw a spotted striped blue worm shake hands with a legless lizard.',
'My Mum tries to be cool by saying that she likes all the same things that I do.',
'Mummy is saying that she loves me being a pilot when in reality she is scared all the time I take off.',
'Where do random thoughts come from?',
'A glittering gem is not enough.',
'We need to rent a room for our party.',
'A purple pig and a green donkey flew a kite in the middle of the night and ended up sunburnt.',
'If I don’t like something, I’ll stay away from it.',
'The body may perhaps compensates for the loss of a true metaphysics.',
'Don''t step on the broken glass.',
'It was getting dark, and we weren''t there yet.',
'Playing an instrument like the guitar takes out the stress from my day.']

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(analyzer='word',
                         max_df=0.8,
                         max_features=50000,
                         lowercase=True
                        )

X = vectorizer.fit_transform(corpus)

from sklearn.cluster import KMeans

num_clusters = 11
kmean = KMeans(n_clusters=num_clusters, random_state=1021)
clusters = kmean.fit_predict(X)

--

If you explore the clusters variable, you will notice the overlaps I am talking about.
For instance the keyword baloon appeara in both cluster 10 and 0.

There are 12 overlaps, which on a 33 unique keywords dataset represents 1/3, so I won't say something I could be happy with.

Any advice is appreciated.
Thanks


texts: Why do most GAN (Generative Adversarial Network) implementations have symmetric discriminator and generator architectures?

texts:
For example, if the discriminator is a vanilla network of n layers, each with n(i) units, then, typically, the generator will also be a vanilla network of n layers, each with n(n-i) units (except the output of the discriminator, where n(n) = 1 whereas for the generator n(0) = NOISE_SIZE).

If the discriminator is a CNN, the generator is typically a symmetric "deconvolution network" where the i-th layer is a transposed convolution layer, symmetric to the n-i-th layer of the discriminator.

Virtually all implementations I've seen follow this pattern, although I can't see anything in the theory why it would have to be the case. And yet, I had a simple vanilla implementation of a digit drawing GAN trained on MNIST, which worked reasonably well. I tried to improve the discriminator by making it a CNN (with the same architecture which I had working well on recognizing MNIST digits) without changing the generator. The GAN no longer worked and converged to a state where the generator always produced the same gibberish drawing. Intuitively, a better discriminator should help the GAN but it was not the case. It seems that improving the discriminator without equally improving the generator does not work (and vice-versa probably). Is this the reason why people always choose symmetric architectures? To preserve a "balance of skills" between the adversaries? Or is there a deeper reason?


texts: Predicting % of demand going to each product

texts:
I work within an industry with products that expire, therefore we would like to be able to choose which specific marketing keywords we should switch on to drive demand to the products that are over-stocked (and likely to expire at the current setup).
Therefore we would like to be able to say, If we turned on Keyword X, we predict that Product A would get a% of the purchases, Product B would get b% of the purchases and so on as these vague keywords can span multiple products.
I've been reading online that a Multinomial Logistic Regression model may be the best choice to predict probabilities to each product for each marketing keyword or campaign however given we have hundreds of different products and thousands of different keywords I was wondering whether this is feasible?
I am new to these models and therefore would like a more informed opinion on whether this is the best route please.


texts: sklearn NearestNeighbors returning misaligned results

texts:
I'm using NearestNeighbors to do name matching and at a certain point the results become misaligned. My standardized list of names is 100s of millions.  My list of names coming in to be matched is considerably smaller but still could be in the 250k to 500k range.  After a certain point it appears the index begins to shift by 1 or more.

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf)
unique_org = set(names['VariationName'].values) # set used for increased performance
#matching query:
def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices

print('Getting nearest n...')
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list
print('Finding matches...')
matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

print('Building data frame...')
matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
print('Data frame built')

It appears that once my standardized list gets over 80k it begins to shift the results down.

The "messy name" of VITALI, ANGELO (has a comma)

  VITALI, ANGELO

The standardized list of names may include these(no comma)

  VITALI ANGELO
  SENSABLE TECHNOLOGIES INC

After running it through the above matching the result below shows that VITALI, ANGELO is a near perfect match to SENSABLE TECNOLOGIES INC, because the index has shifted down one...I think.

  0.00  SENSABLE TECHNOLOGIES INC   VITALI, ANGELO

Is it possible that the size or number of records is  exceeding that matrix limits and it somehow messes the indices?


texts: Is it ok to fill a pandas dataframe with NaN values?

texts:
Is it correct to fill a pandas dataframe with NaN values? In specific: if I have a dataframe with a user name and his age is it ok to fill the age column with int and NaN values.
Names  Age
Lisa   25
Jack   NaN
Tom    32

Later on I want to work with this dataframe, will I get any problems if I have NaN values in it?


texts: Hand-crafted decision tree inspired from learned decision tree

texts:
Goal of this question:
As I am the only 'machine learning guy' in our group, I wanted to get an outsiders view, that is a sanity check if what I am doing adheres at least to 'decent practices' in machine learning (I know its not best practices :) ).

Problem setup:
I am working on a classification task on biomedical signals (detection of hypertension from physiological signals other than blood pressure). Since I do not have too much high quality labeled data for training a powerful classifier (say a larger conv-net), currently my procedure is as follows:

Feature engineering (manually engineered features, mainly driven by (physiology) domain insights.
Train supervised learning classifiers, in particular tree based algorithms such as random forests and simple decision trees.

Now, since currently I do not have enough high quality labels and I need to ship some classifier soon (in addition as it is a medical application I really care about stability and to some degree interpretability), I thought I could go for a manually built expert system, that is some rule-based system (if-else) using the most relevant features according to the learned random forest or decision tree classifier. For the cutoffs on the features I can use the values learned by the single decision tree.
In addition, I would start from a learned decision tree and adapt it (as some learned splits are totally non-sense, i.e., we are in the overfitting regime).

Question:
Is this procedure fine as long as I test my (hand-crafted and machine learning inspired) classifier on unseen data? I'd be happy to hear your experiences in similar situations!

Thanks


texts: How to interpret results of a Clustering Heart Failure Dataset?

texts:
I am doing an analysis about this dataset: click
In this dataset there are 13 features, 12 of input and 1 is the target variable, called "DEATH_EVENT". I tried to predict the survival of the patients in this dataset, using the features.
Hoewever, now I was trying to do a cluster analysis to see if the patients are grouped in clusters.
This is the code I have written.
from sklearn.cluster import KMeans
Features = ['ejection_fraction','serum_creatinine'] #the highest correlated features with death_event
X = heart_data[Features]

wcss = []
for i in range(1, 11):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(X)
    wcss.append(kmeans.inertia_)
plt.plot(range(1, 11), wcss)
plt.title('Elbow Method')
plt.xlabel('Number of clusters')
plt.ylabel('WCSS')
plt.show()

From this graph I can observe that there are 2 clusters. Now
kmeans = KMeans(n_clusters=2, init='k-means++', max_iter=300, n_init=10, random_state=0)
pred_y = kmeans.fit_predict(X)
plt.scatter(X["ejection_fraction"], X["serum_creatinine"])
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=300, c='red')
plt.show()

And I obtained this chart:

Now, what can I say from this chart? I think that it is unuseful, right?
I used only the features ejection_fraction and serum_creatinine because these are the only I used for the prediction. Or I have to use all the variables except from DEATH_EVENT? In this way:
X = heart_data.iloc[:, :11]

But in this case I obtain this:

I am not able to understand these charts, I think that I am doing something wrong, but what? Where are the clusters? How to interpret these results?
UPDATE: I am not able to use Umap_learn, my Mac can not install it, I received a lot of errors. However, I did something related to your advices.
Here there is all the code: https://pastebin.com/RdJb0ydu
The first 2 parts are the code that you have written here.
In the 3rd part I use kmeans(n_clusters=2) because from the silhouette I saw that the best was with 2 clusters. Then I did the prediction and concatenated the results to the original dataset and I printed out the column of DEATH_EVENT and the column with the results of clustering.
From this column, what can I say? How can I understand if the 0 of the prediction refers to the survived patients or to the died patients?


texts: Inconsistent validation accuracy? is that expected?

texts:
I am currently training a pattern classification network, and seem to get very inconsistent result.

The network seem to overfit,  but the validation accuracy is very inconsistent.

I am currently trying to do pattern recognition on audio spectrograms.
The audio consist of utterances of yes/no, and acoustic model (gmm-hmm) has been trained to detect these.  Given i already have a gmm-hmm, I can from the alignment decode which state obervation has the highest probability, and use that as output target for my neural network.

I am currently using different network structure such as Densenet, Resnet.

I once had a run using densenet which gave me a validation accuracy of 0.8, but this seem to only have happened once.  When i tries to train the network now i get validation scores around 0.3, which annoys me...

In general does it seem like the the validation score is very inconsistent,  it loop around 0.2 - 0.4, which while the system keep overfitting.

Is training really supposed to be so inconsistent?..


texts:
I'd like to train deep networks for images and with PCIe 4.0 hitting the market, I was curious if anyone has experienced the theoretical speed being actually utilized.
Thank you!


texts: Do we use the same threshold as in training when classifying using a linear classifier?

texts:
I've got a binary linear classifier and while training I am using 0 as my threshold. What I was wondering is when we change our threshold to be 1 let's say. So our function becomes,
$$
f(X) = \left\{
    \begin{array}{ll}
        1, & w^TX + b \geqslant 1 \\
        -1, & w^TX + b < 1
    \end{array}
\right.
$$
Do we use the same threshold for when classifying unseen samples or do we always use 0 for when classifying unseen samples?


texts:
Binary Classifier:

Assuming I have built a binary classifier and decided on an operating point. And made the classifier live on production data. This classifier returns the probability score for each of the two classes.

Is it safe to say, for production/out-of-sample instances where the classifier returns a probability score close to the operating point, the classifier is confused for those instances?

Moreover, if the I collect those instances where the classifier scores are close to the operating point.
And compare it with my training data. There could be two cases:

The "confusing" instances have significantly different distribution as opposed to the training data. In which case, my classifier is not faulty. And If possible I should manually label these instances and refit the classifier.
The "confusing" instances have similar distribution to my training data. In which case my classifier is at fault. What could be the implications of this case? But I believe this case would have been captured while training itself. Such instances would be very less in cardinality.

If the above mentioned thought process is correct. How can we extend this to multi-class classifier?


texts: Creating a new unique feature from existing features and using it instead in neural networks

texts:
Lets say I have 3 features a,b,c and lets say I am sure they vary between 0 to 100.

Instead of feeding my neural network 3 features what happens if I create a unique number from this 3 numbers something like 10000*a + 100*b + c = d and feed my neural network with d after some feature scaling?

I can see convolution is very common(which is different because it is not unique and more like a mean) that does not suits me because a,b,c are time series and their order matters.

Is there any advantage or disadvantage of combining multiple inputs into a single value in Neural networks?


texts: How to tune weights in Voting Classifier (Sklearn)

texts:
I am trying to do the following:
 vc = VotingClassifier(estimators=[('gbc', GradientBoostingClassifier()),
                                   ('rf', RandomForestClassifier()),
                                   ('svc', SVC(probability=True))],
                       voting='soft',
                       n_jobs=-1,
                       weights=[2, 3, 1])
cross_val_score(vc, X_new, y, n_jobs=-1)

In this, I want to tune the parameter weights. If I use GridSearchCV, it is taking a lot of time. Since it needs to fit the model for each iteration. Which is not required, I guess. Better would be use something like prefit used in SelectModelFrom function from sklearn.model_selection.
Is there any other option or I am misinterpreting something?


texts: Finding the inverse of a matrix with neural networks

texts:
I have a generic matrix A which is symmetric, positive definite and sparsely populated (it's also quite big, say composed of tens to hundreds thousands rows). I would like to have a neural network learn how to find a matrix B that resembles the inverse of A (ie: that multiplied by A will yeld a minimal spread of the spectral radius) as accurately and as fast as possible.

I was wondering if there are some AI branches that are focused on this kind of problems or if not how could I approach it.

I know the classical algorithms used to do this, but I would just like to have a look at the possibilities offered by machine learning on the subject.


texts: How to build a unique decision tree for each subset of data based on a grouping variable?

texts:
Given: a data frame of 2.5 million records and 25 columns, which contains 1 grouping column with a variable number of groups (e.g. 2.5M records / 10 groups today; 2.5M records / 50 groups tomorrow).

How can I build a decision tree for each group?

I was going to do this using Python but thought there might be an easier way in R (tapply?). Here was my initial thought process in Python:

instantiate an empty dictionary to hold the prediction results
identify the unique ids in the grouping column and iterate over the rows of data in a for loop
in each loop, build the decision tree, predict and store the results in the dictionary as a {grouping_var:prediction} pair

Thought on how to do this faster or more efficiently in R?


texts: New values for categorical variable in Prediction dataset

texts: I am performing regression task using *random forest**.In my prediction set I am having *new levels for categorical variable** which are not present in training data .Currently I am performing one hot encoding to handle this.At present I am not getting considerable results. This article is saying that it is not good to perform one hot encoding.How to handle this case?


texts: Technique/Algorithm for product categorization Machine Learning

texts:
I am going in a very abstract layer for the problem, but I think this problem might be a common one so posting it.

So, what I am looking for is any ML algorithm or Data Science technique is there to create a relationship factor between different sets of goods.

I guess, its used heavily in social media, but here I am talking w.r.t. product goods context. Like suppose if we sell a bunch of products together in a store, so in collection there will be many products that are always bought together.

So, accordingly there will be a relationship factor between different products i.e. some product that are always bought together will have a higher factor than the one that are bought together less frequently.

So, is there any approach where we can tackle this problem to get the factor between two products using any technique?

Edit : Narrowing it down to a universe where all items are having same factors and their relationship factor to each other totally depends on their occurance together.


texts: How to determine the VC Dimension of axis-aligned, origin-centered ellipses?

texts:
I've been trying to determine the VC dimension of ellipses which are origin centered and axis aligned. My first approach was to find some equivalence to a threshold classifier function family of the form

$$ H=\big\{\text{ sign}\big(\sum_{x}(Wx)+b\big)\ |\ w \in \mathbb{R}^d, b \in \mathbb{R}\ \big\}$$

For these functions I know the VC dimension is $d+1$, so for axis-aligned origin-centered ellipses I tried

$$ H=\big\{\text{ sign}\big(\sum_{x}(W^2x^2)-R^2\big)\ |\ w \in \mathbb{R}^d, R \in \mathbb{R}\ \big\} $$

I thought from here it would be straightforward but I couldn't figure it out. The limitations on the values that $w,x$ and $b$ (which equals $-R^2$) can get now can affect the dimension, but I'm not sure how.


texts: I am interested in anger detection in dialogues and I want to study multiple methods like LSTM, CNN, etc. Are there any good research papers or books about this subject?


texts: Do logistic regression and softmax regression do the same thing?

texts: If the both do the same thing then which give us better accuracy?


texts: Naming the Axes of my Graph on iPython Using Matplotlib.pyplot

texts:
I am new to data science. I use Anaconda on windows 7.
I plotted a sine curve by doing the following on iPython:

import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 10, 1000)
y = np.sin(x)
plt.plot(x, y)

And I got this:

But when I got ready to name the axes, the curve disappeared. I wrote this code:

plt.xlabel("Time")

And I got this:

I also wrote this:

plt.ylabel("Speed")

And got this:

So my question is, how can I plot a curve with labelled axes? (In other words, I will like the x-axis to be Time, the y-axis to be Speed and the curve intact)


texts: Predicting the variable's score using predictive analytics

texts:
I am new to predictive analytics but I am good at programming (like Spark, R).

I have 10 variables and I know their score's for over a long period of time. I want to know which variable will have high score in the future?

If it is a linear plot (plotted between score and time for a variable), we can extrapolate the values and find the max(future_score) among the 10 variables.

But the plot is non-linear (although continuous).

Any suggestions?

Thanks for your help?


texts:
I used logistic regression to classify 3 classes of images: cars, dogs, cows. The accuracy is 100%, perfectly separated in clusters (scatter plots).
When I try to use the learner to predict 3 images (1 dog, 1 car, 1 cow), the predictor simply assign the probability of 1/3 to each image. How can I fix it?


texts: Forecasting: Regression - Worse performance for a subset of data

texts:
I have a dataset for which I'm using different forecasting methods (Ridge/Lasso Regression / Random Forests / AdaBoost / Gradient Boosting) to compare their performance.
For the full dataset, I achieve some good results overall in terms of the MAPE and the R^2.
But when I apply the regression to a specific subset of the dataset, the performance is much worse.
I know it's impossible to tell the specific reasons for this without having access to the data but what are some general reasons that such a performance gap can exist?
Thanks in advance!


texts: how to do Time Based splitting of Amazon fine food reviews dataset

texts: I want to do time-based splitting on Amazon food reviews dataset (https://www.kaggle.com/snap/amazon-fine-food-reviews ). But I don't understand the time format and also how can I divide the data after it is sorted according to time


texts:
I have a set of documents where I need to extract important keywords in the document and then rank those keywords.

The ranking should be done based on relevance and/or other metrics. Are there any popular algorithms/methods/libraries that I can use for this task?


texts:
I am using Tensorflow.Lite.Support function for Inference of a model that takes two input and gives output in the form of Image. The first input is an RGB image whereas the second image is a single-channel image. When I run the application for inference I get the error that is:

Cannot convert between a TensorFlowLite buffer with 1392640 bytes and a Java Buffer with 4177920 bytes.

I have attached my code snippet below:

protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Toolbar toolbar = findViewById(R.id.toolbar);
        setSupportActionBar(toolbar);

        FloatingActionButton fab = findViewById(R.id.fab);
        fab.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {
                Snackbar.make(view, "Replace with your own action", Snackbar.LENGTH_LONG)
                        .setAction("Action", null).show();
            }
        });

        try {
            tflite = new Interpreter(loadModelFile(MainActivity.this, "converted_model.tflite"));

            Bitmap Image = getBitmapFromAsset(this,"001.png");
            Bitmap Mask = getBitmapFromAsset(this,"tmp_mask.png");

            int imageTensorIndex = 0;
            int[] imageShape = tflite.getInputTensor(imageTensorIndex).shape(); // {1, height, width, 3}
            int imageSizeY = imageShape[1];
            int imageSizeX = imageShape[2];
            DataType imageDataType = tflite.getInputTensor(imageTensorIndex).dataType();

            TensorImage inputImageBuffer = new TensorImage(imageDataType);
            inputImageBuffer.load(Image);

            int imageTensorIndex1 = 1;
            int[] imageShape1 = tflite.getInputTensor(imageTensorIndex1).shape(); // {1, height, width, 3}
            int imageSizeY1 = imageShape1[1];
            int imageSizeX1 = imageShape1[2];
            DataType imageDataType1 = tflite.getInputTensor(imageTensorIndex1).dataType();

            TensorImage inputImageBuffer1 = new TensorImage(imageDataType1);
            inputImageBuffer1.load(Mask);

            int OutputTensorIndex = 0;
            int[] OutputShape =
                    tflite.getOutputTensor(OutputTensorIndex).shape(); // {1, NUM_CLASSES}
            DataType OutputDataType = tflite.getOutputTensor(OutputTensorIndex).dataType();

            TensorBuffer outputBuffer = TensorBuffer.createFixedSize(OutputShape, OutputDataType);

            Object[] Inputs = {inputImageBuffer.getBuffer(),inputImageBuffer1.getBuffer()};

            Map<Integer, Object> outputs = new HashMap<>();
            outputs.put(0,outputBuffer);
            tflite.runForMultipleInputsOutputs(Inputs, outputs);

            Toast.makeText(this,"Working",Toast.LENGTH_LONG).show();

        } catch (IOException e) {
            Toast.makeText(this,"Failed",Toast.LENGTH_LONG).show();
            e.printStackTrace();
        }
    }


texts: Why does Bagging or Boosting algorithm give better accuracy than basic Algorithms in small datasets?

texts:
I was working with a small dataset, with 392 values, and it was kind of an imbalanced dataset, with 262 values belonging to class 1 and rest 130 to class 0. So I did the upsampling technique, importing sklearn.resampling module.

However, the total dataset was now around 520 values. I applied basic, algorithms first like Logistic Regression and SVM Classifier, and since we all know that precision is not a good accuracy metric for imbalanced dataset, I use the f1-score and recall score. In logistic Regression I found out, it was giving 78% f1-score for class 1 and 80% for class 0 , and almost 99% f1-score for class 0 in SVM and 72% for class 1, which shows that it is overfitting.

But to my surprise I found out that Random Forest gave me a better accuracy, with having around 83% f1-score for class 0 and 82% for class 1 . But till now everywhere I have seen that for bagging and boosting algorithms to work well, we need a lot of data, which is not the case in this scenario.

I've searched google a lot, but unfortunately I haven't been able to get any specific answer, and I need to know the fundamentals, why does this happen?

Logistic Regression:

                precision    recall  f1-score   support

          0       0.80      0.80      0.80        91
          1       0.78      0.78      0.78        82

avg / total       0.79      0.79      0.79       173

[[73 18]
 [18 64]] (confusion matrix)

SVM with rbf-kernel:

                precision    recall  f1-score   support

          0       0.80      0.99      0.88        91 (kind of overfitting for class 0)
          1       0.98      0.72      0.83        82

avg / total       0.89      0.86      0.86       173

[[90  1]
 [23 59]]

Random Forest Classifier:

                precision    recall  f1-score   support

          0       0.82      0.86      0.84        87
          1       0.85      0.81      0.83        86

avg / total       0.84      0.84      0.84       173

[[75 12]
 [16 70]]


texts: Struggling to integrate sklearn and pandas in simple Kaggle task

texts:
I'm trying to use the sklearn_pandas module to extend the work I do in pandas and dip a toe into machine learning but I'm struggling with an error I don't really understand how to fix.

I was working through the following dataset on Kaggle.

It's essentially an unheadered table (1000 rows, 40 features) with floating point values.

import pandas as pdfrom sklearn import neighbors
from sklearn_pandas import DataFrameMapper, cross_val_score
path_train ="../kaggle/scikitlearn/train.csv"
path_labels ="../kaggle/scikitlearn/trainLabels.csv"
path_test = "../kaggle/scikitlearn/test.csv"

train = pd.read_csv(path_train, header=None)
labels = pd.read_csv(path_labels, header=None)
test = pd.read_csv(path_test, header=None)
mapper_train = DataFrameMapper([(list(train.columns),neighbors.KNeighborsClassifier(n_neighbors=3))])
mapper_train

Output:

DataFrameMapper(features=[([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
       n_neighbors=3, p=2, weights='uniform'))])

So far so good. But then I try the fit

mapper_train.fit_transform(train, labels)

Output:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-e3897d6db1b5> in <module>()
----> 1 mapper_train.fit_transform(train, labels)

//anaconda/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y,     **fit_params)
    409         else:
    410             # fit method of arity 2 (supervised transformation)
--> 411             return self.fit(X, y, **fit_params).transform(X)
    412
    413

//anaconda/lib/python2.7/site-packages/sklearn_pandas/__init__.pyc in fit(self, X, y)
    116         for columns, transformer in self.features:
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self
    120

TypeError: fit() takes exactly 3 arguments (2 given)`

What am I doing wrong? While the data in this case is all the same, I'm planning to work up a workflow for mixtures categorical, nominal and floating point features and sklearn_pandas seemed to be a logical fit.


texts: Deep Neural Network: Output a Magnitude, not merely a Category

texts:
How does (can?) a deep neural network provide a magnitude estimate, rather than simply guessing what category an input belongs to?

For example, in guessing the weight of a cat, is there is a different way than creating discrete categories of cats (1 kg, 1.01 kg ... 10 kg), having the network guess the category from an image of the particular cat, and then deduce the weight given the category it chose?


texts: List of CNN for Emotion/Sentiment recognition on images with performance on main datasets (IAPS, GAPED, EmoPics, NAPS)

texts:
There are more and more databases of pictures classified or rated with emotions.

For instance, I know of 4 databases (IAPS, GAPED, EmoPics, NAPS) rating pictures on 2 dimensions: Valance (positive vs negative) and Arousal.

There are also databases with pictures classified as representing various emotions (POFA,KDEF,NimStim). Categories are emotions like anger, sadness, awe,...

While I find many papers using pictures from these databases on humans to generate emotions, I was wondering if there were convolutional neural networks (CNN) that were trained to regress/classify emotions. So far I just know of EmoNet was created by researchers from the University of Colorado and Duke University.

Is there a paper, competition, or website that lists and/or compare various neural nets for this task ? Something similar to what we had with ILSVRC to compare the image classification algorithms.


texts: Question about the Silhouette number for K means clustering

texts:
I have used K means clustering. In order to find the best value for K, I've looked at the changes of inertia value vs K and also changes of average Silhouette number vs K. The graph for inertia seems to indicate there are 5 clusters in the data. However, the average Silhouette number reaches a minimum at 5. So, how does one interpret this? If there are 5 clusters in the data, shouldn't the average Silhouette number have a maximum at 5 (not a minimum)?

Thanks


texts: is there any whatever2vec to generate one vector for one document?

texts:
Word2Vec generates a feature vector for each word. I was wondering if exist something2vec that generates one feature vector for one document. In this way I could compute the distance between two documents by only comparing two feature vectors.

Thanks.


texts: Stability of clusters in a unsupervised machine learning

texts: I am new to Unsupervised learning.  I am working on a customer segmentation data (with no labels). I have done K-Means and also calculated the silhouette score for the model. Now I want to study, if the model is good or not (similarity or stability for clusters) using Adjusted Rand Index. Can anyone give an idea how to do it, as I have have only the cluster labels which I believe are 'labels'. In the formula of RAND score, we need the "labels_true, labels" what should be the first one. I have checked on the internet, but not much is available. By the way, I am using Python. Thank you.


texts: Best Method for Using Experts to Score Qualitative Data

texts:
TL;DR: Looking for recommendations or resources on the best method to have subject matter experts score qualitative data to train ML model.
Problem:
I am working on a problem in the biochemistry domain.  I have a set of ~500 protocol results that need to be given a quantitative score.  Historically the biochemists just review each protocol and judge if the result is satisfactory or not.
Data:
The value being evaluated is the quality of a SNP allelic discrimination plot (sorry for the jargon, it actually not that important to the question).  Each allele is represented as a cluster of samples.  A good result has dense clusters with good separation between.  A poor result has clusters that spread out.  Note for this particular application, all data points in the plot are known, so no clustering algorithms are needed (e.g. k-means).

PoC Model:
For a proof of concept model, I gave each protocol result is my data set a binary pass/fail value (note I'm a non-expert on passable SNP plots, so my scoring was a bit capricious).  For feature engineering, I calculated various cluster separation metrics (Davies-Bouldin, Calinski-Harabasz, Silhouette) and did Box-Cox transforms (lambdas all near 0) to get normalized distributions.  All three metrics are highly correlated, so I applied PCA and used the 1st PC (95% variability explained).  I then fit a logistic regression curve, using the 1st PC as the variable and my manual pass/fail scoring values as the response.  I was able to achieve 90% accuracy on test data, which is promising.  I then verified on a new data set and reviewed it with the science team, and the results showed promise.  Now I have 4 real biochemists that are willing to score/grade my data set to use as the true expert scores.
Question:
So my actual question.  So what would be the best method for the subject matter experts to score the quality of the SNP plots?  I have scoured the interwebs for guidance on this problem, but haven't had much luck.  Mulling over the issues, I have 3 ideas so far.

Binary Score:  Simply have them score pass/fail for each plot.  Limits me to classification and logistic regression models.
Numeric Score: Score 1-10.  Values >5 are passing.  The big issue here is score "drift" as each reviewer goes through the data set.  An 8 at the beginning of scoring may match a 6 at the end of the scoring process.  End users really want the model to create a score, so looking at maybe doing regression models (although I think I sold them on using the logistic regression model probability of a "good SNP" as a score.)
Rank Score:  Force rank each protocol relative to the others for a 1-500 score.  Would also include the binary score to find the subgroup of rankings that pass.  Implementation would be to show a pair of plots side-by-side and have the review pick the best.  Use sorting algo in the background (merge sort?) to keep iterations down to O(n * log(n)).  Not sure the best tool to implement this - maybe Dash+Plotly?


texts: Classifying multilabel images with TensorFlow

texts:
The dataset that I am categorizing with TensorFlow ML library contains multiple labels per image. The contents are real estate images photographed from outside that are analyzed for various image features.

The question is how to assign the labels, the most straightforward way is to use either many long labels, or few short labels.

Categories:

Building type ("house", "apartment building", "condominium")
Build year ("old", "new")
Garage (boolean) - only for house
Floors (1, 2) - only for house
Construction ("standalone", "row house") - only for house

Many long labels (softmax_classifier):

house_old_nogarage_onefloor_standalone
house_old_nogarage_onefloor_row
house_old_nogarage_twofloor_standalone
...43 more
apartment_old
apartment_new

Few short labels (something like a Multiclass Support Vector Machine):

house
apartment
condominium
new
old
...7 more

Alternative to this would be to use a multi-label classifier by replacing the default softmax_classifier of the Inception Model v3 by something like the sequence_classifier.

I want to represent the data accurately, but also I don't mind doing simple categorization if the accuracy is acceptable.

Which of the proposed 3 labeling solutions suits the problem?


texts: Which algorithm/technique can I use to identify these sequences?


texts: Model localization: one big model vs two small models

texts:
Suppose I have want to predict how likely people in both US and Canada will buy product A in a month. Basically, it's a binary classification problem.

Assume I have 2 millions rows of historical data with people who has bought/not bought the product (label). One millions rows data are from US, the other are from Canada.

I can build a machine learning models in following two ways:

Build a model with 2 millions rows of data. One model with all the
data.
Model localization: Build a model with US data. Build another model with Canada data.

Intuitively, it seems to me option 1 will always performs better than 2. Because there will be more data in one model. But in practice, option 2 seems always performs better than option 1.

My question is when should I use option 2 instead of option 1? How to make the decision?


texts:
I understand maths better than Python but I would like to progress in Python. It would be amazing if someone would like to talk about it with me.

I have a feedforward neuron network with only one neuron for the last layer and $l$ layer plus one input layer.

We would like to use stochastic gradient descent on the function
$$
L(W,b) = \frac{1}{2M} \| \hat{y}(W,b,x) - y \|^{2}
$$

So we need to compute the gradient for the weight $W$ and the biais $b$. As
$$
\frac{\partial{L}}{\partial W_{j,i}^{k}} = \frac{1}{M} \sum_{i =1}^{n} {\langle \hat{y}(x_{i}) - y_{i} , \frac{\partial{\hat{y}}}{\partial W_{j,i}^{k}}(x_{i}) \rangle}
$$
with $W_{j,i}^{k}$ the $i$ th weight of the $j$ th neuron of the $k$ th layer. We only need to compute $\frac{\partial{\hat{y}}}{\partial W_{j,i}^{k}} = \sigma(z_{i}^{k-1}) \frac{\partial{\hat{y}}}{\partial z_{j}^{k}} $ with $z_{j}^{k} = \langle W_{j}^{k}, \sigma(z^{k-1}) \rangle + b_{j}^{k} $ the output of the $j$ th neuron of the $k$ th layer and $\sigma$ the activation function. This partial derivated can be computed by recursivity thanks to :
$$
\frac{\partial{\hat{y}}}{\partial z_{j}^{k}} = \sum_{s} { W_{s,j}^{k+1} \sigma'(z_{j}^{k}) \frac{\partial{\hat{y}}}{\partial z_{s}^{k+1}}  }
$$
with $s$ wandering along the $k+1$ th layer and $\frac{\partial{\hat{y}}}{\partial z_{j}^{l}} = \sigma'(z^{l}) $ so we can use a recursive algorithm.

Let's code our network :

class Network(object):

    def __init__(self, sizes):
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
        self.weights = [np.random.randn(y, x)
                        for x, y in zip(sizes[:-1], sizes[1:])]

So now we need to have acces to a vector $z$ containing the output. So I code (How to code it better ?) :

def output(self,a):
    res = []
    res.append(np.array(a))
    curlay = 0
    curneu = 0
    for s in self.sizes[1:]:
        z = np.zeros(s)
        for i in range(s):
            z[i] = np.dot(sigmoid(res[curlay]), self.weights[curneu][i]) + self.biases[curneu][i]
        curneu += 1
        res.append(z)
        curlay += 1
    return res

So now let's compute the gradient of $\hat{y}$. I think it could be cool to have a list of the partial gradient according $w^{k}$  [np.array $\big($ partial derivate according $w^{1}_{1,1}$,..., partial derivate according $w^{1}_{1,n}$ $\big)$, ..., np.array$\big($ partial derivate according $w^{l}_{1,1}$,..., partial derivate according $w^{l}_{l,p_{l}}$ $\big)$ ]

Now I'll stop coding a find a way to code it.
First it's easy to get the partial derivated according to $W^{l}_{1,i}$ (only one neuron in the last layer) it's $ \sigma(z_{i}^{l-1}) \sigma'(z^{l})$. So now... how can I continue ?


texts: I tried resuming training with my keras model, to save the model I used model.save('model.h5'). And to load it again I used model = load_model('model.h5'). I am using batch gradient descent i.e. the training data for each epoch is just one large batch. However, when I resume my training the error rate goes upto 100x then comes back down to about 2x, when compared to where the model left off, and this happens every time i resume training. Why is this so? And what is the solution?


texts: Compact model for on-device next word prediction

texts:
I'm an iOS developer with no production ML experience besides some pet projects.
The task is to create a model to predict the next word in English for a custom keyboard. I'd actually prefer to just use a pretrained model, which I tried with BERT half-precision (16 bit) model.
Still, even compact as it is, the model is just too big. Is there any way to solve this task any other way, with a more compact model?
Thanks.


texts: Voronoi Diagram Algorithm for Geo Locations?

texts:
As per my Understanding:

Voronoi diagram: A Voronoi diagram is a simple concept, and it's based on the minimal distance needed to reach a landmark. If you need to go to a metro station, the most natural algorithm is going to the nearest one.

Or

Voronoi Diagram: On a plane, for a set of sites (points in that 2D space with similar properties) the Voronoi diagram partitions the space based on the minimal distance to each site.

General Python Implementation:

This implementation takes in a list of points, each point being a tuple and returns a dictionary consisting of all the points at a given site.

from PIL import Image
import random
import math
def generate_voronoi_diagram(width, height, num_cells):
    image = Image.new("RGB", (width, height))
    putpixel = image.putpixel
    imgx, imgy = image.size
    nx = []
    ny = []
    nr = []
    ng = []
    nb = []
    for i in range(num_cells):
        nx.append(random.randrange(imgx))
        ny.append(random.randrange(imgy))
        nr.append(random.randrange(256))
        ng.append(random.randrange(256))
        nb.append(random.randrange(256))
    for y in range(imgy):
        for x in range(imgx):
            dmin = math.hypot(imgx-1, imgy-1)
            j = -1
            for i in range(num_cells):
                d = math.hypot(nx[i]-x, ny[i]-y)
                if d < dmin:
                    dmin = d
                    j = i
            putpixel((x, y), (nr[j], ng[j], nb[j]))
    image.save("VoronoiDiagram.png", "PNG")
        image.show()
generate_voronoi_diagram(500, 500, 25)

More Voronoi Diagram Implementations

Case

Imagine a city, where a fire is declared. Emergency services need to send a fire truck to this place. If there are N fire stations, what fire truck should be sent?

The nearest fire truck is in the light green Voronoi Area, the same area as the fire. No other truck can reach any light green point faster than this truck.

My questions are

  How to implement the Voronoi diagram for geolocations like places or lat, longs. As in the above cases.

  Are there any approaches/techniques that we can consider for similar cases?

  Can anyone be kind enough to walk me through it?


texts: Is a good practice to sum each rate (i.e. crime rate per 100.000 people)?

texts:
Consider a dataset from 1990 to 2017 that contains the crime rate per 100,000 people in some cities from Latin America.

I want to measure which city is more complex according to this data and other indicators. I'm using The analytic hierarchy process proposed by Saaty.

Consider this example:

city_code  r1990  r1991 r1992 r1993 r1994 r1994
  12345     120     80    91   110   105   99
  23456      10     15    16    12     7   11
  34567      90     91    85    75    77   65

According to my question, I want to sum each rate in each year, and then get for each year the mean of each rate based on the total.

Based on the example above:

city_code  total
  12345     605
  23456      71
  34567     483

city_code  mean1990  mean1991 mean1992 mean1993 mean1994 mean1994
  12345      0.20      0.13     0.15     0.18     0.17     0.16
  23456      0.14      0.21     0.23     0.17     0.10     0.15
  34567      0.19      0.19     0.18     0.16     0.16     0.13

So, is this a good practice? I couldn't find an example following this method? Any orientation about this subject will be appreciated.


texts:
How widely is Theano used in deep learning research?

Is Theano a good start to learn the implementation of machine learning algorithms?

Will learning the implementation of something like a feed forward network really help? Do graduate students implement neural networks or other algorithms at least once during their college days?

Background:

I have a reasonable idea about feed forward and recurrent networks, backpropagation, the general pipeline for a machine learning problem and the necessary mathematics.


texts: XGBoost: predict on only valuable features

texts:
I have a somewhat strange case that I can't find an answer to anywhere. It is really only applicable if you have a legitimately large data set. Let me describe.

XGBoost returns feature importance for all variables in the model. I have been using a data set with over 1000 features and typically about 25-50% of them come back with non-zero feature importance.

Now I want to make predictions with this model. Imagine I am making predictions on A LOT of observations (over 1m, imagine predicting on incoming tweets or something). If I re-train the model, giving it only the columns that the original training picked out as valuable (non-zero importance), then the re-trained model makes predictions MUCH FASTER than the original, and WITH THE EXACT SAME ACCURACY. This last part makes sense, because it is building the exact same trees. Also, not only is it faster, but it takes up a lot less space in memory because you are passing it something like a 1m x 250 matrix instead of a 1m x 1000 matrix.

My question is: is there a way to train a model, and then pass the predict function only the features it actually uses without having to re-train the model to except only those features instead of the full matrix? This blog post discusses a similar concept of using XGBoost for feature selection, but it does it by retraining the model each time.

The obvious answer is "just re-train it" but if I am using a lot of training observations then training the model takes awhile, so I would rather not have to retrain it if possible.

I am doing this in R, but solutions in Python/sklearn would also be helpful.
Thanks!


texts:
When running

import xgboost as xgb
print(xgb.__version__)

I am getting the following error:

xgboost.core.XGBoostError: XGBoost Library (libxgboost.dylib) could not be loaded.
Likely causes:
  * OpenMP runtime is not installed (vcomp140.dll or libgomp-1.dll for Windows, libgomp.so for UNIX-like OSes)
  * You are running 32-bit Python on a 64-bit OS

As stated in this other question here I am running 64-bit Python since this:

import sys
print(sys.maxsize > 2**32)

returns True.

Finally, to install OpenMP I tried this: brew install libomp but the problem persists.


texts: How we compare two paragraphs of unlabelled dataset semantically (STS)?

texts:
Column representation: Unique_id | Text1 | Text2

Unique_id 0

Text1 public show for reynolds suspension of his coaching licence. portrait sir joshua reynolds portrait of omai will get a public airing following fears it would stay hidden because of an export wrangle.

Text2 then requested to do so by spain s anti-violence commission. the fine was far less than the expected amount of about Â£22 000 or even the suspension of his coaching licence.

Unique_id 1

Text1 groening. gervais has already begun writing the script but is keeping its subject matter a closely guarded secret. he will also write a part for himself in the episode. i ve got the rough idea but this is the most intimidating project of my career.

Text2 philadelphia said they found insufficient evidence to support the woman s allegations regarding an alleged incident in january 2004. the woman reported the allegations to canadian authorities last month. cosby s lawyer walter m phillips jr said the comedian was pleased with the decision.

In the above problem, I've to compare two paragraphs texts i.e. Text1 & Text2 and then I've to compare semantic similarity between two texts. If they are semantically similar then it will print '1' if not then '0'

Any refrence implementation link or any suggestions !


texts: Algorithm selection for controlling a model vehicle

texts:
I've built an autonomous sailing robot (https://github.com/kolosy/ArduSailor). Turns out, the problem of piloting it is fairly complex, and my procedural approach to solving it hasn't worked well (or at all). I think that an ML-based approach may be better, and I'm trying to figure out the right algorithm to use.

I'm viewing it as an optimization problem of sorts - I've got a small set of parameters:

Position (lat, lon)
Orientation (in 9 DOF)
Speed
Wind speed & direction
Distance to waypoint
Heading to waypoint
Sail position (basically winch orientation, a single value between 0 and 180)
Rudder position (same as above)

If I'm thinking about this right, I need to vary my rudder and winch over time in response to my current position, orientation and wind direction to minimize the difference between my orientation and the heading to the waypoint, and minimize the distance to the waypoint.

My approach right now, is to train an ANN (using this code) by recording a manual run through my course. Is this the right approach and algorithm? Is there a better / more suitable way of thinking about this?


texts: How to combine features with different temporal scale in machine learning

texts: We have various types of data features with different temporal scale. For example, some of them describe the state per second while others may describe the state per day or per month from another aspect. The former features are dense on the time scale and latter features are sparse. Simply concatenate them into one feature vector seems not proper. Is there any typical method in machine learning can handle with problem ?


texts:
The F beta formula according to the wikipedia is "The weighted harmonic mean of precision and recall". I can not understand why in the left part of equation there is beta and in the right one is beta^2:

To my mind if I claim that Precision is 5 times important than Recall:
F beta = (1+beta)/(beta/P+1/R)=(1+beta)PR/(beta*R+P), where beta=0.2.

Is this right?


texts:
I'm running through a tutorial to understand the histogram plotting. Given the seaborn tips dataset, by running the sns.distplot(tips.tip); function the following plot is rendered.

Looking at the plot, I don't understand the sense of the KDE (or density curve). The middle column (the one with the lower value) between 2 and 4 doesn't seem to support the shape of the curve.

I have to say that I have little if no understanding on the principle used to plot it, so I would love to hear from somebody more experienced on

What's the added value of the KDE?
What's the process behind the calculation

Also, why using the same dataset with the standard matplotlib I get a slightly different representation (in which the density line above probably fit better)?


texts:
I'd like to contribute to Orange. To get used to the code, I tried to run the source code directly under Windows 10, so far without success. I have numpy and scipy installed, and the source code for orange3 from GitHub. If I run "python -m Orange.canvas" in the orange3 directory, I get the following error stack:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python37-32\lib\runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "C:\Program Files (x86)\Python37-32\lib\runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\__init__.py", line 1, in <module>
from .misc.lazy_module import _LazyModule
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\misc\__init__.py", line 3, in <module>
from .distmatrix import DistMatrix
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\misc\distmatrix.py", line 3, in <module>
from Orange.data import Table, StringVariable, Domain
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\data\__init__.py", line 4, in <module>
from .variable import *
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\data\variable.py", line 12, in <module>
from Orange.util import Registry, color_to_hex, hex_to_color, Reprable
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\util.py", line 17, in <module>
from Orange.data.util import scale  # pylint: disable=unused-import
File "C:\Users\Timo\Documents\Eclipse\workspace\Orange Github\orange3\Orange\data\util.py", line 5, in <module>
import bottleneck as bn
ModuleNotFoundError: No module named 'bottleneck'

Does anyone has experience with trying to run Orange directly from the source code? Any help is appreciated!


texts: How to do non arithmetic operation in python 3

texts:
lets say a=3 b=4 and c is an unknown constant.

a=3
b=4
F=0
F=a*b*c
print(F)
Its an error. I want 12*c or 12c


texts: Can TF turn a given graph into a recursive one?

texts:
I have a complex graph that takes an input $x_t$, it does a bunch of convolutions, tensor shape manipulations etc... until it produces $y_t$.
I want now to try a different approach, where I feed $y_t$ at some point into the graph (I concatenate it to some features after a convolution), while the net is processing $x_{t+1}$.

So the net should do the following ($y_0$ is a fixed initial input):

$(x_0,y_0)\rightarrow y_1$

$(x_1,y_1)\rightarrow y_2$

etc...

I have been looking at the standard RNN implementations in TensorFlow (GRU, LSTM, etc), but I don't quite understand how I can do this, because it seems that LSTM and the other RNN units have a fixed graph (like, a fully connected layer) and I cannot use the one that I need instead.

Do I need to subclass tf.nn.rnn_cell.RNNCell? That seems problematic too because it seems to admit only 2D tensors as input.

EDIT: I just discovered this awesome article and I'm going to document myself about tf.scan.


texts: Multi-Class Neural Networks | different features

texts:
This may be a wrong question or something so feel free to correct me :).

I have been studying neural networks for weeks now. I came across the multi-class classification model that uses neural networks.

As we see in this picture, the model allows you to classify your input into different classes. But this also assumes your input always use the same number features, right ? Meaning if I want to recognize handwritten digits, I should have images with the same dimension (for example 24x24) so I can use the same number of features (in this case 24x24=576). But what if one class, for example number 6, requires a different number of features (like the dimension of the handwritten digit 6 is 30x30 pixels)

I know that the logical way to do this, is to have two different neural networks, but Is there any way to simultaneously train a multi-class classification model where inputs might use different features? What does research say about this?

PS: If you have good reading materials about this, please feel free to link them too.

Bests,


texts: Factorization Machines with some some pairwaise effects and some linear effects

texts: I have some covariates $x_1$, $x_2$, $x_3$, .., $x_{10}$. I want a linear term for all these covariates and a pairwise effect for $x_1$ and $x_2$, and I do not want any other pairwise effect. Is there an implementation of Factorization Machines I can use to achieve this?


texts: Interpreting confidence interval results for datasets

texts:
I have created a dataset automatically and wanted to clarify my interpretation of the amount of noise using the confidence interval.
I selected a random sample and manually annotated the sample and found that 98% of the labels were correct. Based on these values I then calculated the confidence interval at 99% which gave a lower bound of 0.9614 and upper bound of 0.9949. Does this mean that the noise in the overall dataset is between the lower and upper bound and is then from 0.005% to 0.038%?


texts: How does the trainable projection layer used in PRADO and pQRNN work?

texts:
Trainable projection layers are said to be a very powerful thing but after reading:

https://www.aclweb.org/anthology/D19-1506.pdf
https://arxiv.org/pdf/2101.08890.pdf

I don't understand how it works. So how does the trainable projection layer used in PRADO and pQRNN work?


texts: Can an Isomap be embedded in a manifold of higher dimension than the corresponding MDS?

texts:
I am using the Isomap algorithm to operate a dimension reduction on a distance matrix $M_{dist}$.
For a given choice of nearest neighbors k to compute the geodesic distance, I use the following method to determine the dimensionality of the corresponding manifold:
I compute a distance matrix of D-simensional gaussian vectors, with the same average square distance as $M_{dist}$, which I call $M_{rand}$.
I then use a loop to compute the reconstruction error $e_{dist}$ of the Isomap fitted on $M_{dist}$ and the reconstruction error $e_{rand}$ of the Isomap fitted on $M_{rand}$ for the whole range of possible number of components for the Isomap, so for $d\in[1,dim(M_{dist})]$. I then consider the number of components maximizing the ratio $\frac{e_{rand}}{e_{dist}}$ as the intrinsic dimensionality of the manifold.
Using this method I have computed the intrinsic dimensionality of the Isomap obtained for every possible number of nearest neighbors $k\in[1,dim(M_{dist})-1]$. I obtain the following graph:
Isomap intrinsic dimension depending on number of nearest neighbors which increases from 4 to about 25 and then decreases to 14 when $k=dim(M_{dist}-1)$. This borderline case actually corresponds to a Multi-Dimensional Scaling (MDS) algorithm.
Therefore, using an MDS algorithm, my data seems to be comprised in a 14-dimension space. Does it make sense to consider an Isomap projection with a parameter k (number of nearest neighbors) resulting in an embedding of higher dimension than 14?


texts: what is the best distance measure to use for clustering high dimensional binary data?

texts: I have a dataset where the input is a dataset for ICU patients where each ICU stay has 40 features (20 vitals, 20 numerical lab values) and multiple time steps (the stays' length is between 6 and 19-time steps). The output is the 20 lab values (represented as binary numbers where 1 means the lab value is in range and 0 is out of range) at (t+1). The task is to predict the future output binary lab values for an input ICU stay. I want to cluster the output binary vectors into clusters and I have tried K-means but the euclidean distance is not suitable for binary data so what other clustering methods and distance measure best suit my problem?


texts: Binary text classification with TfidfVectorizer gives ValueError: setting an array element with a sequence

texts:
I am using pandas and scikti-learn to do binary text classification using text features encoded using TfidfVectorizer on a DataFrame. Here is some dummy code that illustrates what I'm doing:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
data_dict = {'tid': [0,1,2,3,4,5,6,7,8,9],
         'text':['This is the first.', 'This is the second.', 'This is the third.', 'This is the fourth.', 'This is the fourth.', 'This is the fourth.', 'This is the nintieth.', 'This is the fourth.', 'This is the fourth.', 'This is the first.'],
         'cat':[0,0,1,1,1,1,1,0,0,0]}
df = pd.DataFrame(data_dict)
tfidf = TfidfVectorizer(analyzer='word')
df['text'] = tfidf.fit_transform(df['text'])
X_train, X_test, y_train, y_test = train_test_split(df[['tid', 'text']], df[['cat']])
clf = LinearSVC()
clf.fit(X_train, y_train)

This gives the following error:

Traceback (most recent call last):

  File "<ipython-input-151-b0953fbb1d6e>", line 1, in <module>
    clf.fit(X, y)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\svm\classes.py", line 227, in fit
    dtype=np.float64, order="C")

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 573, in check_X_y
    ensure_min_features, warn_on_dtype, estimator)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 433, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)

ValueError: setting an array element with a sequence.

I have found numerous posts (e.g. here, here) mentioning that this error can indicate non-uniformity of the data. This post for the same error suggests it can also be due to a data typing issue. However, I can't see how my very simple example could be due to either of these. There is surely something simple I am missing. Help!


texts:
I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation:

self.R = {1:np.ones(self.W_xh[0].shape),
         2:np.ones(self.W_hh[0].shape),
         3:np.ones(self.W_hy[0].shape),
         4:np.ones(self.b_h[0].shape),
         5:np.ones(self.b_y[0].shape)}

self.lr = 1e-2
self.iota = 1e-6
self.gamma = 0.9
self.epochs = 10000

for e in range(self.epochs):
    for p in range(batches):
        if self.opt_alg == 'rmsprop':
            i = 0
            for param, dparam in zip([self.W_xh, self.W_hh, self.W_hy, self.b_h, self.b_y], [dW_xh, dW_hh, dW_hy, db_h, db_y]):
                i = i + 1
                self.R[i] = self.R[i] + np.multiply((1 - self.gamma), self.R[i]) + np.multiply(self.gamma, np.square(dparam))
                param = param - np.multiply(self.lr, (dparam/np.sqrt(self.R[i] + self.iota)))

i increases so that each entry in the dict can be accessed with its respective parameter. 'param' is a reference to the weights and 'dparam' is a reference to the new gradient, to be applied to 'param' using RMSProp.

Here are some results from training:

Epoch 12, Iter 0: Loss: 88.8836
Epoch 13, Iter 0: Loss: 88.8836
Epoch 14, Iter 0: Loss: 88.8836
Epoch 15, Iter 0: Loss: 88.8836
Epoch 16, Iter 0: Loss: 88.8836

As you can see it seems to be stuck, and is actually underperforming compared to sgd. My data isn't too strenuous either - just around 40 words. How can I fix this, and have I implemented it correctly?


texts: Do anomalous input features to autoencoder result in high errors on the corresponding output features?

texts: An autoencoder is trained by replicating each training instance to both input and output. However, when predicting for anomaly detection, will the output error be local to the same output feature(s) that were anomalous inputs  e.g. if I have 10 features, and I predict with anomalous input on feature #4, will the high output error be limited to output feature #4 as well, or will high error appear on other outputs? The whole detection becomes more useful if localised to specific features.


texts: Adding a bias makes Q-learning algorithm ineffective

texts:
I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success rate over 3000 iterations. I decided to add a bias to the implementation, and found that it severely harmed the results to no better than random.

That is, I added the bias term below:

inputs1 = tf.placeholder(shape=[1,16],dtype=tf.float32)
bias = tf.Variable(tf.zeros(shape=[1,4]))
W = tf.Variable(tf.random_uniform([16,4],0,0.01))
Qout = tf.matmul(inputs1,W) + bias
predict = tf.argmax(Qout,1)

The rest of the code is identical to the original solution. Any ideas why this would so negatively affect performance?

Update
It looks like someone else ran into this issue, and the answer given was that

  Having a bias term with the one-hot encoding prevents each state’s Q
  values from being independent

Any ideas why this is the case? The bias is added after the multiplication, so it's in the dimension of the actions, not the inputs. I don't see why this would make learning fail.


texts:
I'm still pretty new to artificial neural networks. While I've played around with TensorFlow, I'm now trying to get the basics straight. Since I've stumbled upon a course which explains how to implement an ANN with back propagation in Unity, with C#, I did just that.

While test-running the ANN with one hidden layer containing 2 neurons, I noticed, that it doesn't always get XOR right. No matter how many epochs it runs or how the learning rate was set. With some settings it happens more often than with other setting.

Usually I get something like this:

+---+---+------+
| 0 | 0 | 0.01 |
+---+---+------+
| 0 | 1 | 0.99 |
+---+---+------+
| 1 | 0 | 0.99 |
+---+---+------+
| 1 | 1 | 0.01 |
+---+---+------+

But in other occasions it looks more like this:

+---+---+------+      +---+---+------+      +---+---+------+
| 0 | 0 | 0.33 |      | 0 | 0 | 0.01 |      | 0 | 0 | 0.33 |
+---+---+------+      +---+---+------+      +---+---+------+
| 0 | 1 | 0.99 |      | 0 | 1 | 0.99 |      | 0 | 1 | 0.33 |
+---+---+------+  or  +---+---+------+  or  +---+---+------+
| 1 | 0 | 0.66 |      | 1 | 0 | 0.50 |      | 1 | 0 | 0.99 |
+---+---+------+      +---+---+------+      +---+---+------+
| 1 | 1 | 0.01 |      | 1 | 1 | 0.50 |      | 1 | 1 | 0.33 |
+---+---+------+      +---+---+------+      +---+---+------+

I've noticed that in every case, the sum of the outputs is ~2. It also doesn't happen most of the time but still quite often. Depending on what settings I use it happens every two or three runs, or it happens only after 10 or 20 runs. For me it seems more like a mathematical quirk in the stochastic nature of neural networks. But I'm not good enough with math to actually figure this one out by myself.

The question is: Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation? If it's not an error in the implementation, what is going on here? Is it because of the very symmetrical nature of an XOR? Which is the reason a single neuron can't handle it, as far as I understood.

I know I could post the source code as well, but I already double and triple checked everything, since I had a mistake in it with the bias calculation. Back then the values were completely off all the time. Now I'm just wondering if this sort of thing could actually happen with a correct implemented neural network.


texts: Learning sequences consisting of single non-zero entry and remaining zeros

texts: Pattern such as [1 0 0 0], [0 1 0 0], [0 0 1 0], [0 0 0 1] can easily be learned by using LSTM. We have created patterns where above mentioned vectors serve as the basis but we reveal one index randomly and if the random index contains 1 we let the vector as it is and if random index contains zero then we change the entry to -1. For example, [1 0 0 0] remains the same if randomly selected index was 0. However, if index was any other value the vector looks like [0 -1 0 0](index 1 was chosen). Based on this we created a new data set, LSTM can learn and predict these patterns as well but takes more time and examples before converging. However, if the vector length goes above 6 i.e. [0 0 0 0 0 0 -1] then LSTM is unable to correctly predict all the indices. This is probably because there are more zeros in the vectors and merely one non-zero entry, therefore, it is considered as noise. I have tried using lower learning rate, two LSTM layers, larger batch-size but the network is unable to reach above 90% accuracy. Is there anyway to tune the neural network so that it starts learning these sparse vectors or a way we can skip these zeros and focus on non-zero entries?


texts: How can I explain this chart showing 5-days moving average?

texts:
I have plotted the frequency of items sold through time, trying to determine the trends by moving average.
I considered a 5-days window. I would like to know if this approach makes sense and how I could interpret the results.
It's my first time with time-series and moving average (I have no a scientific background at all).
I hope you can help me.


texts: How to give a 3D Tensor as input to LSTM

texts:
I'm having X_train of shape (1400, 64, 35) and y_train of shape (1400,). I want to give X_train as input to LSTM layer and also want to find the Average (using GlobalAveragePooling Layer) of the Output of LSTM at each time step and give it as input to a Dense Layer. For this problem how to connect the layers and build a sequential model?

I'm using Tensorflow.Keras API's


texts: Value Function of Generative Adversarial Network

texts:
My question concerns the notation used in the Value function of  a GAN

Does "x ~ p_data(x)" mean  $$ E_{x\sim p_{data}}[log(D(x)]= \sum_{x} logD(x)(x)p_{data}(x)$$  ??

Many thanks,


texts: Method for Finding All Local Extrema Using Gradent Ascent/Descent

texts:
I have a very abstract model where a set of coefficients controls animal behavior. This model is so abstract that the actual values of a global extrema are not particularly interesting. However, the number of local extrema is extremely interesting (If anyone cares why just ask and I'll explain). I figured the biggest drawback of gradient ascent/ descent, that it can get stuck at local extrema, is actually an advantage for me.
My plan was to first run the gradient ascent algorithm with all the coefficients set to 0 (i.e. G(0) ) Then I do G(1). All the coefficients have a range of [0, 1]. If both runs result in the same point, then I have conclude that only one local maxima exists. If they result in different points, then I run G(mean of those points - small value) and (mean of those points + small value). Repeat until all local maxima have been found.
I know that this plan will give me all the local maxima in 1 dimension (pretty sure we proved this in my Calc II course). What I am unsure about is in more dimensions than that.


texts: clipping the reward for adam optimizer in keras

texts:
I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows:

sgd = optimizers.SGD(lr=0.01, clipnorm=1.)
sgd = optimizers.SGD(lr=0.01, clipvalue=0.5)

What are clipping the norm and clipping the value?

Also, How it is possible to implement the clipping the reward for Adam? Would you please let me know how I can do this?


texts: How flexible is the link between objective function and output layer activation function?

texts:
It seems standard in many neural network packages to pair up the objective  function to be minimised with the activation function in the output layer.

For instance, for a linear output layer used for regression it is standard (and often only choice) to have a squared error objective function. Another usual pairing is logistic output and log loss (or cross-entropy). And yet another is softmax and multi log loss.

Using notation, $z$ for pre-activation value (sum of weights times activations from previous layer), $a$ for activation, $y$ for ground truth used for training, $i$ for index of output neuron.

Linear activation $a_i=z_i$ goes with squared error $\frac{1}{2} \sum\limits_{\forall i} (y_i-a_i)^2$
Sigmoid activation $a_i = \frac{1}{1+e^{-z_i}}$ goes with logloss/cross-entropy objective $-\sum\limits_{\forall i} (y_i*log(a_i) + (1-y_i)*log(1-a_i))$
Softmax activation $a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$ goes with multiclass logloss objective $-\sum\limits_{\forall i} (y_i*log(a_i))$

Those are the ones I know, and I expect there are many that I still haven't heard of.

It seems that log loss would only work and be numerically stable when the output and targets are in range [0,1]. So it may not make sense to try linear output layer with a logloss objective function. Unless there is a more general logloss function that can cope with values of $y$ that are outside of the range?

However, it doesn't seem quite so bad to try sigmoid output with a squared error objective. It should be stable and converge at least.

I understand that some of the design behind these pairings is that it makes the formula for $\frac{\delta E}{\delta z}$ - where $E$ is the value of the objective function - easy for back propagation. But it should still be possible to find that derivative using other pairings. Also, there are many other activation functions that are not commonly seen in output layers, but feasibly could be, such as tanh, and where it is not clear what objective function could be applied.

Are there any situations when designing the architecture of a neural network, that you would or should use "non-standard" pairings of output activation and objective functions?


texts:
I'm new to RL and I'm attempting to train an RL agent to play MsPacman in PyTorch. I've adapted the code from this tutorial on the PyTorch page for my problem. The DQN has the following architecture:
DQN(
  (conv1): Conv2d(1, 32, kernel_size=(8, 8), stride=(4, 4), padding=(2, 2))
  (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
  (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (linear1): Linear(in_features=7040, out_features=512, bias=True)
  (linear2): Linear(in_features=512, out_features=9, bias=True)
)

I'm using the actor (policy) and critic (target) method with replay memory which has the following settings:

Replay Buffer: 100,000
Target Update: Every 10,000 steps
Bath Size: 128
Discount Rate: 0.999

For the exploration trade-off I'm using epsilon-greedy with the following curve:

where the x-axis is the step number (in million) and y the probability of selecting a random action.
The update of the policy network looks like this:
# next_state_values = Q-values precited by the target network
# GAMMA = discount rate (0.999)
# reward_batch = rewards for the states
expected_state_action_values = (next_state_values * GAMMA) + reward_batch

Loss calculation:
# state_action_values = actions taken by the policy agent
# expected_state_action_values - this is calculated above
loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

Updating the policy (gradients are clamped):
optimiser.zero_grad()
loss.backward()
for param in policy_net.parameters():
    param.grad.data.clamp_(-1, 1)
optimiser.step()

While training the agent I plot the duration of each episode shown below (the orange line shows the average of the previous 100 episodes):

After 4,000 episodes the agent isn't really progressing and gets stuck in corners like below:

Any idea what could be the issue? Some tips and pointers would be extremely helpful.


texts: Is there a site like fast.ai or deeplearning.ai but for machine learning instead of deep learning?

texts:
I'm looking for a summary of the pros and cons of different machine learning models in practise.

Something that includes:
- How the model works
- What kinds of outliers will be misrepresented by the model
- Effect of different parameters in each model on speed/performance


texts: the ratio of training set, validation set and testing set in K-fold cross validation

texts: Suppose I have 100 samples, then I want to use 5-fold CV, is the ratio of training set: validation set: testing set is 84:16:16? Is the number of data in validation set should always equal to testing set?


texts: minimization with a negative cost function: works in MATLAB, not in Python

texts:
I'm trying to use a particular cost function (based on doubling rate of wealth) for a classification problem, and the solution works well in MATLAB. See https://github.com/acmyers/compareCostFXs

When I try to do this in Python 2.7.6 I don't get any errors, but it only returns zeros for the theta values.

Here is the cost function and optimization method I've used in Python:

def costFunctionDRW(theta, X, y):

    # Initialize useful values
    m = len(y)
    # Marginal probability of acceptance
    marg_pA = sum(y)/m
    # Marginal probability of rejection
    marg_pR = 1 - marg_pA

    # =============================================================
    pred = sigmoid(np.dot(X,theta))
    final_wealth_individual = (pred/marg_pA)*y + ((1-pred)/marg_pR)*(1-y)
    final_wealth = np.prod(final_wealth_individual)
    final_wealth = -final_wealth

    return final_wealth

result = scipy.optimize.fmin(costFunctionDRW, x0=initial_theta, \
                   args=(X_array, y_array), maxiter=1000, disp=False, full_output=True )

Any advice would be much appreciated!


texts: Mismatching target dimension in Keras LSTM

texts:
I have built this network for text classification:

model = Sequential()
model.add(Embedding(vocabulary_dim, 150, input_length=max_length))
model.add(LSTM(150, return_sequences=False))
model.add(Dense(1, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

When I fit the model:

model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=5, batch_size=128)

I get this error:

ValueError: Error when checking model target: expected dense_9 to have shape (None, 1) but got array with shape (12481, 3)

I suppose because y_train and y_val are one-hot encoded, with shape:

array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 1.,  0.,  0.],
       .............])

but I'm not sure about this.


texts: How to determine semantic differences in NLP

texts:
I would need to determine the difference in meaning between the following two sentences:
I am at home
I am not at home
I am at the office

the first two sentences differs in verb, which changes the meaning of the sentences (to negative); the second one, with the first one, differs because of the place.
I have thought of word2vec, but I am not completely sure if this is the best tool to analyse sentences like the above ones. Also cosine_similarity could be a solution, but I would have not information about the meaning. I think it is more about semantic meaning...


texts: How to learn irrelevant words in an information retrieval system?

texts:
Right now my recommender system for information retrieval uses word embedding stogether with Tfidfs weights like written here:
http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/

Using Tfidf improves results. But I have the problem that irrelevant keywords (high frequent words) still have a large impact.
Can I learn a system such that it learns on which words to pay attention - preferred in an unsupervised way?

What can you suggest for a better information retrieval using word embedings?


texts: While using reindex method on any dataframe why do original values go missing?

texts:
This is the original Dataframe:

What I wanted : I wanted to convert this above data-frame into this multi-indexed column data-frame :

I managed to do it by this piece of code :

# tols : original dataframe
cols = pd.MultiIndex.from_product([['A','B'],['Y','X']
['P','Q']])
tols.set_axis(cols, axis = 1, inplace = False)

What I tried : I tried to do this with the reindex method like this :

cols = pd.MultiIndex.from_product([['A','B'],['Y','X'],
['P','Q']])
tols.reindex(cols, axis = 'columns')

it resulted in an output like this :

My problem :
As you could see in the output above all my original numerical values go missing on employing the reindex method. In the documentation page it was clearly mentioned :
Conform DataFrame to new index with optional filling logic, placing NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one. So i don't understand:

Where did i particularly err in employing the reindex method to lose my original values
How should i have employed the reindex method correctly to get my desired output


texts: In neural networks model, which number of hidden units to select?

texts: In the neural networks model, how many numbers of hidden units need to keep to get an optimal result, as per Cybenko theorem which demonstrates that only one hidden layer is sufficient to solve any regression/classification problem but the selection of the number of units in a hidden layer is very important because it impacts the model performance. Is there a theory to tell us how to choose the optimal number of units for a hidden layer?


texts: Data Science Career: From Researcher To Data Scientist

texts:
I’m currently working as a Research Assistant in Computer Science, specializing in both Human-Computer Interaction (HCI) and Health Informatics (HI) fields. As part of my role, I collect data from several clinical professionals (clinicians, physicians, and doctors) and analyze it (e.g., t-test, ANOVA and so on). Transforming the analyzed data and organizing it into several datasets, we report and use it by creating scientific papers that will fundament our analysis.

Recently, I started to have an interest in the Data Science field, and I’m considering to develop in that direction and become a Data Scientist. As I mentioned, I’m already applying statistical analysis as part of my job. That rises few questions:

How do the HCI and Data Science areas relate to each other?
How can I shift from being a Research Assistant in the HCI field into Data Science career?
What kind of studying/courses should I take?


texts: Update specific columns of dash datatable python

texts:
On dash I would like to update only 2 columns by real time values in every n_intervals of 1 minute. There are 6 columns in the data table, but I would like to update only 2 remaining should be as it is.

My code is the following:

@app.callback(Output('df_table', 'columns'),[Input('interval', 'n_intervals'),
               Input('df_table', 'data')])

def update_data_2(n_intervals,df):

    data_2 = pd.DataFrame(df)

    data_2=data_2[['A','B','C','D','E','F']]

    data_2['A']= real time values list

    data_2['B']=update values based on 'A' column
    return data_2[['A','B']]

While this is running, columns 'C' to 'F' should not change.
Please help me, when I run above code, it is giving me an error.


texts:
I'm doing my master thesis on Big Data Analytics. I'm trying to develop a algorithm to identify associations between some products in a supermakert. Imagine that I've this dataset:

Purchase_ID Product_ID  Purchase_Value
    1       2           4.5
    1       3           1.2
    2       3           1.4
    2       1           3.5
    2       2           7.3
    3       2           0.5
    3       3           1.0

What I want to conclude is that:
"Every people that by Product_ID 2 also buy 3"

Anyone knows If exists any code algorithm available to use in Spark Mllib? I already search on internet but I didn't found anything...

Anyone can help me?

Many thanks!


texts: Understanding Standard Error of Regression Coeficients (ISLR)

texts:
During my reading of Introduction To Statistical Learning, I reached this part and I'm facing a very hard time trying to understand what is being said:

This highlighted words are telling that this is how we can calculate the standard error of a simple linear regression coeficients.
The question is: the variance symbol used in the formulas is referring to the variance of what exactly? This is not explicit in the book.
Thanks in advance.
Paulo


texts: How best to show the best model over multiple labels?

texts:
I have 4 models I trained and I want to display their prediction success over 45 different labels I tested them on.

I get a very messy plot when I naively try to place them one on top of the other.

Here is the bar of the 4 models over 45 labels:

And here is the line of the 4 models over 45 labels:

I thought to maybe show them in parts, meaning create like 9 graph, each graph show only 5 labels.

What I really would like to is maybe show the winner model in each label and so, the winning model's color will pop right in front of you.

How do you suggest to do that?


texts: Siamese Network Triplet Loss for unlabeled images

texts:
Hi I am rather new to machine learning so I'm not really sure whether it's possible or not. Right now I'm trying to train a siamese network to identify whether an image is the same or not(by one-shot learning).

I've found a lot of implementations of such network on GitHub but they somehow need labeled data. Is it possible to do something similar with unlabeled data?


texts: Learning a simple sequence with RNN (Keras)

texts:
I am trying to learn a very simple sequence using an RNN (implemented in Keras)

The input sequence is randomly generated integers between 0 to 100:

x=np.random.randint(0,100, size=2000)

while the expected output value for time t is the (t-2)th input term i.e:

yt=xt-2

such that an example dataset looks like this:

+------+------+
|  X   |  Y   |
+------+------+
|    0 |   NA |
|   24 |   NA |
|   33 |    0 |
|    6 |   24 |
|   78 |   33 |
|   11 |    6 |
|    . |    . |
|    . |    . |
+------+------+

Note: I drop the NA rows before training.

I am trying to train a simple RNN to learn this sequence as below:

xtrain=np.reshape(df['X'], (df.shape[0], 1, 1))
#to match dimension of input shape for SimpleRNN layer.

model=Sequential()
model.SimpleRNN(2, input_shape=(None,1)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(x=xtrain, y=df['Y'], epochs=200, batch_size=5)

however, I find that this implementation results in a local minima which predicts constant value(~50) for all test observations.

Could anyone help me with the right way of implementing a basic RNN in Keras to learn this sequence?


texts: How to get columns from unsorted rows in Pandas? (MALLET)


texts: Why doesn't neural networks use the concept of degree of freedom?

texts:
In most (if not all) NMIST neural network tutorials you will see that the last two layers reduce to a multi-layer perceptron (MLP) and the number of labels is 0-9 for a total of 10 labels. It is well known in statistics that when you have 10 labels, you can set the score of one of them to be 0 and let the other 9 vary. This is the idea of degrees of freedom where 9 variables are allowed to "run free" but one can stay fixed leading to 9 degrees of freedom. More intuitively, when you calculate the probability that an image is a particular number you only need to specify the probabilities of it being one of the 9 numbers, because the probabilities for all ten labels must sum to 1 so the remaining label's probability must be 1 - sum(of the other 9).

The question is: why doesn't neural networks make use of this degree of freedom idea and instead estimate a score for all 10 labels instead of fixing the score of one of them to 0?


texts: How to assure the best cluster number for the given instances?


texts: How does Keras.Model() retrieve the different layers between its passed input and output layers?

texts:
I have recently been learning Keras and am trying to understand how the keras.Model function is able to interpolate the different layers between passed input and output layers. Do the input or output layers contain knowledge of the intermediate layers or are they stored as static data within the keras API?
The following is an example of this:
inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
model.summary()

Which outputs:
_________________________________________________________________
Model: "functional_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_2 (InputLayer)         [(None, 3)]               0
_________________________________________________________________
dense (Dense)                (None, 4)                 16
_________________________________________________________________
dense_1 (Dense)              (None, 5)                 25
=================================================================
Total params: 41
Trainable params: 41
Non-trainable params: 0
_________________________________________________________________

But only the input and output layers (layer 0 and 2 respectively) were provided to the keras.Model() function. Where does the middle layer come from in the summary function?
Thanks for the help!


texts: Any pytorch tools to monitor neural network's training?

texts:
Are there any tools to monitor network's training in PyTorch?

I am looking for an equivalent to tensorboard in tensorflow.


texts: Pandas dataframe with multiple hierarchical indices

texts:
I have a data frame which looks like this
FRUIT       ID      COLOR       WEIGHT
Apple       142     Red         Heavy
Mango       231     Red         Light
Apple       764     Green       Light
Apple       543     Green       Heavy

And I want the following result:
FRUIT                 COUNT
Apple       COLOR
            Red         1
            Green       2

            WEIGHT
            Heavy       2
            Light       1

Mango       COLOR
            Red         1
            Green       0

            WEIGHT
            Heavy       0
            Light       1

I tried different variations of set_index, groupby() and unstack() on the dataframe in combination with ['ID]'.count() and .size(), but my grouping works only on the first level (Fruit).
On the second level I am not able to get color and weight separately, I always end up with weight as another subindex of color so it looks like
FRUIT
Apple       COLOR    WEIGHT   COUNT
            Red      Heavy    1
                     Light    0

            Green    Heavy    1
                     Light    1

How can I get my desired result?


texts: I got 100% accuracy on my test set,is there something wrong?

texts:
I got 100% accuracy on my test set when trained using decision tree algorithm.but only got 85% accuracy on random forest

Is there something wrong with my model or is decision tree best suited for the dataset provided.

Code:

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20)

#Random Forest

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators = 1000, random_state = 42)
rf.fit(x_train, y_train);
predictions = rf.predict(x_test)
cm = sklearn.metrics.confusion_matrix(y_test,predictions)
print(cm)

#Decision Tree

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf = clf.fit(x_train, y_train)
predictions = clf.predict(x_test)
cm = sklearn.metrics.confusion_matrix(y_test,predictions)

Confusion Matrix:

Random Forest:

[[19937  1]
 [    8 52]]

Decision Tree:

[[19938  0]
 [    0 60]]


texts:
What is meaning of zip(kfold.split(X, Y) in sklearn?

for (train, test)in zip(kfold.split(X, Y)):


texts: how does xgboost handle inf or -inf values?

texts:
I am using xgboost for binary classfication. I have infs and -infs in my data due to the fact I am calculating ratios from one col and and another e.g. df[col1]/df[col2]. Since I have zeros and nans in these columns, it leads to me gettings infs, -infs, and nans.
I know xgboost can handle nan values, but if I replace the infs with a very large number e.g. 99999 or -99999 for -inf, is this an issue for xgboost? my understanding is that tree based methods for classification are unaffected by 'outliers'/inf values.
What would be best method to handle these?