texts: Regression: Should "known outputs" be also activated?

texts:
Okay so, I understand that inputs are sent directly to the network (basically being multiplied to weights of the nodes, receive a bias, and gets activated) and then the network produces an output through this feed forward method.

Now, in order for the network to "learn," we determine the error by comparing the predicted output versus the "known output / answer," and then back-propagate. Right?

What I don't get it is if the output goes through some activation function, say Sigmoid (results to a value between 0.0 - 1.0), how can it ever learn if the known output is a continuous value such as "Salary", "Number of Something", or in the use-case I have, "Volume of Orders" which ranges from anywhere between 300 to 1500.

// Example I have 7 inputs
// They go through a hidden layer that consists 16 hidden neurons
// and my output is a single neuron that's supposed to be "Volume of Orders"
//
// MyInputLayers[] --> HiddenLayers[] --> Output=0.123
//
// Since I learn from tutorials that Output layers go through an activation,
// I think it's impossible for the network to learn because a
// prediction of 0.123 compared to the "known output" that is, say 615
// is almost ridiculous

Should I apply Sigmoid to the "known output" as well? If so, how can I remap it when the network has already learned and is ready to do some predictive magic?

I must be missing / not understanding something here. Thank you in advance.

p.s. My dataset is composed of 6 months of records; an Excel file that has 200,000+ rows and 8 columns (7 leading, 1 lagging).


texts: Naive Bayes and Support Vector Machine (NBSVM) Classification

texts:
I am relatively new to datascience and have a question about NBSVM. I have a two class problem and text data (headlines from the newspaper). I want to use NBSVM to predict whether a headline has the label 0 or 1.

How I understood it, how I have to proceed now:

convert the headlines to a document term matrix
calculate the log-count ratio. As I understood it, these are the probabilities of the individual documents for a class (i.e. the probability that a document is in class 0 or class 1). Please correct me if I'm wrong here.
the log-count ratios then serve as input for the SVM. It inserts the ratios and sets the boundary between the two classes. When new data comes, the SVM tells you to which class the data belongs.

Is this right? Please note that this is only a theoretical procedure, not an implementation.


texts: Modelling a small graph with heterogenous nodes

texts:
I want to learn the relationship between 2 nodes, say a User U and a location L. I have a small graph network representing the different interactions between U and L. For example, U is connected to property P, which is in turn connected to L.
A graph G possesses the following properties:

~10-15 nodes
Heterogenous nodes of 3-5 categories with varying trust associated (needs to be learnt)
Heterogeneous edges with different confidences (needs to be learnt)
Upto 10 possible paths between U and L, paths can vary in length from 1-6.

Given a binary labelled set of such graphs representing whether U is connected L , I need to determine confidence of Connection(U, L). This would involve learning the confidence associated with each node and edge and then, the confidence of connection (U, L).
What is a good way to model this data? One way I can think of is to extract signals from all nodes/edges and train a regression model. Most graph techniques work well for large graphs with mostly homogenous nodes. Would there be any techniques I can employ here?


texts: Gumbel Softmax vs Vanilla Softmax for GAN training

texts:
When training a GAN for text generation, i have seen many people feeding the gumbel-softmax from the generator output and feed into the discriminator. This is to bypass the problem of having to sample from the softmax which is a non-differentiable process and hence prevents training.

My question is though, why not just feed the regular softmax (no argmax!) from the generator directly into the discriminator? What is the benefit of using the gumbel-softmax?

Thanks.


texts: Why Decision Tree Classifier is not working with categorical value?

texts:
I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx.

I am trying to model a Decision Tree Classifier as part of an ensemble (soft voting system). The problem is that I have the categorical features already converted to Integer values (i.e. if blood pressure is within the medium range, then blood pressure =2 in the dataset).

The categories for the feature variables are as follows

feature#1 (total cholesterol) = 1, 2, 3

feature#2 (Systolic Blood Pressure) = 1, 2, 3

feature#3 (diastolic Blood Pressure) = 1, 2, 3

feature#4 (smoking rate) = 1, 2, 3

The target/class variable (stroke) = 1, 2 (yes and no, respectively).

The problem is that when I draw the DT (please check the picture), I expect to get a splitting decision where a certain feature equals one of the above-mentioned values (i.e. 1, 2, 3). However, it is giving me other values like 1.5, 0.5, etc.,

I have included the code and a sample dataset for anyone who can help me out. btw, I need to keep the current dataset in the number format as I am using it in conjunction with other classifiers for the soft voting classifier.

Here is the dataset and this is the code for reference, I am using Spyder with Python 3.7


texts: Trouble performing feature selection using boruta and support vector regression

texts:
I was trying to select the most important features of a data set using Boruta in python. I have split the data into training and test set. Then I used SVM regressor to fit the data. Then I used Boruta to measure feature importance.The code is as follows:
from sklearn.svm import SVR

svclassifier = SVR(kernel='rbf',C=1e4, gamma=0.1)
svm_model= svclassifier.fit(x_train, y_train)

from boruta import BorutaPy

feat_selector = BorutaPy(svclassifier, n_estimators='auto', verbose=2, random_state=1)
feat_selector.fit(x_train, y_train)
feat_selector.support_
feat_selector.ranking_
X_filtered = feat_selector.transform(x_train)

But I get this error KeyError: 'max_depth'.
What might be causing this error?
Does Boruta work with any kind of models? i.e linear models, tree-based models, neural nets, etc.?


texts:
I have a 4 class classification problem using speech data. The speech signals are of variable length ranging from 1 sec to 8 sec. I am making chunks of 3 secs by chopping the longer utterances and zero padding the small utterances.
Now, when testing, the final label for an utterance should be aggregate of labels of the smaller chunks produced from that utterance.  What is the recommended way of doing it ?
No of chunks per utterance also varies- eg-

utterance 1 -  3 chunks of 3 secs each

utterance 2 - 2 chunks of 3 secs each.

for utterance 2, suppose if label comes to be class 1 for chunk 1 and for chunk 2 , it comes class 4. How to decide which label to assign to the full utterance?


texts: Some algorithms and approach for identification of specific patterns?

texts: I am working on a hobby project where I have a data related to financial trades(such as stock trades). Now I want to analyse this data and extract possible suspicious patterns and trades. Since I am new to Data Science so I would love to get few suggestions regarding what are some effective algorithms(machine learning) that can be relevant to such a scenario. One category that I think is using a possible outlier detection algorithm but I am unable to determine which one to use. I would also appreciate few pointers related to what my approach should be.


texts: How to calculate LFW accuracy of face recognition model?

texts:
In my research I have observed many of the face recognition algorithms propose their model accuracy interms of LFW dataset accuracy. I understood that LFW is a open source database and I did download that.

I have developed my own DNN model implemented for face recognition which is similar to facenet architecture. Can anyone help with the procedure to calculate LFW accuracy of my model?


texts:
I have a question regarding Hyperparameter Optimization for a Machine Learning Algorithm.

I try to fit a Support Vector Classifier and use Hyperparameter-Tuning (but it could be also another classifier).

My classes are highly imbalanced (20% of one class, lets call it “red” and 80% of the other, lets call it “black”).

Now, the objective of my Hyperparameter-Optimization is the cross-validation loss.

If, say, 20% of the observations are “reds” and 80% are “black”, then a really bad classifier would just label every case as “black” and achieve on average a cross-validation loss of 20%.

Now my question: If I see that the objective of my Hyperparamer-Optimization does not fall significantly below 20%, I could infer that it is useless without further analyzing. Correct? Or is there something I do not understand correctly?

This happens in all optimizations I am running now. I could conclude that my features are not informative.

I guess I have to change the objective for the hyperparameter optimization routine? (If my package allows for that)


texts: Intuitive interpretation of ratios between training set scores and validation set scores

texts:
I'm training models with the usual setup where you hold back a portion (in my case, 20%) of the data just to see how your trained model generalizes to unseen data, to see if it's overfitting.

When doing model selection (selecting hyperparameters), I sometimes come across these cases:

hyperparameter configuration 1:

training set score: 0.6
validation set score: 0.6

hyperparameter configuration 2:

training set score: 0.9
validation set score: 0.65

Now if you look at the raw numbers, it does look like configuration 2 generalizes better than configuration 1 but I'm a little bit worried about the large difference (0.9 to 0.65) between the scores for the training data and the validation data.

My question is: should I just consider the validation score when choosing the best model for actually using in production or does the ratio between training set and validation set scores carry some information?

I'm tempted to believe that cases where the training set and validation set score are more or less similar (as in configuration 1 above) are somehow more stable than cases where there is a large difference, even though, in absolute terms, the generalization score is better for configuration 2.

Is there any actual basis for this feeling? Could someone shed some light into this?


texts: Separating styles of numbers for simple digit classification

texts:
I am just getting started with my first simple digit classifier, so my doubts are at a pretty low level. In every dataset of digit images I've seen so far, different variants of each digit are grouped together, for example:

All of these images represent the number 1, but are fairly different in looks. Won't simple convolutional neural networks have a hard time learning the visual pattern for 1 in such a case? Especially considering how the third image is similar to 7 in design.
My questions are these: Would it be better to create other labels such as "1", "1-alt", "1-serif" etc? The CNN can then add the probabilities of the image being a variant of 1 and then give its prediction, but I'm not sure about this.
How do professional classifiers approach this problem?
Theoretically, will this method affect performance or accuracy in any way?


texts: How to convert cluster information across observations to a meaningful structure?

texts:
I have a dataset in which each row corresponds to a a sentence in a financial report. We have a column date containing the publication time of such report. Given a date, I do a clustering whose the number of cluster is $10$.
As such, we have such information as
day $1$: $(\mu_1, p_1) \quad (\mu_2, p_2) \quad \cdots (\mu_{10}, p_{10})$
day $2$: $(\mu_1, p_1) \quad (\mu_2, p_2) \quad \cdots (\mu_{10}, p_{10})$
$\cdots$
day $n$: $(\mu_1, p_1) \quad (\mu_2, p_2) \quad \cdots (\mu_{10}, p_{10})$
Here:

$\mu$ is the center of the cluster.

$p$ is its weight (usually computed as the proportion of rows that belongs to that cluster).

Given a day, the index of the cluster is just to differentiate them. The cluster $1$ in day $1$ has nothing to do with cluster $1$ in day $2$. It's possible that cluster $1$ in day $1$ is closer to cluster $2$ in day $2$ than to cluster $1$ in day $2$. Hence it's naive to put all clusters $1$ in a column, all clusters $2$ in a column, and so on. Unfortunately, the machine learning model requires a structured input.
I would like to ask for a method (or reference to a method) that takes into account this information permutation. In another word, I'm looking for a meaningful representation of clustering information across days.
Thank you so much!


texts:
I am trying to tune the "n_neighbors" for a kNN model andI have the following problem :

Based on the mean cross validation score the optimal kNN model should be the one with 10 neighbors.

On the other hand, when I plot the "scores vs neighbors" graphs I see that there are models whose score distance between the training and the test data is much smaller ( for instance the model with 20 neighbors ).

I am new to ML and this is still very confusing to me.. but should I stick to the 10 neighbors model, or is the 20 neighbors model better ? How do I decide ?
Any help is much appreciated.

Here is my code and the graphs :

best_score = 0
neighbors = np.arange(1,31)
all_train_scores = []
all_test_scores = []

for n_neighbors in neighbors :

        reg = KNeighborsRegressor(n_neighbors = n_neighbors, metric = 'manhattan')

        score = cross_val_score(reg, X_train, y_train, cv = 5)
        score = np.mean(score)

        if score > best_score :
            best_score = score
            optimal_choice = {'n_neighbors' : n_neighbors}

        reg.fit(X_train, y_train)
        train_score = reg.score(X_train, y_train)
        test_score = reg.score(X_test, y_test)

        all_train_scores = np.append(all_train_scores, train_score)
        all_test_scores = np.append(all_test_scores, test_score)


texts: What is the best ML/DL model to choose to calculate mobile network utilization increase

texts:
Let’s imagine the following scenario:

The marketing department decides to do the promotion next month and would like to give to every single mobile customer extra 10GB of data.

Due to simplicity, I have the following 5 features:

A: Amount of data volume downloaded by all customers per base station/transmitter (MB)

B:  Number of connected customers to the base station (#)

C: Total duration of the connected customers to the base station (seconds)

D: Radio resource utilization (%)

E: Average throughput per customer (kbps)

Considering the marketing requirements, I can work out that the average increase of data volume per base station is 10% (Feature A).

The question is: What impact is this promotion going to have on the mobile network (every single base station), mainly on Radio resource utilization (Feature D) and the average throughput per customer (Feature E).

I have tons of data for each base station from low to high traffic scenario, but I cannot train the model with 10% extra data volume traffic increase as this situation never happened in the network. It is still going to be "the peak data volume per Base station so far" + extra 10%.

How can I train the model, if target label is unknown (peak data volume + extra 10% never happened on the particular base station and I cannot take the data/stast from a different base station as the traffic pattern is different)?
It would be enough to point me into the direction, I can find more info and to study it further. Thanks.


texts: Advice on the learning resources for deep learning

texts: Which is better for beginner in machine learning: The deep learning book written by Yoshua Bengio or the videos and notes in CS231n from Stanford?


texts: Is it possible to use word2Vec to derive hyperonymy (hyponymy or ISA relation)?

texts: It's easy to have hyperonymy in WordNet, e.g. to know that "tea" is a case of "beverage". Is it possible to use word2Vec in this way?


texts: Can colored images have more than 3 channel values?

texts:
I was reading this well-known paper and noticed something in figure 1 below:

It says in the caption (The number of channels is denoted on top of the box). You can see that the number of channels is ranging from 1 up to 1024. I am confused here because it is known that the number of channels in colored images are 3 (R,G,B). Did I misunderstand something here?

Thank you.


texts: What's the best way to use binned data in a tree-based model?

texts:
I have some numeric data that has come 'binned', but the bins are not of equal sizes in terms of scale or quantile

For example, an age variable that is [0-16), [16-21), [21-30), [30-45), [45-65), [65, ]

If I leave it as a categorical variable, a tree will treat each category separately and discount the ordered relationship between the factors.

If I change it to an ordinal variable, e.g. [0, 1, 2, 3, 4, 5] and keep an array of labels for later reporting, the tree might split on, e.g. <2.5, which seems more natural to me, but then again the distance between 1 and 2 is not the same as the distance between 2 and 3 by any measure.

I'm leaning towards the second solution, but I would love some input!


texts: Is Mac OS the most suitable OS for a beginner Data Scientist?

texts: As far as I understand a data scientist should learn both Excel and Bash. And the golden mean is this case is Mac OS, because it has both. And later I can install Ubuntu on Macbook.


texts: What is advantage of using Dryad instead of Spark?

texts:
I found that Apache-Spark very powerful in Big-Data processing. but I want to know about Dryad (Microsoft) benefits. Is there any advantage for this framework than Spark?

Why we must use Dryad instead of Spark?


texts: Shuffle the data before splitting into folds

texts:
I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each iteration of hyperpatameter tuning re-shuffle the data before splitting into batches/folds (i.e., the shuffle argument in the KFold function)? I noticed that the outcome of the hyperparameter tuning process will be different depending on if I am shuffling the data prior to splitting it into folds.

I assume that if the outcome depends on shuffling then the model is not robust. Is this correct?
However, this also may not be 'fair' to the model because the result is not reproducible since the data for each fold is different every time I run cross validation (i.e., each hyperparametr combination is evaluated on totally different folds. For example, the training/validation dataset in fold #1 of the 1st tuning iteration is different than fold #1 dataset of the 2nd tuning iteration.)


texts: Tools for automatic anomaly detection on a SQL table?

texts:
I have a large SQL table that is essentially a log. The data is pretty complex and I'm trying to find some way to identify anomalies without me understanding all the data. I've found lots of tools for Anomaly Detection but most of them require a "middle-man" of sorts, ie Elastic Search, Splunk, etc.

Does anyone know of a tool that can run against a SQL table which builds a baseline and alerts of anomalies automagically?

This may sound lazy but I've spent dozens of hours writing individual reporting scripts as I learn what each event type means and which other fields go with each event and I don't feel any closer to being able to alert on real problems in a meaningful way. The table has 41 columns and just hit 500 million rows (3 years of data).


texts: How much of a problem is each member of a batch having the same label?

texts:
I have a batch size of 128 and a total data size of around 10 million, and I am classifying between 4 different label values.
How much of a problem is it if each batch only contains data with one label?
So for example - batch 0 all have the 3rd label.  Batch 1 all have the 1st.  Batch 2 the 2nd.  Etc.


texts: Transformer model: Why are word embeddings scaled before adding positional encodings?

texts:
While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension before adding positional encodings. Notice that this is different from scaling the dot product attention.
I'm referring to the 3rd line of the call method of the Encoder class here: https://www.tensorflow.org/tutorials/text/transformer#encoder
def call(self, x, training, mask):

  seq_len = tf.shape(x)[1]

  # adding embedding and position encoding.
  x = self.embedding(x) # (batch_size, input_seq_len, d_model)
  x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
  x += self.pos_encoding[:, :seq_len, :]

  x = self.dropout(x, training=training)

  for i in range(self.num_layers):
    x = self.enc_layers[i](x, training, mask)

  return x # (batch_size, input_seq_len, d_model)

I could not find any mention of this scaling in the papers I've read so far. People always show the input to the encoder as WE + PE, that is word embedding plus positional encoding. But this implementation seems to use sqrt(d_model) * WE + PE.
My questions:

Have you ever seen this extra scaling step mentioned in a paper? I didn't find it in "Attention is all you need" (Vaswani et. al.).
What is this additional scaling trying to achieve?


texts: How to segment a scanned Page so that it can be fed into an OCR

texts: I have an OCR that can read a given character. Now I want to build something that can take a scanned document and convert it into text. The problem is how do I segment each character? I know this can be done manually but how to automate it? I know that segmenting italics is different but since I want to go step by step I am considering normal fonts.


texts: Create new performance indicators (error metrics)

texts: I am wondering if any of you happen to know of a procedure/approach/rationale to develop new performance indicators (error metrics) that can be used to evaluate the prediction capability (say, accuracy) of a ML model? I am looking for any information that could be helpful to develop metrics for regression or classification problems. Thanks!


texts: How can a neural network get unstuck from a local minimum?

texts: Since, there are so many local minimums in so complex neural network function, it is common for a neural network to get stuck on a local minimum. How will the neural network get unstuck from that local minimum.


texts: Relationship Between Independent and Dependent variables

texts:
I calculated the distance correlation among the independent variables and the dependent variables to verify the nonlinear relationship among the variables and the values I am getting for each independent and dependent variable pair is not more than 0.1, which means that the relationship among the variables is not strong.
So is there any way of, still, applying the neural network model to get the better the results.


texts: Load test data to pass into predict_classes()

texts:
I am using the Sequential model to train a CNN. The test data are in the submission_data_path as .png images.

Is it possible to use ImageDataGenerator's flow_from_directory to load test data to pass into predict_classes()? Since flow_from_directory expects images to be inside folders named by classes is there a way to achieve that?

Or else what is the most preferred way? Thanks.

This is what I am trying to do,

test_data_gen = ImageDataGenerator()
test_it = dataGen.flow_from_directory(submission_data_path, target_size=(64, 64), class_mode='categorical')

predictions = classifier.predict_classes(test_it)


texts: State of the Art/Research 2020 of Time Series Forecasting/Prediction

texts:
Im looking for the state of the art/research of time series data for forcasting/prediction.
As far as im aware it is Extrem Gradient Boosting (XGBoost) or LSTM (neuronal networks) or are there other algorithms/models/methods which achieve good results.

I have a very small dataset less than 1000 data points. my guess/results are that LSTM has way to less data and XGBoost is the better alternative but im fairly new to forcasting so maybe im missing a method/model.


texts: Statistical test to establish that event B is caused by event A

texts:
I want to have a statistical test to establish that event B is caused by event A
Here
event B is customers that are supposed to leaved leave for competitor stay in our company
event A is an attractive offer for some customers (some may leave, and some may stay)
If we know that some of our customers is going to go for the competitor, and we come up with an attractive offer for them to stay.
If in the end, we notice that there were fewer customers that are leaving for competitors (event B), is there a statistical test that helps to establish a relationship that it is the attractive offer (event A) that retains those customers that will otherwise leave (event B)?


texts: I would like to pick up on the topic of deep learning. Should I begin from the topic of AI before working my way into Deep learning?


texts: Create a measure that converts minutes into hours

texts:
My searching is weak so I didn't find anything that satisfied my need.

I have a dataset of work items. I have a lead time column present on each record. It is in minutes, and I want to create a measure that displays lead time in hours. I want that measure to be (in pseudo code):

lead time (hrs) = lead time / 60

I'm a Tableau user and am finding this difficult to do in Power BI. What is your advice?


texts: Which algorithm should be used for an accurate job recommendation system

texts:
I'm building a testing project to get an introduction to DS & ML.
As a person part of the working force, sometimes finding a job is harder than it should be. I thought I could built a testing project to help workers find a job that best match their interests and their skills.

I could use a classifier, but I do not know if regression if the best way to approach this as a first pass. I was thinking of a GA system that could find some way to find way to approach this over time in a X generation. Maybe this isn't the best way for recommending jobs.

What I'm looking for isn't the code for the problem but more of algorithms ideas I should take a look at and implement in any given programming language. I'm looking to implement a system that can take a peak at job descriptions, interests and skills of an individual. I think this doesn't sound too crazy to be able to train an agent and then make it look at every jobs there is on LinkedIn for instance and take a look at 'Software Engineer' and give me 45% matching and here's the three skills that you have to work on or 98% match, please apply.


texts: Avoid iterations while calculating average model accuracy

texts:
I am fitting a model in R.

use createFolds method to create several k folds from the data set
loop through the folds, repeating the following on each iteration:

train the model on k-1 folds
predict the outcomes for the i-th fold
calculate prediction accuracy

average the accuracy

Does R have a function that makes folds itself, repeats model tuning/predictions and gives the average accuracy back?


texts: Identifying anomalies in spatial (latitude, longitude, time) data?

texts:
I'll start off by prefacing this with the fact that I'm not even sure if I'm asking the right question and what I'm really looking for is some guidance to dig in the right direction.
I have a set of 3D spatial data (latitude, longitude, time) on a regular spatial grid.
I expect there to be anomalies/outliers in this data that are of interest. They may be spatial or temporal in nature.
Are there machine learning or statistical methods that are tailored towards identifying such anomalies?
I think I'm struggling as it's such a vague question so some direction (even just the name of a method to go and google) would be really helpful.


texts: What is weight and bias in deep learning?

texts:
I'm starting to learn Machine learning from Tensorflow website. I have developed a very very rudimentary understanding of the flow a deep learning program follows (this method makes me learn fast instead of reading books and big articles).

There are a few confusing things that I have come across, 2 of them are:

Bias
Weight

In the MNIST tutorial on tensorflow website, they have mentioned that we need bias and weight to find the evidence of the existence of a particular pattern in an image. What I don't understand is, where and how the values for Bias and Weight are determined?

Do we have to provide these values or does the TensorFlow library calculates these values automatically based on the training data set?

Also if you could provide some suggestions on how to accelerate my pace in deep learning, that would be great!

Tensorflow Beginners Tutorial


texts: Text classification with multiple documents per labeled datapoint

texts:
I have a dataset with a label TRUE or FALSE for each person, but each person has multiple documents associated with them (emails and documents).

Right now I use a Random Forest Classifier on a bag of words consisting of all words in all documents put together per person (so that I have one row with all words and a label). It performs reasonably well, but I was wondering if you guys have some suggestions about how I can use the information of separate documents.

When I try to find information about this I only encounter multi-label classification, which is the exact opposite problem: multiple labels per document, instead of multiple documents per label.


texts: Generalized softmax derivative for implementation with any loss function

texts:
I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of the methods and mathematics behind it, instead of just working with the framework we are provided with.

My next goal is to implement a generalized version of softmax, so it can be combined with any desired loss function, but I am having some trouble understanding how to use the jacobian matrix that is the derivative of softmax in the backpropagation step.

My current implementation calculates the derivative of the loss function with respect to $z^{l} = w^{l}*a^{l-1}+b^{l}$ as follows:

$$
\frac{\partial L}{\partial z^{l}} = L'(a^l, y) \circ \sigma'(z^l),
$$

with:

$l$ denoting the output layer
$w^l$ and $b^l$ denote the weight matrix and bias vector for layer $l$
$a^{l-1}$ is the activation of layer $l-1$
$L'$ is the derivative of the loss function
$\sigma '$ is the derivative of the activation function for the output layer $l$
$\circ$ is the element-wise multiplication of two vectors or matrices

When for example using the MSE loss and a linear or sigmoid activation for the last layer, the size of the matrices in the element-wise multiplication make sense, both are of size $n_{out}\times 1$, where $n_{out}$ is the number of output nodes. But when using the softmax function as the activation function, the sizes are now incompatible, $L' \in \mathbb{R}^{n_{out} \times 1}$, and $\sigma ' \in \mathbb{R}^{n_{out} \times n_{out}}$, which does not permit element-wise multiplication.

I know that the Softmax function is almost always combined with the cross entropy loss function to make calculations simpler, but is this a necessity for the maths to work?

I have searched through a bunch of different forums and threads, but all the threads I can find deal with the derivation of the softmax function, and stop there, never delving into how it interacts with backpropagation. Am I missing something really simple in order to use softmax with other loss functions, or does it only work with cross entropy?

Edit: After some more research, I found the following thread: Applying the gradient of softmax in backprop. However there seems to be some differing opinions in the comments, and the derivative is no longer in the expected jacobian form, so I do not feel that this sufficiently answers my question. ANy help would be much appreciated!


texts: How to change the neural network model to improve the accuracy of mAP without learning from 0, but retraining already trained model

texts:
how you can change the Faster R-CNN Resnet 101 or Faster R-CNN Inception Resnet V2 model to improve the accuracy of mAP detection without learning the model with 0, but retraining already trained model

Accuracy mAP needs to be improved though by a small amount.

maybe there are some approaches, even the name of these approaches


texts: Is SKLearn's KS Test Implementation Wrong?

texts:
I was looking at the implementation for the KS TEST in sklearn here: https://github.com/scipy/scipy/blob/v1.5.0/scipy/stats/stats.py#L6580-L6745
data1 = np.sort(data1)
data2 = np.sort(data2)
n1 = data1.shape[0]
n2 = data2.shape[0]
data_all = np.concatenate([data1, data2])
# using searchsorted solves equal data problem
cdf1 = np.searchsorted(data1, data_all, side='right') / n1
cdf2 = np.searchsorted(data2, data_all, side='right') / n2
cddiffs = cdf1 - cdf2

The basic concept is to map the sorted inputs to the whole dataset and inspect the differences in the CDF. However, shouldn't you sort the whole dataset first so that the search sorted algorithm correctly matches to the distribution?
data_all = np.sort(np.concatenate([data1, data2]))

I seem to get the same metric for cdf1 and cdf2 without and with sorting data_all, but the values for cdf1 and cdf2 are different. Maybe I'm missing some mathematical reason why the metric is the same, but the resulting cdfs are different.


texts: How do we decide which kernel needs to be used in SVM?

texts:
How do we decide which kernel needs to be used for a particular dataset?
Is there any criteria needs to be followed?
And also what is the criteria to select C and gamma values?

Kindly excuse me if this question looks silly.

Thank you.


texts: I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?


texts:
I want to build a speech emotion classifier and I labeled my data into 3 emotions {negative, neutral, positive} the speech files I have are different of length, thus my audio features (mfcc,zcr, etc.) are flexible in length I learned that lstm (RNN generally) are the kind of model that solves this kind of problem, but all the codes I found are implementing a CNN model or a CNN with LTSM
So here is my questions:
According to the definition of my emotions what are the audio features I should use.?
how we use cnn for speech emotion classifier?
What kind of model I should use and why?


texts: How to find the most important attribute for each class

texts:
I have a dataset with 28 attributes and 7 class values. I want to know if its possible to find out the most important attribute(s) for deciding the class value, for each class.

For example an answer could be: Attribute 2 is most important for class 1, Attribute 6 is most important for class 2 etc. Or an even more informed answer could be: Attribute 2 being below 0.5 is most important for class 1, Attribute 6 being above 0.75 is most important for class 2 etc

My initial approach to this was to build a decision tree on the data and find the node that had the largest information gain/gain ratio for each class and that would be the most determining factor for that class. The problem with this is that the decision tree implementations I have found don't give the information gain/gain ratio for each node and as this is time bound I don't have the time to implement my own version. My current thought is to create multiple datasets which are all one class vs the rest and then perform attribute selection (eg. information gain) on them  to find the most important attribute. Is this the right direction to go down or is their a better option?


texts: Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classication for three classes)? Is it a problem that the documents are large enough and contain many common words? Can I train my data together with Wikipedia articles (using unique tag for every article) for a more accurate calculation word-embedings, or it can't give positive effect?


texts: TypeError: float() argument must be a string or a number, not 'function'

texts:
I am trying to clean the data. But I don't know how to remove a function from a column in data frame.

At row number 473 it show column N has a function . How it should be filtered out ?


texts:
I'm an AI student I need to train a deep neural network using the Alpha Zero (Silver et al) for a simple game using this implementation: http://web.stanford.edu/~surag/posts/alphazero.html. I was wondering if any cloud provider like Google or Amazon offers a free trial which suffice to train the model and supports the implementation mentioned above. The game is an Android app called Soccer Stars which is a fairly simple game with simple strategies.

Thank you very much for your time.


texts: Architecture choice for variable input dimension sizes

texts: So, I am doing this project where I have lets say a bunch of points. Each of those points can have a different number of RGB values. So lets say point 1 might have 30 RGB values, point 2 might have 500 rgb values. The idea is I want to decide something based on the variances those RGB values have across them. The idea has more than this to it, however I wanted to state it here in simple terms. I am kind of at a loss as to how I would fit this varying size of data as input and what architecture should I be approaching towards. Any clue or pointer would help.


texts: Gradient boosting Regression with zero-inflated outcome

texts:
I am trying to tune a Regression gradient boosting model where my target variable is zero inflated (80% zero) and the rest of the values are distributed as positive and negative values (not necessary symmetrically). What are good practices when training a model like this?

Any specific issues which I should be aware of to generate a good model? Based on my research, Tweedie Gradient Boosting is not fitted for this model because my target variable has a mix of negative and positive values around the zero mode; therefore, it doesn’t follow a tweedie distribution.


texts: Imbalance classes in Named Entity Recognition

texts:
I am currently working on a NER problem which attempts to extract 2 entities - place-of-interest(POI) and street from an address string in the Indonesian language.
I used IndoBert (available here) and attached FC layers onto the BERT model, utilising cross-entropy loss to predict the class each word belongs to. However a typical sample sentence label in my dataset looks like this [1, 4, 5, 5, 1, 1, 2, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Where 2-5 represents the POI and street class, 0 are the pad tokens and 1 codes for O tags(BIO tagging). However the model ends up constantly predicting majority class for all words.I then used ignore index and weights in my loss function which looks like this:
WEIGHTS = torch.Tensor([0.05, 0.2, 1, 1, 1, 1])
def loss(predicted, target):
    predicted = torch.rot90(predicted, -1, (1,2))
    criterion = nn.CrossEntropyLoss(weight = WEIGHTS,ignore_index=0, reduction='mean')
    return criterion(predicted, target)

However while this solves the problem above. The model also does not learn and the losses simply fluctuates around the same level. I am therefore wondering if there is a way to adjust for this class imbalance or stop the model from predicting the classes for [PAD] tokens.

This is the code for my model:
class aem(nn.Module):
    def __init__(self, no_class):
        super().__init__()
        self.bert = AutoModel.from_pretrained("sarahlintang/IndoBERT")
        self.drop1 = nn.Dropout(p=0.1)
        self.l1 = nn.Linear(self.bert.config.hidden_size, no_class)
        self.out = nn.GELU()

    def forward(self, inputs, attn):
        hidden = self.bert(inputs, token_type_ids=None, attention_mask=attn, return_dict = True)
        L1out = self.out(self.l1(self.drop1(hidden[0])))
        return L1out


texts: 1d time series to time series approximation using deep learning

texts:
I have a basic question about choosing right architecture of my deep learning task.

I have input signal $X(t)$ as a function of time when it is fed to the system, it generates output $Y(t)$ which is also a function of time.

I have a bunch of experiments performed with many input signals $X_1, X_2, X_3 ,...$ and many output signals $Y_1,Y_2,Y_3,...$.

Instead of performing more experiments I would like to generate neural net where I can feed input signal X (t) and get output Y (t).
I want to use already measured data as training data.

To me, this looks like like regression problem but involves time series data.
One of the neural network architectures that come to mind is RNN LSTM OR CNN.
People mainly use LSTM for forecasting problem (knowing history to predict the future) not regression.

So can I use CNN, use 1D filters and then some pooling and fully connected layers?
Will this work? What kind of set up should I use?


texts: how to export the tables into a csv file pandas

texts:
The following is a piece of code I wrote to create a pivot table for categorical vs continuous variable.

for row in categorical:
    for col in numeric:
        ptable = pd.pivot_table(df, values = col, index = row, aggfunc = ['min','max','median','mean','std',lambda x: 100*x.count()/df.shape[0]])
        print(ptable)
        writer = pd.ExcelWriter('report.xlsx')
        ptable.to_excel(writer, 'Sheet1')
        writer.save()

It displays the output as in the image:

but this is not a data frame and when writing into an excel file it displays only the last iteration values.

how do I get all the iterated tables into the excel file or separate excel files?


texts: How to implement hierarchical labeling classification?

texts:
I am currently working on the task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results (84% acc) than categories (94% acc). But subcategories are more precise as labels, what's important for the whole task. And then I got an idea to first do category classification and then based on the results continue with subcategories within the predicted category.
The problem here is that I do not know how to approach this problem/define network architecture. Any hints on the neural networks, how to deal with it?
Currently I defined network like this:
model = Sequential()
model.add(Dense(400, input_shape=(FEATURE_NUM,)))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))


texts:
While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per documentation, which is not clear, it says

For The momentum, type a value to apply during learning as a weight on
nodes from previous iterations.

Although that is not very clear. Can someone please explain?


texts: Map similar like clusters in two different cluster sets

texts:
I have two sets of clusters (Cluster_set_1 and Cluster_set_2) obtained by analysing two different sets of news paper articles.

A cluster includes word/word phrases as given in the examples below.

C1 in Cluster_set_1: Energy, Fuel, Oil, Mining
C2 in Cluster_set_1: school, education, students, schools, million, read
...
...
etc.

C1 in Cluster_set_2: Gas, oil, pipeline
C2 in Cluster_set_2: program, business, management,information, reports
...
...
etc.

Now I want to find clusters in the two cluster sets that are similar/related by considering the word/word phrases in the cluster as given in the example below.

Example:
Cluster of 'Energy, Fuel, Oil, Mining' in Cluster_set_1 is mostly similar/related to
Cluster of 'Gas, oil, pipeline' in Cluster_set_2

Reason: Because they both includes word/word phrases related to Energy

Since, I am dealing with two separate sets of clusters, what would be a suitable approach that I can use to connect the clusters in the two different sets?


texts: Xgboost - How to use feature_importances_ with XGBRegressor()?

texts:
How could we get feature_importances when we are performing regression with XGBRegressor()?

There is something like XGBClassifier().feature_importances_?


texts:
Hi guys.
I'm not a data analyst and I need some direction in this. I'm looking for a way to know the events of the fuel during a range of time, could be a day or a month, etc. If the consumption was like the picture above would be easy.

The problem is that the data I have is like this picture.

I need to take the fueling and the possible thefs, that considering that I have false data, that could go up or down. I can see that those abrupt changes are fake because it always comes to the original value, the main problem is that I don't know how long they are gonna stay fake. Could be just one point or n points of bad points.

So, how can I detect these events (fueling and possible theft) whitout counting the errors as events?.
What algorithms or formulas can I apply here?


texts: How to do node classification without deep learning?

texts:
I have a dataset of coordinates with labels, a set of those coordinates (a graph) follows globally the same pattern.
ie : I have coordinates of eyes,mouth and nose on thousands of images. A graph can be formed from those coordinates on a single image and I want to use that structure to learn from that pattern to predict the class of the nodes (point on image).
I'd like to create machine learning model to accurately predict the class of each element.
the task is pretty simple but all references I come across are about fancy research Graph Neural Networks.
How can I do it efficiently?


texts: Performance gain of GPU when learning DNNs

texts:
Currently, I learn deep neural networks on my CPU (i7-6700K) using TensorFlow without AVX2 enabled. The networks need about 3 weeks to be learned. Therefore, I am searching for a (cheap) way to speed up this process. Is it better to compile TensorFlow enabling AVX2 or to buy a cheap[1] GPU like the GeForce GTX 1650 Super (about 180€ and 1408 CUDA cores)? What is the estimated performance gain of using a cheap[1] GPU?

[1] Cheap compared to current top edge GPUs.


texts: Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

texts:
I am using a three node system:

Master node: $64$ GB RAM
2 slave node: $32$ GB RAM

I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function of Apache spark then $64$ GB memory of master node got totally consumed and start to use swap memory which makes execution too slow and eventually lead to heap memory full.

If matrix size is $3000 \times 3000$   then no issue but when we consider size greater than $3000$ (like $3500$ or $5000$) then above-mentioned issue arises.

Note: Even my MATLAB is able to compute the inverse of $10K \times 10K$ matrix on 32 GB RAM but Apache Spark got Memory full if I used inbuilt SVD function on above system.

Sample Code is as below:-

import org.apache.spark.mllib.linalg.{Vectors,Vector,Matrix,SingularValueDecomposition,DenseMatrix,DenseVector}
import org.apache.spark.mllib.linalg.distributed.RowMatrix

def computeInverse(X: RowMatrix): DenseMatrix = {
  val nCoef = X.numCols.toInt
  val svd = X.computeSVD(nCoef, computeU = true)
  if (svd.s.size < nCoef) {
    sys.error(s"RowMatrix.computeInverse called on singular matrix.")
  }

  // Create the inv diagonal matrix from S
  val invS = DenseMatrix.diag(new DenseVector(svd.s.toArray.map(x => math.pow(x,-1))))

  // U cannot be a RowMatrix
  val U = new DenseMatrix(svd.U.numRows().toInt,svd.U.numCols().toInt,svd.U.rows.collect.flatMap(x => x.toArray))

  // If you could make V distributed, then this may be better. However its alreadly local...so maybe this is fine.
  val V = svd.V
  // inv(X) = V*inv(S)*transpose(U)  --- the U is already transposed.
  (V.multiply(invS)).multiply(U)
}


texts: R, keras: How to get output of a hidden layer?

texts: I am using package Keras in R to do a neural network. How may I extract the output from a hidden layer? I found an example in python, but it is just I have no idea how to do that in R.


texts: Prove data points not enough? Estimate required number of data points?

texts:
I am working on a problem with only ~ 30 data points, I need to use these 30 points to build a machine learning model (ridge regression, random forest ... etc.), then use it to predict on another thousands of data points.

After some experiments, I believe the data set is too small. However, I am wondering is there a standard approach to prove that whether we have enough data points? And how to determine how many data points are needed? Thanks!


texts: How to deal with multiple categorical data set

texts:
Please tell me how with sex, smoker, region?
Should I perform one hot encoder for all?


texts: Find optimal P(X|Y) given I have a model that has good performance when trained on P(Y|X)

texts:
Input Data:

$X$ -> features of t shirt (colour,logo,etc)

$Y$ -> profit margin

I have trained a random forest on the above $X$ and $Y$ and have achieved reasonable accuracy on a test data.
So, I have

$P(Y|X)$.

Now, I would like to find $P(X|Y)$ i.e probability distribution of $X$ features given I am expecting this much profit margin.

How do I do that with a random forest(or any other discriminative model)?

One suggestion for me could be to start with a generative model rather than a discriminative model. But, my understanding is generative model generally require a lot of data to train unless that make some very restrictive assumptions such as conditional independence of $X$'s in case of Naive Bayes?

Other suggestion could be to just switch $X$ and $Y$ and train a discriminative model. Now $X$ will be profit margin and $Y$ will be features of a t shirt. $P(Y|X)$ will directly give me the probability distribution of t shirt features, given a target profit margin. But this approach doesn't seem right to me, as I have always though of $X$ as casual variables and $Y$ to be effect.

Also, from what I have heard, similar question has been posed for drug discovery and algorithms have been designed which come up with candidate new drugs that have high degree of success. Can someone point me to research literature in this domain?

Update:

I have come across this and this which talks about GANs being used for drug discovery. Generative adversial networks seem like a good fit for my problem statement so I have been reading about them. But one thing I understood is GAN generate samples in an unsupervised way. They try to produce sample which is like first capturing the underlying distribution of X and then sampling from that distribution. But I am interested in X|Y. X and Y are defined above. Should I explore something other than GANs? Any pointers please?

Follow up Question:

Imagine I have a GAN trained that has learned how to make t shirts(output sample Xs). How can I get the top 5 shirts for given Y?


texts: Dataset with some mislabeled data (around 1%)

texts:
I have a dataset with around 1% of mislabeled data, it is a multi label problem and i want to find a way to correct those incorrect labels.
Assuming that the amount of mislabeled data is low i divided the dataset in Train/Test and trained a classifier taking care that the classifier does not overfit.
After that i know that the accuracy on the Test set is as high as possible i evaluated the whole dataset using the classifier and the result is a new set of labels which i assume that are the corrected labels.
Is this the correct approach to solve a problem like this?


texts:
I'm a freshman undergraduate student (mentioning this so you may forgive my unfamiliarity) who is currently doing research using neural networks. I've coded a three-node neural network (that works) based on my professor's guidance. However, I'd like to pursue a career in AI and Data Science, and I'd like to teach myself more about these properly in-depth. Are there any books or resources that will teach me more about neural network structures, deep learning, etc. Are there any recommendations?

Note: I'm proficient in Java, Python, Bash, JavaScript, Matlab, and know a bit of C++.


texts: target variable prediction among possible answers

texts:
I have a dataset on which I would like to apply a Machine Learning algorithm for multi-class classification. An example of my target variable (in string format, will be later OneHotEncode-d):
|       target      |
+-------------------+
|  (Mike, Michael)  |
| (Anne, Annabelle) |
|    (Joe, Joseph)  |
|    (Mark, )       |
|    (Aaron, )      |
+-------------------+

I want to have a model that, for example, predicts Joseph but does not consider it wrong because it is one of the two possible answers.
Is this a multi-class or multi-label problem? Can the two definitions go alongside or is it one or the other? I am a bit confused, though I propend towards the latter.
In case my guess was correct, would it be sufficient to just pick one of the models that supports multi-label classification, fit() it and score() it?


texts: XGBoost model performance barely budging, and max_depth is basically irrelevant

texts:
I have a dataset that includes 62 features and around 1 million observations. The 62 features are mostly socioeconomic status indicators for students as they start a school year. The labels are the students grade in the next school year. My goal is to help identify students at the beginning of the year who will perform poorly, and then give them access to additional resources to prevent poor grades. More specifically, my goal is to properly predict which students will be in the bottom 3%ile of grades in coming school year. The magnitude of errors is important - if I predict a student will be in the bottom 3% and the student ends up being an A+ student, that is very costly to me. For this reason, I tried making an XGBoost regression (since classification doesn't punish wildly wrong answers).
However, my results so far have been disappointing. My features are either percentile data (continuous from 0 to 1) or binary. My labels are percentile data (continuous from zero 1). It seems that no matter which hyperparameters I adjust, the RMSE starts at 0.28 and barely budges from there. Here is the model with output for max_depth = 1.
#Regression Model
params = {
  'tree_method' : 'exact',
  'min_split_loss':5,
  'colsample_bynode':0.5,
  'reg_lambda':20,
  'reg_alpha':100,
  'subsample':0.5,
  'learning_rate': 0.3,
  'max_depth':1,
  'objective': 'reg:squarederror',
  'verbosity':0,
  }
watchlist = [(train, 'train'), (test, 'val')]
reg = xgb.train(params, train, num_boost_round=100, early_stopping_rounds=3, evals=watchlist)

[0] train-rmse:0.281513 val-rmse:0.28249
Multiple eval metrics have been passed: 'val-rmse' will be used for early stopping.

Will train until val-rmse hasn't improved in 3 rounds.
[1] train-rmse:0.281126 val-rmse:0.282324
[2] train-rmse:0.280893 val-rmse:0.282113
[3] train-rmse:0.280809 val-rmse:0.281991
[4] train-rmse:0.280657 val-rmse:0.281931
[5] train-rmse:0.280601 val-rmse:0.281875
[6] train-rmse:0.280551 val-rmse:0.281833
[7] train-rmse:0.280432 val-rmse:0.281818
[8] train-rmse:0.28036  val-rmse:0.28174
[9] train-rmse:0.28033  val-rmse:0.281737
[10]    train-rmse:0.280312 val-rmse:0.281732
[11]    train-rmse:0.280273 val-rmse:0.281694
[12]    train-rmse:0.280262 val-rmse:0.281682
[13]    train-rmse:0.280207 val-rmse:0.28168
[14]    train-rmse:0.280184 val-rmse:0.281649
[15]    train-rmse:0.28017  val-rmse:0.281634
[16]    train-rmse:0.280159 val-rmse:0.281638
[17]    train-rmse:0.280128 val-rmse:0.281641
[18]    train-rmse:0.280114 val-rmse:0.281637
Stopping. Best iteration:
[15]    train-rmse:0.28017  val-rmse:0.281634

And here is the output when I increase to max_depth = 6:
[0] train-rmse:0.281331 val-rmse:0.282431
Multiple eval metrics have been passed: 'val-rmse' will be used for early stopping.

Will train until val-rmse hasn't improved in 3 rounds.
[1] train-rmse:0.280839 val-rmse:0.282097
[2] train-rmse:0.280446 val-rmse:0.281867
[3] train-rmse:0.280329 val-rmse:0.281756
[4] train-rmse:0.280185 val-rmse:0.281703
[5] train-rmse:0.280153 val-rmse:0.281693
[6] train-rmse:0.280151 val-rmse:0.281695
[7] train-rmse:0.280106 val-rmse:0.281692
[8] train-rmse:0.280081 val-rmse:0.28167
[9] train-rmse:0.280081 val-rmse:0.281671
[10]    train-rmse:0.28008  val-rmse:0.281672
[11]    train-rmse:0.280062 val-rmse:0.281654
[12]    train-rmse:0.280062 val-rmse:0.281654
[13]    train-rmse:0.280025 val-rmse:0.281638
[14]    train-rmse:0.280025 val-rmse:0.281638
[15]    train-rmse:0.280025 val-rmse:0.281638
[16]    train-rmse:0.280014 val-rmse:0.281642
Stopping. Best iteration:
[13]    train-rmse:0.280025 val-rmse:0.281638

It doesn't make sense to me that a max_depth of 1 yields virtually the same results as max_depth = 6, when I have 62 features... Also the RMSE at 0.28 isn't much better than just random, which would be around 0.4. I feel like there is something blatantly wrong given max_depth is essentially irrelevant.
Does anyone have some advice to improve the performance? Or perhaps a different machine learning technique rather than regression?


texts: How to search collection of podcasts (.mp3 files)?

texts:
I have a collection of podcasts, in particular the "Talkingmachines" Podcasts ("Human conversation about machine learning"), see http://www.thetalkingmachines.com/blog/

There are 22 episodes, each 1 hour long. MP3 files are available on my local disk.

Now  when I want to search the audio files for the episode where they talk about, say, the kmeans++ algorithm , what should I do?

Specifically, I want to search for that episode where a male voice (Ryan) talks about kmeans++ in the first half of the podcast.

Again, what should I do? Is there a desktop tool available that can do this search? Should I upload the files to some webservice that can do audio recognition?


texts: Ideal aggregation function for Partially Connected Neural Network (PCNN)

texts:
I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and edges in the hidden layers.

Example Data:

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float)
Y = np.array([[0],[1],[1],[0]], dtype=np.float)

Example Graph:

I am currently using the sum product aggregation function to calculate each layer's values:

$$
\sum_{i=0}^n w_{ij}x_i
$$

The synapse weights are denoted by two subscripts $ij$. Subscript $i$ represents the previous neuron and subscript $j$ represents the current neuron under consideration. The sum product for current neuron $j$ is computed as:

$$
s_j = w_{0j}x_0 + w_{1j}x_1 + ... + w_{nj}x_n = \sum_{i=0}^n w_{ij}x_i
$$

Is sum product an ideal aggregation function for PCNNs? Dot product won't work as the tensor sizes are almost always incompatible.

Update: As Bence Mélykúti suggested, my solution was to use a weight vector $w$ that is padded with zeros and reshaped for dot product. Here is how I accomplished it in my Python library:

from functools import reduce
import numpy as np
import math

# An example of some input/sensor data
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float16)
# From the graph, two inbound synapses in layer 1, here randomly generated
W = np.random.random((2,))

# Enumerate the element count in each array
xElements = reduce(lambda x, y: x*y, list(X.shape))
wElements = reduce(lambda x, y: x*y, list(W.shape))

# If previous layer has more elements than the weights in the current layer
if xElements > wElements:
    # Create a list of zeros of shape X.shape
    A = np.zeros(X.shape).flatten().tolist()
else:
    # Otherwise, create a list with the larger number of elements in W
    A = np.zeros(W.shape).flatten().tolist()

# If the list is odd, add one element to ensure it is even
# Done so the array can be reshaped to fit the data and dot product
if len(A) % 2 != 0:
    A.extend([np.float16(0.0)])

# Fill the list with the elements from W
for ix,ele in enumerate(W):
    A[ix] = ele
A = np.array(A)

# Convert list to an array and reshape to be compatible with dot product
A = A.reshape(X.shape[-1],(math.floor(A.shape[0]/2)))

result = np.dot(X,A)

The result of the dot product looks like:

print(result)
array([[0.        , 0.        , 0.        , 0.        ],
       [0.        , 0.        , 0.        , 0.        ],
       [0.45510922, 0.79058734, 0.        , 0.        ],
       [0.45510922, 0.79058734, 0.        , 0.        ]])

Thanks Bence!


texts: Multi class product classification according to description of product

texts: I have some products, along with their description. I wish to assign USPSC code to each product. This would require classification on 4 levels. All the examples I could find online were that of sentiment analysis, which is binary and therefore different from my problem statement here. Could anybody guide me to some resources? Or how the binary classification can be extended to multiple classes so I can pick keywords from product description and assign them categories? Thanks!


texts: How to deal with missing continuous data

texts: So I have a dataset where I have a continuous variable for only about 10% of the entries. How would you incorporate this in a model. Imputing does not make much sense to me, because there are so few values, however for those entries where I do have it, it is quite an important feature.


texts: Why Overfitting sometimes appears when compile model multiple time, is it normal?

texts:
At the time I got small datasets of brainwaves (EEG) (105 samples) for 3-class classification problem. I split my data into 3 part:
Train data = 90 (data)
Validation data = 10 (data)
Test data = 5 (data)
And here is my code:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = test_size, shuffle = True)

# create model
model = Sequential()
model.add(Dense(200, input_dim = 95, activation = 'relu', bias_initializer = 'ones', kernel_initializer = 'normal'))
model.add(Dense(30, activation = 'relu', bias_initializer = 'ones', kernel_initializer = 'normal'))
model.add(Dense(3, activation = 'softmax'))

# Compile model
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

history = model.fit(X_train, y_train, validation_split = 0.1, epochs = 150, batch_size = 10)

_ , acc_train = model.evaluate(X_train, y_train)
_ , acc_test = model.evaluate(X_test, y_test)

I run my model for about 10 times, 9 times I got normal accuracy, loss graph but 1 times I got overfitting. Here is its graph:
1st Run:

2nd Run:

3rd Run:

4th Run:

5th Run:

6th Run:

7th Run: (Overfitting Appear)

8th Run:

9th Run:

10th Run:

So I wonder that:
(1) Why the Overfitting sometime appear? It is normal?
(2) Although the Overfitting appear the model still can predict very well on test data (sometime 80, 90 or even 100%), is it because of my small dataset?
(3) Should I use the K folds cross validation method instead of this method for small dataset?
Thank you very much.


texts: Understanding the training phase of the tutorial "Using Keras and Deep Deterministic Policy Gradient to play TORCS" tutorial

texts:
I am trying to understand the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS (mirror, code) by Ben Lau published on October 11, 2016.

The tutorial says:

  Then the actor policy is updated using the sampled policy gradient:
  $$\nabla_\theta J = \frac{\partial Q^{\theta}(s,a)}{\partial a}\frac{\partial \mu(s|\theta)}{\partial \theta}$$

which in the code corresponds to: actor.train(states, grads).

In the actor.train() method, I fail to see where $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ gets multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$.

I did read:

self.params_grad = tf.gradients(self.model.output, self.weights, -self.action_gradient)

where self.action_gradient corresponds to $\frac{\partial Q^{\theta}(s,a)}{\partial a}$, and tf.gradients(self.model.output, self.weights) corresponds to $\frac{\partial \mu(s|\theta)}{\partial \theta}$, but I see no  multiplication.

Where does $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ get multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$?


texts:
I strictly need to use the summarise_at to compute a weighted mean, with weights based on the values of another column

    df %>% summarise_at(.vars = vars(FACTOR,tv:`smart tv/console`),
                  .funs = weighted.mean, w=INVESTMENT, na.rm=TRUE)

It always shows the error: 'INVESTMENT' is not found.

I then tried with:

df %>%summarise_at(.vars = vars(FACTOR,tv:`smart tv/console`),
               .funs = weighted.mean, w=vars(INVESTMENT), na.rm=TRUE)

But in this case :  Evaluation error: 'x' and 'w' must have the same length.

Why is this? Am I doing anything wrong? Do you have hints to solve this issue? Thanks


texts: If the set of all possible states changes each time, how can Q-learning "learn" anything?

texts:
I found this resource that explains q-learning with a very simple example. Make it a 2D problem, a rectangle instead of a line, and it's still simple. The only difference is that now there are 2 more possible actions (up and down).

My question is: if the length and height of the rectangle are random, as well as the starting position and the location of the Treasure, how can the bot apply the knowledge acquired to the new problem? Is there an evolved version of q-learning for problems with dynamical-states?


texts: How does KNN handle categorical features

texts:
For a K nearest neighbors algorithm using a Euclidean distance metric, how does the algorithm compute euclidean distances when one(or all) of the features are categorical? Or does it just go by the most commonly occurring value among the neighbors?

So e.g. if the 2 features of 3 neighbors are age and gender with values:
age,gender=[ [20,M], [31,F], [23,M] ], and we need to pick the 2 nearest neighbors for a new observation [20,F], how do we convert the gender to a numeric scale to compute euclidean distances?


texts: Understanding Loss function and Learning Algorithm

texts: In Keras, when specifying a loss such as Mean absolute error, does it replace the cost function in the learning algorithm (Adam or SGD) with the mean absolute error? I'm new to ML and a bit confused on this aspect.


texts: Single vs Multiple deep learning networks for multi-label classification?

texts:
Given a machine reaches a broken state, there are potentially fixes that can to be applied to get the machine to run again. We'd like to know if, for the problem further defined below, with millions of datapoints, hundreds of features, and tens of labels, should we be looking at using a single deep neural network with multiple outputs, or create an ensemble of binary networks with human-selected features presumed better for their relevance? Is there a standard approach?

The state of the machine is captured in potentially hundreds of features, a good mix of continuous values and categorical data. We have millions of documented cases of machine states, and can identify when a machine had broke and what fixes were applied to get it running again. There are less than 40 fixes we are interested in, and we are considering a bucket for "other" fixes, and a bucket for "unfixable".

We are treating this as a multi-label classification, because it may take multiple discrete fixes (Fix-1, Fix-2...Fix-N) to get the machine up and running again. Not all features would be relevant to each fix, so the question we're wondering is whether each fix should have its own binary classification (each outputing a single value representing Fix-i or not Fix-i) network with what we think are the relevant features, or should we create one giant deep neural network with multiple labels, sigmoid over each fix (Fix-1, Fix-2...Fix-N). With the latter approach, should combinations of fixes be represented as their own label (Fix-1, Fix-2, Fix-1&2, Fix-3, Fix-1&3, Fix-2&3, Fix-1&2&3 ... etc).


texts: How to predict future sales based upon last year sales data?

texts:
I have past 1 year of sales data and now i want to predict sales for next coming year.
How can i predict it.


texts:
My Jupyter notebook's python kernel keeps dying when attempting to train an XGBoost logistic classifier. Previously, I have run all of the following code successfully. Presently, there are issues. First, I will show you the code chunk that I am able to run successfully:

import xgboost as xgb
xgtrain = xgb.DMatrix(data = X_train_sub.values, label = Y_train.values)       # create dense matrix of training values
xgtest  = xgb.DMatrix(data = X_test_sub.values,  label = Y_test.values)        # create dense matrix of test values
param   = {'max_depth':2, 'eta':1, 'silent':1, 'objective':'binary:logistic'}  # specify parameters via map

where my data is small:

X_train_imp_sub.shape
(1365, 18)

however, my notebook's kernel keeps dying on this chunk:

xgmodel = xgb.train(param,  xgtrain, num_boost_round = 2)                      # train the model
predictions = xgmodel.predict(xgtest)                                          # make prediction
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_true = Y_test,
                          y_pred = predictions.round(),
                          normalize = True) # If False, return # of correctly classified samples. Else, return fraction of correctly classified samples
print("Accuracy: %.2f%%" % (accuracy * 100.0))

As stated before, I have been able to run both chunks successfully before. I have shut down all other notebooks and restarted my computer without luck. I am launching jupyter through Anaconda Navigator on a Macbook Pro.

UPDATE:

To further debug, I used Terminal CLI and ran:

conda install runipy
runipy MyNotebook.ipynb

which was very helpful, because it produced the following error message that I could then Google:

OMP: Error #15: Initializing libomp.dylib, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/

I found this thread:

https://github.com/dmlc/xgboost/issues/1715

Which shows others having the same problem on MacOS. I also suspected the problem being linked to matplotlib somehow because running any cell with  matplotlib related code somehow causes the xgb.train to kill the kernel.


texts: How to train decision trees for class imbalance binary classification?

texts:
sklearn documentations says

Decision tree learners create biased trees if some classes dominate. It is therefore recommended to balance the dataset prior to
fitting with the decision tree.
Balance your dataset before training to prevent the tree from being biased toward the classes that are dominant. Class balancing can
be done by sampling an equal number of samples from each class, or
preferably by normalizing the sum of the sample weights for each class
to the same value.

I get the under-sampling and oversampling part, but what exactly does weighting do? How does this affect the ways the trees are created?
Also, what's the correct evaluation metric for such a problem? Since it doesn't product probability outputs, I know I can't use ROC/AUC/Precision/Recall etc... Is balanced accuracy the right one?
Thanks,


texts: What is the best Data Mining algorithm for prediction based on a single variable?

texts:
I have a variable whose value I would like to predict, and I would like to use only one variable as predictor. For instance, predict traffic density based on weather.

Initially, I thought about using Self-Organizing Maps (SOM), which performs unsupervised clustering + regression. However, since it has an important component of dimensionality reduction, I see it as more appropriated for a large number of variables.

Does it make sense to use it for a single variable as predictor? Maybe there are more adequate techniques for this simple case: I used "Data Mining" instead of "machine learning" in the title of my question, because I think maybe a linear regression could do the job...


texts: How can I identify the most predictive factors?

texts: I've been playing around with bagged trees and random forests. How can I tell what factors most influenced the categorization? Will scikitlearn just spit it out, or is it trickier than that?


texts: Imbalanced training set vs smaller balanced training set?

texts:
Say I am using a maximum likelihood approach and my output unit computes a softmax function. My training set is distributed as follows over 6 classes:
class_samples[0]=23, class_samples[1]=5, class_samples[2]=44,
class_samples[3]=14, class_samples[4]=19, class_samples[5]=31

What should I do?

use the training set as given above with a normalizing weight balancing(e.g. using sklearn.utils.class_weight.compute_class_weight).

or should I simply use the minimum number of samples in a class(i.e. 5) to extract a balanced distribution of examples?

Why should I choose one over the other? Intuitively, I would think that using as many training examples as possible is the better option. However, I have tried to do some computations but I fail to show that usage of all examples with a normalizing weight balancing is better.
I have of course tried to do some heavy research but for some reason I cannot find the answer. If you know a good article, I would accept a reference as an answer, just as I would accept a "self-made" answer!


texts: Model Predictive Power and better prediction of 1 or 0 in Python scikitlearn?

texts:
I have 2 questions about Logistic Regression model in Python scikit-learn:

Which statistic can show me model predictive power ?
Which statistic can show me whether my model better predict event 1 or event 0 ?


texts: Relu with not gradient vanishing function is possible?

texts:
I'm beginner in ML.
In the ANN, relu has the gradient of 1 in x>0

how ever, i wonder in x=<0 relu has gradient of 0 and may have gradient vanishing problem
in deep neural networks.

if activation function like y=x(for the all the x) has no gradient vanishing problem,
why we dose not use this function in deep neural networks?
Is there any side effect for y=x(for all x)? (maybe, the weight may go infinity in deep neural networks...... however, I think this problem is also being happen in ReLU. so it is not a problem(I think.))


texts:
I was working with continuous system RL and obviously stumbled across this Policy Gradient.

I want to know is this something like cost function for RL? It kinda gives that impression considering we are finding out how efficient the system is as a whole (weighted sum of rewards multiplied by all the policies).

Let's take the example of vanilla PG:

$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)}{(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$

Here, the gradient is nothing but the expected value of the Return (which is nothing but the discounted sum of all the reward) multiplied by the how the policy (the network output) needs to change according to the network weights.

This seems similar to a cost function where we use the total error using cross-entropy (something similar to the information given by return) and then we use this to see how the weights of the neural network can be changed through backpropagation.

Let me know if I've got this right.


texts: How to correctly calculate average F1 score, precision and recall of a Named Entity Recognition system?

texts:
My Named Entity Recognition (NER) pipeline built with Apache uimaFIT and DKPro recognizes named entities (called datatypes for now) in texts (e.g. persons, locations, organizations and many more).

I have a gold corpus and a result corpus which I want to calculate precision, recall and F1 score on. As of now, I calulate these metrics like this:

1. Calculate precision, recall and F1 score from TP, FP and FN per datatype per document
2. Average precision, recall and F1 score per datatype for all documents

In the table you can see the results of step 2 in the corresponding datatype rows.

Regarding step 2: I think the way of calulating the F1 score is neither macro nor micro. I calculate precision and recall in the macro way (like explained here). But I don't calculate the F1 score as the harmonic mean of the average precision and recall (macro way), but as the average F1 score for every datatype for all documents. I am getting higher results for the macro-way F1 score compared to the way I am doing it at the moment.

Question: What is the right way to calculate the average F1 score for every datatype? Both ways seem to be correct to me. Please name sources for your answers.


texts: What is the best file format to store an uncompressed 2D matrix?

texts: For what it's worth my particular case is a symmetrical matrix, but this question should be answered more generally.


texts: Ethically and Cost-effectively Scaling Data Scrapes

texts:
Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models.

For instance, the Data Science Toolkit (or RDSTK for R programmers) allows me to pull lots of good location-based data using IP's or addresses and the tm.webmining.plugin for R's tm package makes scraping financial and news data straightfoward. When going beyond such (semi-) structured data I tend to use XPath.

However, I'm constantly getting throttled by limits on the number of queries you're allowed to make. I think Google limits me to about 50,000 requests per 24 hours, which is a problem for Big Data.

From a technical perspective getting around these limits is easy -- just switch IP addresses and purge other identifiers from your environment. However, this presents both ethical and financial concerns (I think?).

Is there a solution that I'm overlooking?


texts: Alternatives to regression to decide weights in an expression

texts:
I have a use case in which I am required to predict variable $y$ which depends on five variables $x_i$. Consider something like

$$ y=w_1 x_1+ w_2 x_2+ w_3 x_3+ w_4 x_4+ w_5 x_5.$$

This expression doesn't necessarily  need to be linear. I applied regression (linear, ridge, lasso, ARD) but I am not getting any good results for example, the original $y= 200$ but the predicted $y= 1$ or $2$.

Is there any other ML approach to train a model which takes as an input five variables and predict a new 6th variable?


texts: How to extract sub sentences from sentence mentioning a particular subject?

texts:
I am trying to solve an NLP problem. For a given sentence like :

"The Pasta was delicious, the Pizza was average"

I want to extract the sentiment attached to food items. Having built my own NER model, I am able to extract Pasta and Pizza and hence the sentences containing them. But using a sentiment analyser on the entire sentence would be wrong in this case

EXPECTED OUTPUT:

Pasta - Good score
Pizza - Average score

CURRENT OUTPUT:

Pasta - Kinda Good Score
Pizza - Kinda Good Score

I know I am getting this output because I am considering the same sentence for getting the sentiment attached to both the subjects in hand.

Is there a way to extract the sub sentence like "Pasta was good" and "Pizza was average" to associate sentence with each item instead of whole sentences which I am currently doing ?


texts: How to plot multiple variables with Pandas and Bokeh


texts: How do I extract these vocal features using Python?

texts:
I'm creating a model to predict, based on several acoustic features, the probability that a person has a certain disease, using this dataset: https://archive.ics.uci.edu/ml/datasets/parkinsons. These are the features:

MDVP:Fo(Hz) - Average vocal fundamental frequency

MDVP:Fhi(Hz) - Maximum vocal fundamental frequency

MDVP:Flo(Hz) - Minimum vocal fundamental frequency

MDVP:Jitter(%),MDVP:Jitter(Abs),MDVP:RAP,MDVP:PPQ,Jitter:DDP - Several measures of variation in fundamental frequency

MDVP:Shimmer,MDVP:Shimmer(dB),Shimmer:APQ3,Shimmer:APQ5,MDVP:APQ,Shimmer:DDA - Several measures of variation in amplitude

NHR,HNR - Two measures of ratio of noise to tonal components in the voice

RPDE,D2 - Two nonlinear dynamical complexity measures

DFA - Signal fractal scaling exponent

spread1,spread2,PPE - Three nonlinear measures of fundamental frequency variation

Problem is, I have no idea what these mean. I need to be able to extract them from a voice recording. Does anyone have any libraries or resources pertaining to these specific features?


texts: pandas - under a column, count the total number of a specific value, instead of using value_counts()

texts:
value_counts()

function outputs the number of all unique values in a column, for example

 apple   3
 orange  2
 banana  1

I want to search the total number of (value = 'apple') only, which function can replace value_counts()?


texts: Applying GradCam to video classification models

texts:
In the original paper, it says that GradCam visualization can be applied to any convolution based model. The problem is stated for convolutions that process images. In my case, I am classifying videos so therefore I should apply GradCam to every frame individually by calculating the gradients with the loss of the entire video (At least that's how I think).
The problem is that I am using different models to experiment, such as a ConvLSTM. These use convolutions in each LSTM gate, and though I return intermediate results of each frame, these are maxpooled when passed to the next layer so I cannot get activations corresponding to each frame.
But I also work with a model that uses MobileNet to feature extract each frame, and pass that to a GRU network. In this case mi approach should work?
I am nothing attaching example code because I believe this to be a theoretical question, but if need be I will.


texts: Why doesn't loss go down during Neural Net training?

texts:
I am working on a Kaggle competition and have tried 2 different code approaches and have the same issue: the loss is large (18247478709991652.0000) and does not go down or is nan.

I'm not sure if there is something wrong with the code or with the data. I tried both scaled and non-scaled data and got the same results. I tried it with the full data set (3,000 examples) and an abbreviated data set.

Here is the abbreviated data.

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

dataframe = pandas.read_csv('data/tmdb/train_processed.csv')
dataframe.drop('id', axis=1, inplace=True)

Y = dataframe['revenue'].values
dataframe.drop(columns=['revenue'], inplace=True)
X = dataframe.values

def baseline_model():
  model = Sequential()
  model.add(Dense(13, input_dim=3, kernel_initializer='normal', activation='relu'))
  model.add(Dense(1, kernel_initializer='normal'))
  model.compile(loss='mean_squared_error', optimizer='adam')
  return model

seed = 7
numpy.random.seed(seed)

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Result: %.2f (%.2f) MSE" % (results.mean(), results.std()))


texts: Automating the training and deployment of the model as more data becomes available

texts:
My question is about automating the training of the model as more data becomes available.
In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ultimately deploy the model.
As more data becomes available on a daily basis, is it common in real-life projects or even advisable to automate the process above, as to leverage the new data and potentially make the model more accurate?


texts: Are there pretrained models on the ImageNet Bounding Boxes dataset?

texts: When searching for some pretrained models for object detection with bounding boxes, I was wondering if there are also pretrained models on the ImageNet dataset for bounding boxes. Most of the time, I find models trained on the VOC or COCO dataset. Is there a reason for this? I'd like to have a model that is pretrained on many categories. If I saw correctly, the ImageNet dataset is much richer than the VOC or COCO dataset.


texts:
As part of big data analysis project, I'm working on,

I need to perform PCA on some data, using cloud computing system.

In my case, I'm using Amazon EMR for the job and Spark in particular.

Leaving the "How-to-perform-PCA-in-spark" question aside, I want to get an understanding of how things work behind the scenes when it comes to calculating PCs on cloud-based architecture.

For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.

When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.

How then is the covariance matrix calculated in such case when each node have only partial data?

This is just an example. I'm trying to find some paper or documentation explaining all this behind-the-scenes voodoo, and couldn't find anything good enough for my needs (probably my poor google skills).

So I can basically summarize my question(s) \ needs to be the following:

1. How distributed PCA on cloud architecture works

Preferably some academic paper or other sorts of explanation which also contains some visuals

2. Spark implementation of D-PCA

How does Spark do it? Do they have any 'twist' in their architecture to do it more efficiently, or how does the RDD objects usage contribute to improving the efficiency? etc.

A presentation of even an online lesson regarding it would be great.

Thanks in advance to anyone who can provide some reading material.


texts:
Zero-shot learning used to predict the unseen classes using the attributes for each image.

For example, we feed the model pictures of horses and pandas. The model will learn how horses look like and what the difference between different colors. For the testing phase, if we give the model a picture of a zerba, it will be able to check the unseen classes and predict it correctly.

Recently, I got to know that Zero-shot can be used on text. But I am not sure how to create attributes for text and how does it work. There is not any helpful posts or links for Zero-shot learning on text.

Would you please explain it in a simple example or provide a helpful link for an article or a model for zeroshot learning on text.


texts: Is is possible to see the input labels for a randomforest model?

texts:
I have a random forest model built like this:
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
model = RandomForestClassifier(random_state=0)

X_train,X_test,y_train,y_test = train_test_split(X, y, test_size = 0.2, random_state=42, stratify=y)

model.fit(X_train, y_train)

Is there any method I can call (or attribute I can use) to see the column names used to make the  model?
Thanks


texts:
I'm a newbie in data science. I'm working on a regression problem. I'm getting 2.5 MAPE. 400 MAE 437000 MSE. As my MAPE is quite low but why I'm getting high MSE and MAE?  This is the link to my data

from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import Normalizer
import matplotlib.pyplot as plt
def mean_absolute_percentage_error(y_true, y_pred):
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100

import pandas as pd
from sklearn import preprocessing

features=pd.read_csv('selectedData.csv')
import numpy as np
from scipy import stats
print(features.shape)
features=features[(np.abs(stats.zscore(features)) < 3).all(axis=1)]
target = features['SYSLoad']
features= features.drop('SYSLoad', axis = 1)
names=list(features)

for i in names:
    x=features[[i]].values.astype(float)
    min_max_scaler = preprocessing.MinMaxScaler()
    x_scaled = min_max_scaler.fit_transform(x)
    features[i]=x_scaled

Selecting the target Variable which want to predict and for which we are finding feature imps

import numpy as np
print(features.shape)
print(features.describe())
from sklearn.model_selection import train_test_split
train_input, test_input, train_target, test_target =
train_test_split(features, target, test_size = 0.25, random_state = 42)
trans=Normalizer().fit(train_input);
train_input=Normalizer().fit_transform(train_input);
test_input=trans.fit_transform(test_input);

n=test_target.values;
test_targ=pd.DataFrame(n);

from sklearn.svm import SVR
svr_rbf = SVR(kernel='poly', C=10, epsilon=10,gamma=10)
y_rbf = svr_rbf.fit(train_input, train_target);
predicted=y_rbf.predict(test_input);
plt.figure
plt.xlim(300,500);
print('Total Days For training',len(train_input)); print('Total Days For
Testing',len(test_input))
plt.ylabel('Load(MW) Prediction 3 '); plt.xlabel('Days');
plt.plot(test_targ,'-b',label='Actual'); plt.plot(predicted,'-r',label='POLY
kernel ');
plt.gca().legend(('Actual','RBF'))
plt.title('SVM')
plt.show();

test_target=np.array(test_target)
print(test_target)
MAPE=mean_absolute_percentage_error(test_target,predicted);
print(MAPE);
mae=mean_absolute_error(test_target,predicted)
mse=mean_squared_error(test_target, predicted)
print(mae);
print(mse);
print(test_target);
print(predicted);


texts: Is there an point to using LassoCV with `cross_val_score`

texts:
I've seen some jupyter notebooks that seem to combine LassoCV with cross_val_score, and I'm confused what the point is.
Usually the working flow looks like:
alpha = [some array of alpha values]
lasso = LassoCV(cv=5, alphas=alphas)
cross_val_score(lasso, X_train, y_train ,cv=5)

What is the point of using cross_val_score if there's already CV embedded within LassoCV?


texts:
I am just starting getting into deep learning with tf.keras. I am at the point where I have to decide where I want to develop. The thesis project will be timeseries prediction. My options are PyCharm on Macbook Air M1 2020 or a 2013 4th Gen Intel i5 Linux desktop and Google Colab (please let me know if there are others).
So the question I have now is which one is faster/better suited for my puropses. M1 got hyped a lot so I thought the M1 would savage my desktop (and acutally the hype biased my purchase decision), but well its only slightly better (like 1.2-1.5x faster in my cifar10 benchmark) and I wonder if its worth the effective 1-2 GB of RAM left on MacOS vs the ~14 GB on my Linux machine. Further there is Colab and I can't really tell which one will win the race, since Colab limits resources by demand but also allows distributed fit on cloud TPUs, which would introduce some extra coding efforts. Then again I have to say: so does ML on Apple Silicon, which comes with a handful of limitations, a peculiar MiniConda setup, a lot of issues (also severe ones, like training errors etc., problems which I would not even recognize) which are actually not really being worked on.
Is from the perspective of professional data scientists, which I hope to find here, a clear indication on which I should choose?


texts: Is the ultimate challenge in ML simply computational power?

texts:
I am stuck on a theoretical roadblock in learning about machine learning, because I have not seen this explicitly addressed anywhere. In my studies, it seems as if Cross-validation (or some variant thereof, like LOOCV, or potentially another, but similar, validation scheme like bootstrapping) is the be-all-end-all of model selection. Choosing models and their parameters via exhaustive CV to maximize fit but also balance overfitting seems the optimal way to create models, and computational power is only getting cheaper. So what is there left to do for the human analyst?

I apologize in advance for this amateurish question, but could anyone fill in this gap for me, and potentially suggest some sources on model selection?


texts: Understanding a Neural Network with Keras (preferably), TensorFlow or PyTorch

texts:
I would like to try a technique I saw in an article a while ago where, in order to understand what each neuron is doing, you apply specific inputs to the net and see which one of them is most activated.

Is there a simple way to do it with Keras (preferably)? I can take the weights of each neuron after the NN has been trained and use forward (predict) functions to activate it, but how would I get the activation result of each individual neuron? Is there a function for that?

TensorFlow and PyTorch offer more low level functions so I could do the same thing if I programmed the NN almost from scratch, but even so, does anyone know a simpler way?

If you have tips for this task also, they would be more than welcome. I don't know if functions for this kind of task already exist, but, if they don't, I would say it would be worthwhile creating them, people on the market have complained often to me that the main reason why they are not yet using NNs is that they don't know how to understand the NN's thinking.


texts:
AFAIK, Non-Negative Matrix Factorization (NMF) is the procedure of looking for matrices $A$ and $B$ such that

$$Data_{ik} = \sum_j A_{ij} B_{jk}$$

My data matrix is in fact 3D. I would like to fit the following model to my data

$$Data_{ikl} = \sum_j A_{ij} B_{jk} C_{jl}$$

It would be great to know if this model has a name and is already implemented somewhere (preferably Python)


texts:
I am using GridSearchCV to find the best combination of parameters for SVM. However, the parameters chosen by GridSeasrchCV do not seem to be the best ones. I tried some parameters randomly and they are working much better. I tried both combinations in the test set. Here is the picture:
The second confusion matrix + classification report is the random but improved one. Why isn't GridSearchCV selecting this combination?


texts: How to "reshape" into square matrix for numpy.linalg.solve()?

texts:
I'm trying to find the intersection of lines $y=a_1x+b_1$ and $y=a_2x+b_2$ using numpy.linalg.solve(). What I can't get my head around is how to correctly make $A$ a square matrix for solve() to work. I'm familiar with solving linear equation systems, but there's something here I don't get.

What I'd like to do is:

def meeting_lines(a1, b1, a2, b2):
    a = np.array([[a1], [a2]])
    b = np.array([b1, b2])
    return np.linalg.solve(a, b)

def main():
    a1=1
    b1=4
    a2=3
    b2=2

    y, x = meeting_lines(a1, b1, a2, b2)

Where I expect $y=-3$ and $x=1$. However, this fails with numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square.

Thank you very much for your help, trying to figure this out has messed up my day already!


texts: What is an intuitive explanation for the log loss cost function?

texts:
I would really appreciate if someone could explain the log loss cost function And the use of it in measuring a classification model performance.

I have read a few articles but most of them concentrate on mathematics and not on intuitive explanation and also a basic implementation using python with a small dataset would really helpful, So I can understand it better.

It would really help many looking for the same here. Thanks.