texts: Decision Tree: Efficient splitting of nodes, minimize number of gini evaluations

texts:
I have a dataset specific problem where i need to use a splitting function other than gini_index. This requires me to re-write a decision tree from scratch. I have a working model, but itis highly inefficient.

To make a split i currently iterate though each feature and then through each unique datapoint in that dataset for each node (total of nodes x features x unique levels gini evaluations). Cause of this my DT on a 300k X 145 dataset has been running for 2 days.

How can I cut down on the number of splitting evaluations, or speed up the program. I read Fisher Yates algorithm in Sklean's code, but I don't understand the logic. Any help would be appreciated.


texts: Is it bad to have a large gap between training loss and validation loss?

texts:
Say my training loss is 0.5 and my validation loss is 2.5 (both have stopped decreasing, validation loss never increased). I am clearly overfitting. If I add regularization, my training loss becomes 1 and validation loss 3.5.

The first model clearly has better validation loss, and the second model overfits less.

Which model should be selected? Is it even possible that a model that overfits more performs better on unseen data or does this mean that there is data leakage of some sort?

The problem here is that I'm not doing simple things like image classification but trying something more complicated and I'm not finding a lot of resources on overfitting on problems that have not yet been solved. Mostly it's disussions of binary image classification where it's possible to find models that get a very good validation error, so the explanations don't apply to my problems.


texts: How to count the number of missing values in each row in Pandas dataframe?

texts:
How can I get the number of missing value in each row in Pandas dataframe.
I would like to split dataframe to different dataframes which have same number of missing values in each row.

Any suggestion?


texts: What is done first, cross validation or grid search?

texts:
When I have the data set to train a model with SVM, which procedure is performed first, cross validation or grid search? I have read this in a couple of books but I don't know in what order all this should be done. If cross-validation is first performed, what hyperparameters do I use there if I have not found the optimal values provided by the grid search? In addition, throughout this procedure, where should the confusion matrix be calculated?

Thanks in advance


texts: What are the top schools for online part-time Masters in data science/analytics programs? Is it much better to do the program in person?

texts: I am looking to get my Masters in Science in Data Science or Analytics. I will continue working during the program so it needs to be part-time. I would prefer to do an online course, but if in-person is much better, I could be open to it. So far, I found Georgia Tech and Johns Hopkins for online, part-time programs. For in-person, I could attend Georgetown part-time. Any insight you can provide regarding which of these programs would be best, if there is another school that could be better for my situation, or if any school is essentially the same would be very helpful! Thanks!


texts:
I have two dataset A and B. We have 93 persons in A and 88 in B.

Dataset A contains number of person who had made a payment. So A is

number of payments      number of persons made that payment
1                       93
2                       41
3                       35
4                       28
5                       20
6                       14
7                       7
8                       4

Similar way with dataset B.

My question is this: How can I calculate if A has a higher performance than B, ie if we have significant more person in A who make a payment compared to B.


texts: How can I do tree_method ='exact' in XGBoost classifier?

texts:
I am doing XGBoost classification on a huge data set and its showing:

Tree method is automatically selected to be 'approx' for faster speed. to use old behavior(exact greedy algorithm on single machine), set tree_method to 'exact'

How can I shift it to be exact?

I have tried clf = xgb(tree_method='exact') and it returned error!


texts: What is the best algorithm for deterministic belief propagation?

texts:
Here is a simple example:

In a 3D space, if point A is the geocenter of a planet, point B is its north pole, and point C has a fixed latitude/longitude on the planet surface. Then the position of point C can be inferred from the position of A and B (using geodetic projection).

Now assuming that I have thousands of points in the space, their incomplete relations are in a big multipgraph. I would like to deduce the position of a point that is initially unknown. What is the best algorithm to search search through all its relations and try to deduce it?


texts: What to choose for null hypothesis and alternate hypothesis?

texts:
In this article which explains types of t-tests
for “Independent Two-Sample t-test” example:

We can confirm that the t-statistic is again less than the t-critical
value so we fail to reject the null hypothesis. Hence, we can conclude
that there is no difference between the mean screen size of both
samples.

Null hypothesis was that the average screen size of the sample does not differ from 10 cm.
Alternate hypothesis was that the average screen size differs. And also the data is only about sizes of screens.
Why can't null hypothesis be that the average screen size differs. and alternate hypothesis, that the average screen size doesn't differ.
Not just this example, in every article I have read, every author only told what was chosen for null and alternate hypothesis. But, no one talked about why they chose what they chose. I always think that, why can’t the statement chosen for alternate hypothesis be chosen for null hypothesis and vice versa. And the data won’t have much information too. It’s always up to the article writer to what to choose for null and alternate hypothesis.
So, my question is, are there some set of rules, that have to be followed to choose a statement/condition for hypothesis. Or what actually causes a particular statement to be chosen for hypothesis. Or am I missing something.


texts: API to find out how many executors are running my Spark jobs?

texts: I'm writing code that captures statistics from Spark runs and stores the results in XML files. I'd like to capture, ideally with the REST API, information about the number of executors and the number and size of the Data Frames that were created. Is there an easy way to do this? What are the API calls?


texts: Tensorflow neural network TypeError: Fetch argument has invalid type

texts:
I am making a simple neural network using tensorflow, with data I collected myself, however, it's not cooperating :P I have run into an error that I can't fix or find the fix for and I would love your help.
The errormessage:
TypeError: Fetch argument 2861.6152 of 2861.6152 has invalid type <class 'numpy.float32'>, must be a string or Tensor. (Can not convert a float32 into a Tensor or Operation.)

The error refers to the following line in my code:
_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})

I have already figured out that the error doesn't occur when I comment out the following lines in my code:
prediction = neural_network_model(champion_data)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(prediction, item_data))
optimizer = tf.train.AdamOptimizer().minimize(cost)
_, cost = tf_session.run([optimizer, cost], feed_dict = {champion_data: batch_input, item_data: batch_output})

So somewhere one of these lines is getting something that doesn't look exactly like it expects it to look. I have already tried the obvious (removing the np.array() from batch_input and batch_output or replacing it by list()) but that doesn't solve the problem. My current hypothesis is that the output of neural_network_model(champion_data) is somehow of the wrong shape or type, however I'm not sure how to test that or how to solve it if that turns out to be the case.
The full code can be found here.
Edit: I have verified that the champion data that are input into the neural_network_model, the prediction and the cost are all tensors. I've been trying to solve the problem using the hypothesis that the problem somehow lies in the feed_dict = {} part of the code, but not getting anywhere so far


texts: How to use InceptionV3 without retraining

texts: I know I can retrain Inception to label images, but can I just provide an image to an untrained Inception and get a label back? Is there an API for this? If there is, then why does Google open source this if they're trying to make money off of google cloud vision?


texts:
I have a pre-trained tensorflow model for image classification. It has has some convolution and maxpooling layers followed by dense layers.

I would like to split it in two pats. The first containing the convolutions/maxpooling part, the second containing the dense layers. Then I would like to feed the first part with an image, store the result into a file/variable and then using it as input for the second part.

The idea is that I could encode on the fly images with the first part, save them to disk and then using the encoded files as input. In this way I can avoid storing the original images.

I am writing an example of a whole model (based on Coursera's/deeplearning.ai "Introduction to TensorFlow for Artificial Intelligence, Machine Learning, and Deep Learning")

import tensorflow as tf
mnist = tf.keras.datasets.fashion_mnist
(training_images, training_labels), (test_images, test_labels) = mnist.load_data()
training_images=training_images.reshape(60000, 28, 28, 1)
training_images=training_images / 255.0
test_images = test_images.reshape(10000, 28, 28, 1)
test_images=test_images/255.0
model = tf.keras.models.Sequential([
  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),
  tf.keras.layers.MaxPooling2D(4, 4),
  tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
  tf.keras.layers.MaxPooling2D(2,2),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()
model.fit(training_images, training_labels, epochs=5)

In the example above,

first_part would have the first 5 layers
second_part would have the last two (and possibly an input layer)

Thank you.


texts: Grid Search on Unsupervised Sklearn Clustering?

texts: I am trying to use clustering algorithms in sklearn and am using Silhouette score with cosine similarity as a metric to compare different algorithms. My question is due to the varying hyperparameters of the different clustering algorithms is it possible to run some type of grid search on these algorithms in order to find the best set that optimized Silhouette score. I found this PR on Github https://github.com/scikit-learn/scikit-learn/pull/6160 but it looks like work on it died off.


texts:
The definition of information entropy is defined below:

This looks fine but I got no intuition why it is defined this way. Could any one share their ideas on this? Thanks!


texts: how can i plot a stacked bar chart by year with two values(columns)?

texts:
not sure if this is correct medium, but i am having trouble plotting this simple data:

how do i get the graph below - aplogies for this poor picture.
basically i want it stacked with the risk label being the legend, but the predicted and actual side by side for each year.. how can i achieve this in pandas? btw this is a pandas dataframe
e.g.
df_partial = df[['year','risk','predicted','actual']]
grouped = df_partial.groupby(['year','risk']).sum()[['predicted','actual']]
grouped.plot(kind='bar')

it does not give me what i would like to see
import plotly.graph_objs as go
df = df.melt(id_vars=["year", "risk"], value_vars=["predicted", "actual"])
fig = go.Figure()

for risk in ["low", "medium", "high"]:
    tmp_df = df.query(f"risk == '{risk}'")
    fig.add_trace(
        go.Bar(
            x=[tmp_df["year"], tmp_df["variable"]], y=tmp_df["value"], name=risk,
        )
    )

fig.update_layout(barmode="stack", height=400, width=800, margin=dict(b=60, t=20, l=20, r=20))
fig.show()

however i get :

how do i remove these faint lines? i've tried opacity does now work


texts: Can I use fit() function with images in Keras?

texts:
I want to use vgg16 to train a dataset that contains images.

Can I use fit() function instead of fit_generator() in Keras? How?


texts:
I've developed an algorithm to define conditional entropy for feature selection in text classification. I'm following the formula at Machine Learning from Text by Charu C. Aggarwal (5.2.2). The author mentions that Conditional Entropy values are between (0, log(number of classes)) in which my case is (0, 0.6931472).
The author also mentions that features with the largest values can be removed, but he don't give further information about the criteria to define 'largest' (is it only the max value of entropy or a set of the largest entropy values?)
Have you ever guys applied Conditional Entropy for feature selection? If so, based on results, what criteria was used to define features to be removed.
Here a summary of my Conditional Entropy results:
        E.tj.
 Min.   :0.5701
 1st Qu.:0.6562
 Median :0.6563
 Mean   :0.6558
 3rd Qu.:0.6564
 Max.   :0.6564


texts: Extract datatable column by indexing a character vector

texts:
I am asking a similar question to this one except I am trying to index a vector of column names.

If I have a data.table like a:

a <- data.table("NAME" = c("A", "B", "A"),
                "PASS_FAIL" = c("F", "P", "P"),
                "TEST_COUNT" = c(NA, NA, 1))

And a vector of column names:

col_names <- c("NAME", "PASS_FAIL", "TEST_COUNT")

Then try to index with this syntax:

a[col_names[1]]

I get this error:

Error in `[.data.table`(a, col_names[1]) :
  When i is a data.table (or character vector), the columns to join by must be specified
  either using 'on=' argument (see ?data.table) or by keying x (i.e. sorted, and, marked as
  sorted, see ?setkey). Keyed joins might have further speed benefits on very large data
  due to x being sorted in RAM.

I also tried this a$col_names[1] which results in NULL (but since it is a data.table this is the wrong syntax.

Calling just col_names[1] gives a character "NAME" which is what I expected.

Any ideas on what I am doing wrong here? Thanks!


texts: Effect of removing duplicates on Random Forest Regression

texts:
I have a dataset with several million samples that have 5 features and 1 target, which I am using for a regression model.  With very large sample counts some models (like Random Forests) become very large (several GB when pickled).
These data often have duplicates or near duplicates - these are real observations - but the measured values are just coincidentally identical (a consequence of the limited input range and precision of the instruments).
What is the effect (in theory) or removing duplicates on model accuracy?


texts: Are there any deep models that are better than BERT-CRF in NER task?

texts:
Named entity recognition (NER) is task that mark tags of the input text sequence.
BERT-CRF is a good NER model.
I want to find a better NER model. Or I want to improve the BERT-CRF model.
What can I do?


texts: Create a graphical viz of list of elements residing in a column in desired order

texts:
I have a list of elements in a column. Example:

UID.................Flow

1............................qwerty, asdfgh, zxcvbn, poiuyt, lkjhgf, mnbvcx

2............................qpwoei, alskdj, zmxncb, qmwneb, zpxoci, glafhk

3............................uinbwe, cvbnfg, sfjvlr, sdfvll, sfvlas, fogjwl

UID and Flow are the column names.

Now for each UID, I want to create a graph which shows the flow like

1 - qwerty -> asdfgh -> zxcvbn -> poiuyt -> lkjhgf -> mnbvcx

Remember, if any element is repeated, it should point back to it, unlike just putting arrows in between all the elements.

So ultimately, for every UID, I should be able to see this graph which shows me the flow.

Can someone help me build the code for this? Or is there any package which we can use to create graphical viz like these? Thanks in advance.

-- Anirudh


texts:
I would like to build a scorecard from scratch.

Currently I have been working on data cleaning, and I created a dataset for training and another for test.
What I have is a dataset of around 700 rows (maybe not so many as I could expect; it is for a computer science project), 20 columns. The target is if the borrower has defaulted or not (indicator). Most of the factors are categorical or Boolean.
I am assigning a score based on these parameters, specifically:

- if the account has defaulted in the last 90 dats, then assign 60;
- if the account has defaulted in the last 30 days, then assign 50;
- if the account has not defaulted in the last 90 days, then assign 80;
...

I would like to know if this is a good approach and/or if there is an algorithm to assign scores to the accounts or if I should do something different.
I do not know very well the logic behind a scorecard, so any information would be extremely useful.

Thanks


texts: Creating a new Dataframe with specific row numbers from another

texts:
I've found other posts that refer to creating a new dataframe using specific conditions from another (like ID = 27, etc.) but nothing that allows me to make a new dataframe from specific row numbers of another.
Here is what I have tried so far:
To add a single row to a new dataframe:
test.append(original.loc[300])
test

To add a list of rows to a new dataframe:
entries_to_move = [10, 20, 30]
for i in entries_to_move:
   test.append(original.loc[i])
test

Neither method works, so help would be appreciated. The output for either code is just a __.
Thank you!


texts: In supervised learning, are more data entries always better?

texts:
I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 entries as they contained significant missing data. When I redo the analysis except use mean, median, and mode imputation methods to fill in the missing data entries and repeat the training/testing - my accuracy drops by 4%.
Why is this? I thought more data would be better or at the least stay the same. Does this imply the imputation was inaccurate? Or perhaps I was biasing the sampling process by only selecting records with no missing entries the original time? How could I possibly know which case since I have no way of knowing how accurate the imputed values are with the true values that were missing?


texts: How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

texts:
Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me to build a model that can generalize to new data, specifically new job descriptions that may be very different from the already labeled job descriptions.

What I have done so far (in Python):

(1) Trial 1: I tried the classic bag-of-words + TfidfVectorizer + binary classification (LogisticRegression specifcally) approach. With parameter tuning based on the train + valid sets, precision and recall on the test set could reach over 98%. However, after I built the model, I collected a new set of documents (again, some are job postings and others are not), and asked annotators to label these new data. I then used the model to classify these new docs. Perhaps expectedly, the precision and recall dropped pretty drastically to, ~65% and ~40%.

(2) Trial 2: I thought perhaps using tfidf alone might cause some overfitting, so I tried applying TruncatedSVD (with 100 components) on the TfidfVectorizer (bag-of-words + TfidfVectorizer + TruncatedSVD + binary classification).  Again, I tuned the parameters using the train and valid sets, and the precision and recall were similarly high on the test set. When I applied this new model to the newly collected data, the recall improved a bit, from 40% to 50%, but the precision dropped a bit, to 58%.

So my question is: is there a way to leverage the initial set of coded documents to build a kind of model that can generalize better to new data set. My specific interest is identifying job descriptions/postings. It doesn't have to be a binary classification task since the non-job-descriptions can be much more varied than job descriptions/postings


texts:
I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically relevant terms for "kw1" using word2vec. For example if the keyword is "apple" I would expect to see related terms such as "ipad" "os" "mac"... based on the input file. So this set of related terms for "kw1" would be different for each input file as word2vec would be trained on individual files (eg., 5 input files, run word2vec 5 times on each file).

My goal is to find sets of related terms for each input file given the common keyword ("kw1"), which would be used for some other purposes.

My questions/doubts are:

Does it make sense to use word2vec for a task like this? is it technically right to use considering the small size of an input file?

I have downloaded the code from code.google.com: https://code.google.com/p/word2vec/ and have just given it a dry run as follows:

 time ./word2vec -train $file -output vectors.bin -cbow 1 -size 200 -window 10 -negative 25 -hs 1 -sample 1e-3 -threads 12 -binary 1 -iter 50

./distance vectors.bin

From my results I saw I'm getting many noisy terms (stopwords) when I'm using the 'distance' tool to get related terms to "kw1". So I did remove stopwords and other noisy terms such as user mentions. But I haven't seen anywhere that word2vec requires cleaned input data...?
How do you choose right parameters? I see the results (from running the distance tool) varies greatly when i change parameters such as '-window', '-iter'. Which technique should I use to find the correct values for the parameters. (manual trial and error is not possible for me as I'll be scaling up the dataset).


texts: Retrieving follower and followee (friend) counts using Twitter API

texts:
I want to retrieve Twitter users' follower and followee (friend) count using Twitter API. (I  am reluctant with using a crawler since I need this data for research purposes) However, I was only able to find get_friend_list DOC which returns the whole list of followers.

I just want to get the number of followers and follweees not the while list of follower id (since this will require me to use more POST requests for Twitter API)

Could someone help me how to retrieve follower count with Twitter id (or username)?


texts: In model validation (log regression), can AUC for the test sample be higher than that for the train sample?

texts: I have a relatively simple model (with 8'000 cases, 5 predictor variables) predicting a dihotomised outcome. It has an AUC of 0.82 (95%CI 0.82-0.83). Prediction in temporal (N=240) and external validation (N=170) datasets had AUCs of 0.85 (95%CI 0.81-0.88) and 0.76 (95% CI 0.68-0.84%). A reviewer told me it is impossible (can't happen, theoretically) for the temporal validation dataset to have higher AUC. Is this true?


texts: Comparison of performance of autoencoder with PCA

texts: I am running PCA and autoencoder (2 hidden layer with relu) on a data. Both PCA and autoencoder give similar accuracy of the order 50%. I have tried different variations of autoencoder: changing layers, neurons,activations. But it did not perform any better. What can be the possible reason behind this. Can anybody suggest any analysis of data to identify the reason for autoencoder to not perform any better.


texts: Sklearn Pipeline for mixed features: numerical and (skewed) categorical

texts:
I am working on a dataset from Kaggle (housing price prediction). I have done some pre-processing on the data (missing values, category aggregation, selecting ordinal vs one-hot). I am trying to implement a pipeline to streamline the code. The pipeline consists of a ColumnTransformer with two components: one component contains a standard scaler applied to numerical and ordinal features; the second component has a one-hot encoder for the remaining set of features. I am passing this transformer to a GridSearchCV object to tune hyperparameters. In this case, it is a LASSO model. So, I am trying to tune the coefficient of the penalty term. The problem is some of the one-hot encoded features are highly skewed with the count in mostly one category. When GridSearchCV tries to run cross-validation, it raises an error saying that unknown categories are found while validating the model. I think this happens because while fitting the one-hot encoder the train set doesn't contain data points with specific labels that show up in the validation set. One obvious way to handle this would be to fit a one-hot encoder, keep it aside and then build a pipeline and carry on with the grid search (/validation) steps. This seems a bit disconnected to me considering the notion of pipeline was defined for exactly this purpose. Maybe I am missing something here. Is there a better (/efficient) way to achieve the above rather than separating the one-hot encoder from the pipeline?

For reference, the data for the above histogram,

Category   Counts
CompShg    1434
Tar&Grv      11
Tar&Grv      11
WdShngl       6
WdShake       5
Roll          1
ClyTile       1
Metal         1
Membran       1


texts: How to represent genre or artist name in a neural network

texts:
I am writing a music recommendation system using machine learning. I'm attempting to make sense of ensemble networks to allow the system to learn from both the content-based features, as well as the global 'meta' features, such as the genre, year, artist etc.
However, I do not know how I should represent the genre and artist information. Making them integers would imply some sort of order. If Kanye West is 24, what should 25 be? 26?
Or is there something fundamentally wrong with my approach?


texts: How to deploy machine learning models as a chrome extension?

texts: I have trained a stance detection model using SVMs. Wanted to know how can I deploy this as a chrome extensions. I do understand that the question is a bit broad but any links, suggestions etc. will be appreciated:)


texts: Search for redundant filters(channels) in CNN

texts:
When training a CNN one specifies in each layer the number of channels. In the input we have 1 channel for grayscale image and 3 for RGB image, and then usually the image resolution is decreased, whereas the number of channels increases (64, 128, 512).
These filters capture some patterns, useful for solving a particular task - classification, segmentation, detection, whatever. However, the resulting collection of numbers in the filter does not have a particular structure, and I may happen, and I suppose, mostly happens, that there is a lot of redundancy in the channels.
If one is interested to make the CNN as small as possible in order to use less memory or speedup inference procedure, what are the best procedures to eliminate the redundant, useless filters?
One possibility I see is to measure the correlation between different channels in ordet to determine, whether they are linearly dependent. But I suppose, that It can get help to get rid only of very limited number of channels.
Another possibility, I would think about us to perform variational dropout https://arxiv.org/pdf/1701.05369.pdf but a filterwise version to determine noisy channels. And then eliiminate them.
Third, since filters are $N$-dimensional arrays, there exist certain decompositions, namely HOSVD  https://en.wikipedia.org/wiki/Higher-order_singular_value_decomposition. But the layers are connected with each other, and truncation needs to be performed in some self-consistent way.
Which approaches turned out to be succesful in CNN and CV tasks?


texts: Making fake result in data mining using weka j48 algorithm

texts:
I have 130 records in one CSV dataset and I'm using j48 decision tree. I used the whole training set for testing, and the result was 79 correctly classified and 51 incorrectly classified records.
Now! I want a result between 95% - 100% and don't have ability to add or remove any record to those 130. But I'm allowed to play a little with my dataset, for example use 10 out of 130 in such a way that all of those 10 must be classified correctly.

The Quality of tree is not Important
The Number of test samples is not so important (10-15 is good)
The Only Important thing is Accuracy between 95-100 %

FIRST, I tried to move 10 correct samples to bottom of dataset and use "split by percentage" (92.30%) but it wasn't useful.

At SECOND, I tried to choose them by "try and error" and chose randomly, and I was just testing the last sample (130th), the I found some correct samples and keep it there, then tried another below the last successful sample and tested the last two lines, the result should have been 50% or 100%, but surprisingly was 0% (Total instants:2 correct:0 Incorrect:2).

Can anyone help me please?
I already thank a lot dear readers...


texts: What is the difference between "fit_transform" and "transform" methods when using "SimpleImputer"?

texts:
I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this case.

In the following code, when I am using fit_transform(X_valid) instead of transform(X_valid) it gives me different outputs.

from sklearn.impute import SimpleImputer

# Imputation
my_imputer = SimpleImputer()
imputed_X_train = pd.DataFrame(my_imputer.fit_transform(X_train))
imputed_X_valid = pd.DataFrame(my_imputer.transform(X_valid))


texts: On which step should use SMOTE technique for over sampling?

texts:
I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it.

My preprocessing steps are:

Missing values
Removing Outliers
Smoothing Data

Should I use SMOTE before all of these steps or its better to use it after these steps?


texts: Is it possible to use an array of graph coordinates as an input variable?

texts:
Say I have 1000 graphs that shows sales every year for the last 10 years for 1000 different companies. And say each of those graphs belong to either domestic countries or foreign countries.

Is it possible I could input the different graphs into a classifier? That is, could a model predict based on the graphs whether or not the country was domestic or foreign? If so, how would you do that in python or r?


texts: Cost Function for evaluating a Regression Model

texts:
There are several "classical" ways to quantify the quality of (any!) regression models such as the RMSE, MSE, explained variance, r2, etc...

These metrics however do not take "costs" into account, for example, for me it is worse to under-predict a value (Real: 0.5, Predicted: 0.4) than to over-predict it (Real: 0.5, Predicted: 0.6).

How can I model such costs into an evaluation function? I just need a first idea to start with and will welcome any suggestions.


texts: How to set kernel size (height and width) for 1D convolution layer in CNN Keras R API for doc2vec input?

texts:
I am using a doc2vec matrix (6000,1024) as feature space to input in a classification model which can categorise each document(sentence in my case) into one of the two classes (A,B). My feature space has 1024 features for each of the 6000 sentence in the dataset.

So basically each row of doc2Vec matrix represents a feature vector for that sentence. I am trying to use this single matrix as input to a 1D CNN. Usually, the input to CNNs for NLP tasks have one matrix per sentence, hence the batch size is equal to the number of sentence. Whereas my situation is slightly different, I have one matrix as described above.

So basically I want the filters to have kernels (of various widths) which slide across just one sentence vector, as opposed to usually kernels which slide across a group of words (i.e slide across multiple word vectors at a time with a height greater than 1). However, I am finding it difficult to set the parameter values to the following layers in keras API for R

layer_input(shape = 1024, batch_shape = c(6000,1024), dtype = 'float64', name = 'input1')
does this input layer represent what I want it to represent, if not what should it look like otherwise?

and not sure how to set the parameter values for
 layer_conv1d(filters =      , kernel_size =      ).

I am not sure how to give kernel_size list of values which represent (height = 1 and width = n) where n is an integer greater than 1.

So in summary I am expecting CNN to select a feature space (6000*n) from an original feature space(6000*1024) and I would column bind it with another feature space to feed it to fully connected layers in the CNN architecture.


texts: Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

texts:
I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, full-batch Gradient Descent.

Does it make sense to use Mini-Batch Gradient Descent for this kind fo task? I guess that would generate a noisy reduction of dimensionality?


texts: Multidimensional Output from Radar Imagery and Climate Data

texts:
I am trying to predict what my rainfall field will look like at a future timestep using:

Radar imagery of rainfall fields at previous timesteps: A set of 2D matrices where each element in each matrix corresponds to the rainfall intensity [mm/hr] and the dimensions of each matrix - the rows and columns, corresponds to the X,Y coordinate (i.e. the location). Therefore, each matrix denotes the rainfall intensities at every location at a specific timestep.

Climate data: A spatial averaged (i.e. averaged over every location for each timestep) climate dataset. An example is illustrated below (the 't's refer to the timestep).

So my goal is to forecast what the rainfall field will look like (i.e. the output would be a 2D matrix with rainfall intensities at each location) at a future timestep (e.g. t + 1 hour) using inputs from BOTH (1) and (2). I know I can use the radar image alone and pass it through a CNN and it should work fine but I want to incorporate both climate data and radar image data.
Would I use a multi data/multi-type input Neural Network and concatenate at the end? If so, then how would I do so and are there any resources I can be directed to? If not, then what should I do?


texts: Slow keras fit method with 100x100 array, how can I make it faster?

texts:
how can I make this training faster ?
when I call the fit method on a 100 x 100 matrix goes very slow
my model it's a sequential
h = self.model.fit(
        inputs,
        targets,
        epochs=epochs,
        batch_size=16,
        verbose=1,
)

this is my matrix
def build(n):
    mat=np.ones(N*N)
    return mat.reshape((N,N))

this is the of my Qtraining
qt = Qtraining(model,
               env,
               n_epoch=200,
               max_memory=500,
               data_size=100,
               name='model100')

This is the experience method
def get_data(self, data_size=10):
        env_size = self.memory[0][0].shape[1]  # env_state 1d size (1st element of episode)
        mem_size = len(self.memory)
        data_size = min(mem_size, data_size)
        inputs = np.zeros((data_size, env_size)) # metti Nsize righe di 0 , e envSize elementi 0
        targets = np.zeros((data_size, self.num_actions))#
        for i, j in enumerate(np.random.choice(range(mem_size), data_size, replace=False)):
            env_state, action, reward, next_env_state, game_over = self.memory[j]
            inputs[i] = env_state
            # There should be no target values for actions not taken.
            # Thou shalt not correct actions not taken #deep (quote by Eder Santana)
            targets[i] = self.predict(env_state)
            # Q_sa = derived policy = max quality env/action = max_a' Q(s', a')
            Q_sa = np.max(self.predict(next_env_state))
            if game_over:
                targets[i, action] = reward
            else:
                # reward + gamma * max_a' Q(s', a')
                targets[i, action] = reward + self.discount * Q_sa

        return inputs, targets


texts: Calculating mean of data frame inside a series object

texts:
I have a data frame with following structure:

df.columns

Index(['first_post_date', 'followers_count', 'friends_count', 'last_post_date','min_retweet', 'retweet_count', 'screen_name', 'tweet_count',  'tweet_with_max_retweet', 'tweets', 'uid'],
      dtype='object')

Inside the tweets series, each cell is another data frame containing all the tweets of an user.

df.tweets[0].columns

Index(['created_at', 'id', 'retweet_count', 'text'], dtype='object')

I want to perform calculation on the tweets of each users, for example, finding average number of retweets of each user, the tweet with maximum retweets etc.

How can I do these?

Edit
link to sample data


texts: Combining categorical and continuous input features of different shapes to predict continuous variable

texts:
I have two types of input, one is continuous data with shape like (6000 x 3 x 3 x 256). The other is categorical / binary, a shape of (6000 x 1).

The dependent variable is continuous.

This is how I'm currently training my model for the continuous data:

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

model = Sequential()
model.add(layers.Dense(10, activation='relu', kernel_regularizer=regularizers.l2(0.001), input_shape=(3,3,256)))

model.add(Flatten()) # Fix input shape error

model.add(BatchNormalization()) # Fix predictions coming out all the same

model.add(layers.Dropout(0.6))
model.add(layers.Dense(10, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(2, activation='sigmoid'))

# compile the model
model.compile(optimizer='rmsprop',loss='mse',metrics=['accuracy'])

# training the model
history = model.fit(X_train, Y_train, epochs=20, validation_data=(X_test, Y_test))

predictions = model.predict(X_test)

I'm wondering how I can combine this with my other binary feature - do I make a model to predict a continuous variable from categorical and just concatenate with the model I have above?

Edit

I've been trying something like this but getting an error:

# Define input
colour_histogram_input = Input(shape=(3,3,256), name='colour_histogram_input')
contains_face_input = Input(shape=(1,), name='contains_face_input')

# Layers
colour_histogram = Dense(64,  activation='relu', kernel_regularizer=regularizers.l2(0.001))(colour_histogram_input)
contains_face = Dense(64,  activation='relu', kernel_regularizer=regularizers.l2(0.001))(contains_face_input)

# Concat
concatenated = concatenate([colour_histogram, contains_face])

# Output
output = Dense(2, activation='sigmoid', name='output')(concatenated)

# Model
model = Model(inputs=[colour_histogram_input, contains_face_input], outputs=[output])

Error

A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 3, 3, 64), (None, 64)]


texts: None of the known overfitting prevention techniques works for me, according to learning curves

texts:
I am working on HTRU2 dataset to evaluate classification models. Even though I obtain good results in terms of accuracy-MSE:

I have an overfitting problem according to the learning curves below. In some of my models (Extra Tree, BaggingClassifier, AdaBoost, Random Forest, Decision Tree, k-NN), the red training curve never goes down as you can see:

The ways that I have tried to prevent overfitting:

I was using 30% of the whole dataset for the test set. I have decreased it to 20%, therefore I got a bigger training set but it has no effect on the curves (only slightly in the accuracies)
Incremented the cv count in the GridSearchCV from 10 up to 50: The difference is only in the accuracies.
I have tried to use PCA with k-NN to simplify: Again there was no difference on the curve.
Tried different pre-processing techniques: I was using StandardScaler and I have changed it to MinMaxScaler, however, I could not achieve different curves (now I have slightly better accuracies though)

I do not know how to solve this problem. In the beginning, I have oversampled the dataset with SMOTE method because of the imbalance. Might this cause the problem? Any help will be appreciated!


texts: Extract 2 pieces of information from a string - what to use?

texts:
First of all, I am a complete newbie in regard to data science and I am not asking for the complete solution but some guidance as to what I should read up to achieve my task (what algorithms, techniques etc are used to tackle similar problems).

I have different lists of strings which contain one or two useful pieces of information I would like to extract. In the following is an example I need to extract the bold and italic part from each line. This is just an example though, eventually I will need to end up with a process I can apply to different lists with different context. Here's a small sample from a list of 500:

50" Sony KDL 50W756CSAEP Smart LED Full HD
55" Samsung UE55JU6400 Smart LED HD
LG 55LF652V 55" SMART 3D FULL HD
HITACHI 55HGW69 55'' LED ULTRA SMART WIFI
TV 65" SAMSUNG UE65KS7500 4K LED Smart

In my full list I have already manually extracted the brand and model. So what I need now is a way to automate the process for a new list containing more brands and models. I thought I could go about this heuristically but since I am not just doing this for this type of data it won't work well.

So can someone give me some suggestions on a good way to go about it?

Thanks!


texts:
I have a dataset of 25 instances these instances are divided into 2 classes Green Circles and Blue Squares

data distributed as this graph

I want to predict X's class based on "Likelihood Weighted KNN with k =3"

In normal KNN this is easy

the nearest 3 points are 2 Blue Squares and 1 Green Circle

which means X will be Blue Square

there are more Blue Squares neighbours than Green Circles (2 vs 1)

But What is needed is to find the Likelihood Weighted KNN with k =3

This is my try

In this case we have to calculate the weight (Likelihood) for each instance

Each Green Circle likelihood is $\frac{1}{5} $ , we have 5 Green Circles

While for Blue Squares it is  $\frac{1}{20} $ , we have 20 Blue Squares

Therefore the weights around X will be $\frac{1}{5} $  Green Circle, and $\frac{2}{20} $ Blue Squares.

which means $\frac{1}{5} > \frac{2}{20} $

Then X is Green Circle

Well, this is wrong :(

Can someone help me find the


texts:
Is there any good library in stanfordNLP which is meant for name parsing? Basically i want to separate first name, last name, nickname, surname, givenname from the full name. Something similar to HumanName and parserator.tag, parserator.parse? Lets say i have a name Mr George "Gob" Bluth II and if i use parserator.parse, it gives me below output. Is there something like this in stanford NLP?

>>> probablepeople.parse('Mr George "Gob" Bluth II')
[('Mr', 'PrefixMarital'), ('George', 'GivenName'), ('"Gob"', 'Nickname'), ('Bluth', 'Surname'), ('II', 'SuffixGenerational')]

Thanks!


texts: How do I create a data set that has a set of features for multiple options, with one option being the expected outcome?

texts:
Most datasets I see are:

feature 1, feature 2, feature 3, outcome

Where outcome is binary e.g. if they are cancer positive outcome will be 1 and 0 if they don't have cancer.

How do I create a dataset where there are multiple outcomes and each possible outcome has a set of features for it?

e.g. I have a question with 3 possible answers:

"What organ pumps blood around the human body?"
A. Heart
B. Liver
C. Church Organ

And each answer has a set of features with one answer being correct. How would I display this in a csv file? I want to read it into an xgboost algorithm for training.

question, option1 and features, option2 and features, option3 and features, correct option

Many thanks for your help!


texts: Difference between coefficient of determination and least squared error?

texts:
Where and when should I consider R^2 as a measure of goodness-of-fit for regressions?
Usually I choose the least squared model as the best model.
Is it possible that the least squared model does not have the highest R squared?
Should I use both as the evaluation of regressions?
Could you give me an intuitive example?


texts:
I am currently working with Georgian texts processing. Does anybody know any stemmers/lemmatizers (or other NLP tools) for Georgian that I could use with Python.
Thanks in advance!


texts: Deriving Prediction Intervals for Orthogonal Distance Regression using `scipy.odr`

texts:
Questions

How can I derive prediction intervals for predictions based on new observations from the output of scipy.odr?
Is it also possible (or necessary) to take into account uncertainties in the new observations?

Background
I would like to perform a linear regression between two sets of variables, both of which have uncertainties associated with them, and also be able to derive prediction intervals as part of the output.
I have identified Orthogonal Distance Regression (ODR) as a possible method by which this could be done, and the scipy.odr library as an implementation of this method.
What I have been unable to find is how to use the output of this library to derive prediction intervals. I understand from this page that I need to add - in quadrature - the estimate of the standard deviation of the predicted value and the estimate of the residual standard deviation obtained when fitting the model to the data, but I am unsure how to derive these quantities from the output of scipy.odr. I am also almost completely unsure how to take into account any uncertainties in new observations, as surely these will affect the resulting prediction intervals.


texts: conceptnet api documentation for /wn/ part of the URI

texts:
for url http://api.conceptnet.io/c/en/music/n/wn/communication

/c/en/music/n is explained the doc (https://github.com/commonsense/conceptnet5/wiki/URI-hierarchy) as:

"the initial /c to make it a concept, a part that indicates its language (usingthe BCP 47 language code for that language), and a part with the concept text. An optional fourth component gives the part of speech (as a single letter, following the convention of WordNet)."

But couldn't find documentation for the part /wn/communication. Could you point to the documentation for this or explain this part of URI .. Thanks


texts: How can I know the name of the features selected by a Deep Belief Network?

texts: I want to use DBN to reduce the 41 features of nslkdd dataset after transforming nominal data to numeric the number of features increases from 41 to 121 . I used 3 RBMs (121-50-10) now I want to know the 10 selected features i.e know their names to put them as an input to the classifier. how can I do it?


texts: How to manage missing data in meteorological time series?

texts: How to know the type of missing data is what it is: MCAR, MAR or NMAR, knowing that I'm working on time series multivariate, and is that going to help me deal with the missing data, and what is the best techniques of processing missing data in time series, knowing that I work on meteorological data?


texts: How to choose inputs to maximize reward for an ordered dataset?

texts:
Consider the following problem, simplified:

Let's say that you have some data where you have only 5 columns/attributes.
One of them can be seen as a reward. How well we performed at a given run.
The other two attributes, let's say x and y are our input to the system. A human hand picked those x, y values and they were recorded.
Another two attributes are observations from sensors. These are not inputs and we have no control over them. Let's call them ob1 and ob2.
All are real values (nothing is discrete).

One idea is to look at this dataset as an ordered dataset.
We are asked to choose input values x and y that maximize our reward.

We have available ~70.000 instances of this five-tupled dataset

This is one approach that comes into mind but not sure if it is correct in principle or the simpler one.
We could build a predictor which takes as input the four attributes (x, y, ob1, ob2) and has as target the reward.
Then try a repetitive process for the inputs by using reinfocement learning (?) to get the inputs that maximize the reward?..


texts: How to predict the dealer whether pick up the goods next month?

texts:
Here, I want to predict dealers(about 600) whether pick up the goods(about 30) next month. As you see, there are about 18,000 possibilities and it's difficult to predict.

By the way, now I have nearly three years'data(2017,2018 and 2019), and now I just use the common object in the last several months as the prediction.

For example, if dealer A purchases the goods item 1 in September, October and November, then I predict he will purchases item 1 in December. However, this is not exactly accurate, and the precision is about 0.6 while the recall is about 0.3.

So any other way to predict it?


texts:
I can't figure out how to install the python package named skmca ( MCA for sickit-learn ) : https://github.com/TomAugspurger/skmca

There is no pip or conda command available.

Thanks for your help.


texts: Why K-harmonic means is less sensitive to random initialization than K-means?

texts: As we know that K-Means is a powerful clustering, however it often stuck with local minimum problem because of bad initialization . One of the solution is K-Harmonic means that use Harmonic Average as the performance function instead of minimum distance btw the data and cluster's centroid. I am curious that why using harmonic function makes less sensitive to init centroids than ordinary K-Means.


texts: Including the validation file in the training process after tunning

texts: Should I include the validation file in the training process after finishing the tuning process (e.g. searching for params using the validation file)?


texts: Multivariate linear regression accounting for threshold / data cleaning

texts:
I am trying to make a linear regression model for the sale price of a house based on many variables (based on the data from this Kaggle challenge https://www.kaggle.com/c/house-prices-advanced-regression-techniques)

The distribution above is the 2nd-floor size in square feet and the y-axis is the sale price. It shows a clear linearity, except for the fact that homes without a 2nd floor clearly are not for sale for 0 dollars.

I have many variables like this that have some threshold either upper or lower that has a large distribution of response for a single value. If I simply exclude these values then the intercept for this curve will be through the origin. Should I let that be the case and assume that the $0 price tag will be corrected by the other variables in my regression?

What is the best way to treat/fit data such as this?

Thanks!


texts: Is there any market for data science consulting

texts: Is there any market for data science as service which provides service without selling software, just consulting, modelling and developing algorithms?


texts: Limitations while using orange for clustering

texts:
I have tried clustering using kmeans in Orange but it looks like there are certain limitations as listed below,
- Supports up to 5000 records only
- no. of clusters can be only 30

Can someone please confirm on this limitation? And is there a way to overcome the same.


texts: Visualizing Support Vector Machines (SVM) with Multiple Explanatory Variables

texts:
I was wondering if anyone was aware of any methods for visualizing an SVM model where there are more than three continuous explanatory variables. In my particular situation, my response variable is binomial, with 6 continuous explanatory variables (predictors), one categorical explanatory variable (predictor). I have already reduced the number of predictors and I am primarily using R for my analysis.

(I am unaware if such a task is possible/ worth pursuing.)

Thanks for your time.


texts: It seem that the textbook "Machine Learning - A Probabilistic Perspective" uses input and output in a opposite way, is it?

texts:
Chapter 1 of "Machine Learning - A Probabilistic Perspective" by Kevin Patrick Murphy says

  We now consider unsupervised learning, where we are just given output data, without any inputs. The goal is to discover “interesting structure” in the data; this is sometimes called knowledge discovery.

post this post

  Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses.

I saw this kind of explanation more times than the one in that book.

It seems that the book uses the terms in a opposite way, where the dataset is viewed as the output of some input, is my understanding right?


texts:
Consider the following Decision table :

The following is the reduction process of this table :

The above table is the reduced table. But my question why we can't reduce further rule number 3 and 4  Since they differs only in A ?
And Also rule number 1&5&7 should has  don't care at the condition B ?


texts: Is it possible to know the output vectors of MLP Classifier of scikit learn?

texts:
I'm a beginner with scikiti-learn library.
I have an ANN with 3 input, 2 hidden layers and 3 output.

mlp = MLPClassifier(hidden_layer_sizes= hidden_layers,max_iter=iterations, activation=activation_fun)

I read on the documentation that the classifier uses softmax for the output activation function and cross-entropy loss function.
I have a multi-class problem where the three outputs will predict the classes 0,1,2.
My question is that. How can I retrieve the vectors that enconds the classes 0,1,2?
example:
[1,0,0] -> 0
[0,1,0] -> 1
[0,0,1] -> 2


texts:
Let's assume I have the following dataframe in PySpark:

Customer    |  product  |   rating
customer1   |  product1 |   0.2343
customer1   |  product2 |   0.4440
customer2   |  product3 |   0.3123
customer3   |  product1 |   0.7430

There can be several customer product combinations but every combination is unique already. I want to archive the following outcome in the most efficient manner:

Customer (Index) | product 1 | product 2 | product 3
customer 1       |   0.2343  |  0.4440   |  0.0000
customer 2       |   0.0000  |  0.0000   |  0.3123
customer 3       |   0.7430  |  0.0000   |  0.0000

Each combination which is not represented in the first table will be set to zero. It has to be efficient because the output matrix will have a size of 59578 rows × 21521 columns and I want to avoid the computational cost as good as possible.

Is there any solutions for this? I didn't found a good solution on the web so far.

Thanks for your help up front.


texts:
I'm trying to study the effect of curse of dimensionality in classification algorithms.

I have a simple dataset with only 1 feature. So i can represent the set in a 1 dimension.
If now i want to introduce some noise in this dataset, is it correct to add another feature with random values to my dataset ?


texts:
Im studying machine learning, and I would like to know how to calculate VC-dimension.

For example:

$h(x)=\begin{cases} 1 &\mbox{if } a\leq x \leq b \\
 0 & \mbox{else } \end{cases} $, with parameters $(a,b) ∈ R^2$.

What is the VC-dimension of it?


texts: How to work with input which is a combination of metadata+ vectorized text data + image pixel data to build a Regression Model (predict views)?

texts:
There are 4 datasets (all in csv format), each has a uniqueID column by which each record can be identified. Image and text datasets are dense datasets.(need to be converted to ndarray).
Can someone suggest how to use all these 4 datasets for building a regression model?
This is how the datasets look,
Metadata having some input features and target variable(views)
uniqueID    ad_blocked embed  duration language hour views
     1        True     True    68        3      10   244
     2        False    True    90        1      15   63
     3        True     False   195       3      7    350

Vectorized title data - one entire row represents a title
uniqueID  title_1   title_2    title_3
     1   -0.977637  -0.543310  0.079403
     2    0.041873   0.644655  -0.406487
     3    0.503560  -0.085412  0.841144

Vectorized descriptions data - one entire row represents a description
 uniqueID  title_1   title_2    title_3
     1   -0.052256  -0.016036  0.079403
     2    0.000106  0.356706  -0.025788
     3    0.015774 -0.085412   0.712229

Thumbnail pixel data - one entire row represents an image
uniqueID  image_1    image_2   image_3
     1   -0.484456  -0.543310  0.032915
     2    0.666147  0.644655  -0.005733
     3    0.035018  -0.011111  0.841144


texts: How to use auxiliary target variables only present in train data

texts:
Imagining in my train data I have 3 target variables y1, y2 and y3, all binary.
My main goal though, is to predict the final variable Y = y1 * y2 * y3.

What should be the approach of a model when dealing with this kind of target variable, that I can decompose into different ones?

Should I try to predict y1, y2, y3, then calculate Y and pass it to a loss function
Should I try to predict Y standalone and ignore y1, y2 and y3?
Should I try to predict the 4 of them and penalize my model when Y != y1 * y2 * y3 ?


texts:
I am trying to update my spark configuration to solve some dependency problems.

This pom.xml file seems to be useful for this purpose.

I am using a spark docker image.

ls /spark/conf

Gives:

docker.properties.template    slaves.template
fairscheduler.xml.template    spark-defaults.conf.template
log4j.properties.template     spark-env.sh.template
metrics.properties.template

I've searched pom.xml in the container by using find / -name "pom.xml" and got nothing.

Is there a way to use a pom.xml file to update spark configuration?


texts: avoiding premature convergence with neural networks (EA's)

texts:
I am currently writing a program that would be able to play snake on an 25*25 grid. It works by optimizing a set of weights of 300 different solutions (each solution would be a different neural network) with the aid of an evolutionary strategy, thus by random mutation and parent selection. I have decided not to apply crossover on the parents pool due to the black box problem concerning MLPs and other neural networks. My population size is 300 and 10 parents are selected every generations (200 gen. total). Every other solution which is not a parent gets deleted to make place for 290 new mutated solutions based on the parents.
My network structure (MLP) is the following : 6 input nodes, (10,10) hidden nodes, 3 output nodes.
With my current version of the programme, I am able to reach a steady score of 60 apples (or points) eaten (around 15/200 have a score above 60). And this only after 50 generations! But now, it seems like i have bumped into a local minima that I am not able to overcome. Does anyone have any idea if further progress is still possible or that i might have hit the ceiling? The quickly elevating performance might be a sign that my exploration/eploitation balance might not be ideal, but with each solution containing 190 weights, more exploration would seem too computationally expensive and would take forever on my i7 10gen laptop :/.
Can i call this a premature convergence and is there a way to deal with this, or have i hit the limits of my idea?


texts: Can I use model which is trained using Keras to Scala?

texts:
I train the NLP classification model using Keras by Python.
But to deploy the trained model to a platform I need to use Scala.

Can I use Scala with Keras model what I trained using Python?

Thanks


texts: I got the following error : 'DataFrame' object has no attribute 'data'

texts:
I am trying to get the 'data' and the 'target' of the iris setosa database, but I can't. For example, when I load the iris setosa directly from sklearn datasets I get a good result:

Program:

from sklearn import datasets
import numpy as np
iris = datasets.load_iris()
X = iris.data[:, [2, 3]]
y = iris.target
print('Class labels:', np.unique(y))

output:

Class labels: [0 1 2]

But if I try to load it directly from extension '.csv' I get the following error:

Program:

import pandas as pd

iris = pd.read_csv('iris.csv', header=None).iloc[:,2:4]

x = iris.data
y = iris.target

output:

'DataFrame' object has no attribute 'data'

Why does this happen?


texts:
I have a dataset like:

dframe <- structure(list(ind1 = c(1L, 2L, 1L, 1L, 3L, 1L, 1L, 2L), ind2 = c(0L,
0L, 4L, 3L, 0L, 1L, 0L, 2L), ind3 = c(1L, 1L, 1L, 1L, 0L, 0L,
1L, 1L), ind4 = c(1L, 0L, 0L, 0L, 0L, 2L, 0L, 0L), ind5 = c(0L,
1L, 0L, 0L, 3L, 0L, 0L, 0L), ind6 = c(0L, 0L, 0L, 0L, 4L, 0L,
0L, 0L), ind7 = c(1L, 1L, 1L, 2L, 1L, 1L, 1L, 1L), d1 = c(3L,
3L, 0L, 5L, 3L, 3L, 5L, 5L), d2 = c(3L, 2L, 0L, 5L, 5L, 3L, 5L,
4L), d3 = c(3L, 2L, 0L, 1L, 5L, 3L, 5L, 4L), sex = c(1L, 1L,
2L, 1L, 1L, 2L, 1L, 1L)), row.names = c(NA, -8L), class = c("data.table",
"data.frame"), .internal.selfref = <pointer: 0x0000000002661ef0>)

     ind1 ind2 ind3 ind4 ind5 ind6 ind7 d1 d2 d3 sex
1:    1    0    1    1    0    0    1  3  3  3   1
2:    2    0    1    0    1    0    1  3  2  2   1
3:    1    4    1    0    0    0    1  0  0  0   2
4:    1    3    1    0    0    0    2  5  5  1   1
5:    3    0    0    0    3    4    1  3  5  5   1
6:    1    1    0    2    0    0    1  3  3  3   2
7:    1    0    1    0    0    0    1  5  5  5   1
8:    2    2    1    0    0    0    1  5  4  4   1

I would like to use the gml model to find correlations between dependent variables d1,d2 and d3 and independent variables ind1, ind2, ind3, ind4, ind5, ind6, ind7 and sex

m1 <- glm(d1 ~ etiology + local1 + local2 + local3 + local4 + local5 + handedness, data = data,poisson)

Is poisson distribution the right option?


texts: What are some resources to test your data science skills?

texts:
I can't get a job to save my life so I am guessing my lack of skills is an issue. I've been doing a lot of reading on statistics and I am getting antsy - I want to move from theory to application and test my skills to see how I measure up with real practitioners.

Most books I have don't have solutions. They have questions and labs but without some semblance of an answer, I don't know if I am doing it right. I know when it comes to fitting models, there isn't a "right" answer, but there are definitely bad answers. I just want to make sure my answers are good enough.

What are some resources where I can practice my data science, statistical inference, machine learning skills - with answers? I know of https://www.kaggle.com/competitions has tasks but those seem beyond my ability right now - and obviously won't come with right answers.

Are there websites that test your ability to build a predictive model?

Any suggestions?


texts: custom tensorflow layer that inherits another class

texts:
I am trying to make my own custom layer in tensorflow which looks like:

class custom_LSTM(RNN):
    # implementation of class

where RNN is another class.

However, to use this custom layer in model it needs to inherit tf.keras.layers.Layer. I am creating models using tf.keras.

Any idea how do I solve this problem?


texts: Gradient calculation for proportional odds (logistic) model

texts:
I am trying to calculate a gradient for a proportional odds model. http://fa.bianp.net/blog/2013/logistic-ordinal-regression/

What steps are required to take the derivative with respect to w?

$$\mathcal{L}(w, \theta) = - \sum_{i=1}^n \log(\phi(\theta_{y_i} - w^T X_i) -  \phi(\theta_{y_i -1} - w^T X_i))$$

Give that the derivative of a sigmoid function is the following:
$$\log(\phi(t))^\prime = (1 - \phi(t))$$

I should get this, right?

$$
\nabla_w \mathcal{L}(w, \theta) = \sum_{i=1}^n X_i (1 - \phi(\theta_{y_i} - w^T X_i) - \phi(\theta_{y_i-1} - w^T X_i))$$


texts: How to perform feature selection on dataset with categorical and numerical features?

texts:
I am working on a dataset with 30 columns (29 numerical, 1 non-ordinal categorical). I hot-encoded the categorical feature and reached at 35 columns. To improve training efficiency, I want to perform feature selection on my dataset. However , I am confused with how to handle a dataset with categorical and numerical features combined.

I read that it is not reasonable to apply PCA on dummies given they are discrete. Is it reasonable to apply PCA first on numerical features then concatenate them with dummies?
I tried to implement recursive feature elemination with cross-validation (RFECV) to the entire feature space. But I don't think it is reasonable to remove some but not all dummy features given they are generated out of one category.

Any suggestions? Any help is appreciated.
python pandas scikit-learn feature-selection


texts: How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps?

texts:
I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels
$ x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$
But for the second layer, where we already have $N$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $N$ weighted sums we have from feature maps?
If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.

In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).
How is the value (convolution)  for this particular pixel/cell calculated? showed we take average like below?
(I can add more details if the question is not clear)
weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4


texts: When to use Dense, Conv1/2D, Dropout, Flatten, and all the other layers?

texts:
I have a binary classification problem and want to build a NN model which classifies the data whether class 0 or class 1.

My actual implementation looks like the following:

# Split dataset in train and test data
X_train, X_test, Y_train, Y_test = train_test_split(normalized_X, Y, test_size=0.3, random_state=seed)

# Build the model
model = Sequential()
model.add(Dense(23, input_dim=45, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=200, batch_size=5, verbose=1, callbacks=[tensorboard, time_callback])

and I get a val_accuracy of 79.85% at the last training epoch.

I used a confusion matrix to evaluate the model:

y_pred = model.predict(X_test)
y_pred =(y_pred>0.5)
list(y_pred)

cm = confusion_matrix(Y_test, y_pred)
print(cm)

and I get these values:
[[ 622  205]
 [ 216 1055]]

which makes 79.93% (approximately the same as the val_accuracy on the last epoch) of right predicted classes (622 TN + 1055 TP).

Now my question is: How to improve my NN so that I get above that? Okay, I am using 1 hidden Dense Layer, with 23 nodes. When to use Dense layers, and when to use Conv2D or Dropout, or any of the other layers of Keras?

I am classifying numerical data. Here is how my data looks like (the dataframe separated in 2 photos, because it's too wide for just 1):

PS: the categorical features were One-Hot-Encoded using:

basic_df = pd.get_dummies(basic_df, columns=['industry', 'weekday', 'category_name', 'page_name', 'type'])

And the label column is 'successful'.


texts:
I'm trying to implement an active mask code derived from Active mask segmentation for the cell-volume computation and golgi-body segmentation of hela cell images, Srinivasa et al 2008.

I have found a code that will allow me to do this, however, there is one section of the code that is giving me errors and I don't know why. Here is the section of the code:

def _converge(P, R, b, max_iters_converge=1000):
    Pm = numpy.empty(P.shape, numpy.float32)  # Pre-allocate: saves time
    argmax = numpy.zeros_like(P)
    maxval = numpy.zeros(P.shape, numpy.float32)
    for i in range(max_iters_converge):
        maxval *= 0
        maxval -= 1e8
        for m in range(int(P.max()) + 1):
            Pm[:, :] = (P == m)
            Pm = ndimage.gaussian_filter(Pm, b) + R[m]
 # Pm=ndimage.convolve(Pm,numpy.ones((2*b+1,2*b+1)))+R[m]
            argmax[maxval < Pm] = m
            maxval = maxval * (argmax != m) + (argmax == m) * Pm
        if (P == argmax).all(): break
        P = argmax.copy()
    mis = defaultdict(range(1, P.max() + 1).__iter__().__next__())
    mis[0] = 0  # set 0 to 0, because it is special
    for i in range(P.size):
        P.flat[i] = mis[P.flat[i]]
    return P

The area that is giving me the most problem is the:

mis = defaultdict(range(1, P.max() + 1).__iter__().__next__())

Originally the code was written for Python 2.4 but I made some changes so that it will work on Python 3.6, but I don't really understand what the defaultdict is used for.

Here is a link to the full original source file: active mask


texts: Getting different precisions for same neural network with same dataset and hyperparameters in sklearn mlp classifier

texts:
I get WAY DIFFERENT results in each run despite using random state for making sure that network outputs same result for same hyper parameters, here is some sample outputs(I've printed the hyper parameters manually to show that they are the same) and I think I may missing something...

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.8888888888888888

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.3333333333333333

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.4444444444444444

{'hidden_layer_sizes': [20, 20, 30], 'activation': 'tanh', 'solver': 'adam', 'alpha': 0.02, 'learning_rate': 'adaptive', 'iteration': 400}
0.7777777777777778

here is some of the code:

mlp = MLPClassifier(random_state=2,
                    hidden_layer_sizes=(20, 20, 30),
                    max_iter=400, alpha=0.02,
                    activation='tanh',
                    solver='adam',
                    learning_rate='adaptive')
mlp.fit(X_train, y_train.values.ravel())


texts: Training a convolutional neural network for image denoising in Matlab

texts:
I am currently trying to train CNNs to remove Poisson noise from images. The software I am using is Matlab 2018b, however the results I am getting are poor.
I have followed the steps provided in the following link. Here it is stated that we need to make a denoisingImageDatastore, which holds patches of our training images, and applies Gaussian noise to them. This isn't what I am looking for, but for the sake of practice I decided to try it out on my of PASCAL VOC dataset. I will now roughly outline the steps in my matlab code for this.

I first created an imageDatastore holding 45 of our images used for training.
Fifteen of those images will be used for validation
Now we create denoisingImagedatastores for both the training and validation set. There will be 60 patches per image, with the patch size 50x50
Then we specify the training options
I then specified the network layers using dnCNNLayers function from matlab
The next step is to train the network using the trainNetwork function

Here is a screen shot from the end of the training process.

I have done only 100 iterations, but the results are not too bad.
Here we can se the pristine image, the noisy image and the denoised image using this network.

On the following link is the code I used for this. If you have any suggestions how the RMSE could be further lowered, it would be appreciated.
Please note that I have previously trained the same network on a much larger dataset and with nearly 10000 iterations, and the best rmse it could achieve was about 2.3. So obviously some parameters need to be changed.


texts: R: for each row in DF1 subset/count corresponding rows in DF2


texts: While downsampling training data should we also downsample the validation data or retain validation split as it is?

texts:
I am dealing with class imbalance problem.
In this case, I am down sampling the majority class lables in the training set.

Among training, validation and test splits, the majority class in training split is down-sampled, and test split is retained as it is. However, should the validation split be downsampled according to the training-set or should it be retained as it is?

This is because the validation set controls the training process.


texts: How to approach All vs All classification problem

texts:
Let's say you are building a Star trek style medical tricorder which can diagnose any medical condition. It needs to be able to detect comorbidities where a patient has multiple conditions (e.g. perhaps the patient has COVID, diabetes and lung cancer all at the same time).
How would you build a classification system to detect the most likely set of conditions?
I see two approaches:

Build one model per disease, run predictions using every model and report that the patient is suffering from any disease for which the prediction probability exceeded a threshold (e.g. 0.95)

likely_diseases = []
THRESHOLD = 0.95
for disease in disease_columns:
  model = xgb.XGBClassifier()
  model.fit(X_train, y_train[disease])
  pred_proba = model.predict_proba(patient_data)[:, 1]
  if (pred_proba > THRESHOLD):
    likely_diseases.append(disease)

Build one model per combination of diseases and choose the combo with the highest probability

df['has_covid_lung_cancer_and_stroke'] = df.apply(lambda patient: patient['has_lung_cancer'] and patient['has_covid'] and patient['has_stroke'])

# create all other possible permutations of diseases

highest_probability = 0.0
most_likely_disease_combination = None
for disease_combination in disease_combinations:
  model = xgb.XGBClassifier()
  model.fit(X_train, y_train[disease])
  pred_proba = model.predict_proba(patient_data)[:, 1]
  if (pred_proba > highest_probability):
    most_likely_disease_combination = disease_combination
    highest_probability = pred_proba

It strikes me that approach two would probably be more accurate but might be so computationally expensive that it is intractable. Perhaps some pruning would take place where combinations that have exceedingly low occurrences in the training data set are discarded.


texts: Cluster doc2vec using Affinity Propagation

texts:
I want to cluster my document vectors (doc2vec) using affinity propagation.

However, I am just confused if I should use cosine similarity or cosine distance to cluster my document vectors. Currently, I am using cosine similarity for my affinity propagation clustering. Thus, my first question is;

Is it correct to use cosine similarity to cluster my doc2vec document vectors?

Moreover, I would like to visualize my cluster results using t-sne. However, I saw that t-sne requires distance matrix as the input. Hence, my second question is;

Is it correct to use distance matrix (cosine distance) for t-sne, while I use cosine similarity for clustering?

If my code is required I can post it too.

Please help me.


texts:
I'm looking for a way to avoid removing ending s when s isn't a suffix. In order to do that, I first check if a word exists in my index, if it does, I don't remove the ending s but If it doesn't, I go on and remove the ending s and add it to the index. But the problem is what to do when starting to build the index.
Imagine we encounter books, I remove s and add book to my index. On the other hand, I may encounter dangerous for the first time, since it doesn't exists in my index yet, I remove s and add dangerou which is obviously wrong. What should I do?
Specifically I'm looking for ways to properly detect if suffixes and prefixes are indeed one or part of the original word. one way that comes to my mind is using a formal dictionary and instead of my own index, check the words in that dictionary.
P.S: I'm not working on English docs. It's a college/prototype thing Therefore I'm looking for general, good ideas with good accuracy. I'm not looking for advanced stuff with superb accuracy and considerable complexity.


texts:
I had no idea how to explain this in the title but anyway... Let's say I have a data points like this:

John, Happy | Greedy | Smart | Funny, 0.8
Ann, Smart | Sad | Funny, 0.6
Joel, Greedy | Prideful | Stupid, 0.2

Where the first part is the name the second is there characteristics and the third is their overall character score (how nice they are to be around or something). Is there a good way to work with this data so that I can work out what the best possible combinations are? Assume I have a large enough data set. There may be say 30 of those characteristics and any person can have any number of them and every characteristic is equally valid to every person.

Hopefully that explains it. Essentially I am want a way to organise the traits so that I can say "smart and happy" make a better combination than "sad and greedy". I also need to be able to asses the ultimate combination and be able to compare any two possible combinations.


texts: how to handle different categorical values in train and test dataset?

texts:
I have a dataset in which if i do
train_df["era"].value_counts()

then it will show 120 different type of categorical values
and then if i do
test_df["era"].value_counts()

It do have 310 totally different categorical values like
train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

So if i will convert it to onehot encoding then it will give me 430 new features and datasize will explode. So my question is

if
train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

Then can we remove that feature ? Because how will model relate i
can think for continuous variable but this is categorical and if it
do affect performance then how ?

how to handle this number of different values in categorical
data ?


texts: XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC)

texts:
My dataset is of shape – 5621*8 (binary classification)

Label/target : Success (4324, 77 %) & Not success (1297, 23 %)
(success and Not success were been equally important for my prediction i.e, TP & TN)

I split my data into 3 (Train, Validate, test)

For train & Validate i perform 10 fold CV.

Test is the seperate data, which I evaluate for each folds

I tune my scale_pos_weight ranging between 5 to 80, and

Finally I fixed my values as 75 since I got average higher accuracy rate for my Test set (79 %) for those 10 folds
But, If i check my average auc_roc metrics it is very poor, i.e only 50 % for all 10 folds.

If i did not tune scale_pos_weight my avg.accuracy drops to 50% & my avg auc_roc increases to 70 %.
How can I interpret from the above results between AUCROC & Accuracy in this situation?
What might be the problem in my case?


texts:
When using the EM algorithm in Gaussian Mixture Models (GMM), in the E-step, we take each x set in the training dataset to calculate and update the "weight" and parameters of each Gaussian distribution of the clusters (M-step). I have read that we do this until it converges. I am a little confused here. Does that mean it loops through the whole training dataset X every time in "one step" of the EM algorithm? Or is "one step" corresponding to calculating ONE particular x set in the whole dataset and using it to update the parameters and weights of each distribution?

To make the question more clear, does each "step" in the EM algorithm in GMMs involve the whole training dataset X or does each step mean one particular x set in the training set, and we loop over the dataset one at a time to improve one at a time?

Any help will be appreciated! Thanks.


texts: Size of data required to build instruction Embeddings

texts: I am using different sizes of ARMv7 data to build instructions embedding using a conventional word2vec model which would output a numpy array of Embeddings that would be used in the second stage of model training. I was initially thinking the quality of the Embeddings would increase when the size of the training data was increased. However, after carrying out the experiment with different amounts of data such as 2GB, 1GB and 500MB I realized the AUC value of the final trained model that use the Embeddings generated from 500MB is higher than the embeddings generated from higher amounts of data such as 2GB and 1GB. Could there be a reason for such an observation? Couldn't we expect the embeddings quality is proportional to amount of data that we have?


texts: How to use spectral clustering to predict?

texts:
In an academic paper, they talk about using a nearest neighbour algorithm to predict the cluster of a new point. And how the number of nearest neighbours is set to 10 in their example.
What do they mean with this? The two things I could think of were:
1. Look which 10 points used in the training set (neighbours) are closest and then assign it to the cluster of which the majority of the points come from.
2. Collect one by one the closest points from the training set until you have 10 points that come from one single cluster. That is the cluster to which the point belongs.

What are the other ways to assign a(n) (existing) cluster to a new point?


texts: Question on Scipy - Minimize. Adding additional constraints

texts:
I am trying to using scipy minimize function for the following optimization:

V = np.matrix(pd.read_csv('V.csv'))`

R = np.matrix(pd.read_csv('R.csv', index_col = 'Ticker'))`

w0= list()
for i in range(0, 84):
    w0.append(1/84)

def calculate_portfolio_var(w,V):
    w = np.matrix(w)
    return (w*V*w.T)[0,0]

cons = ({'type': 'eq', 'fun': lambda x:  np.sum(x)-1.0})
myBound = [(0, 1) for i in range(0, 84)]
res= minimize(calculate_portfolio_var, w0, args=V, method='SLSQP',constraints=cons, bounds = myBound)

where V is the variance-covariance matrix, R is the series of annualized return of stocks.

In addition to the 2 constraints (cons and myBound), I want an additional constraint that the result portfolio return, which is the weighted average of the result weights and stock returns, be equal to certain number and the number of stocks to be less than equal to certain number..

For example, it should look like:

cons = ({'type': 'eq', 'fun': lambda x:  np.sum(x)-1.0},
        {'type': 'eq', PortfolioReturn = 10%,
        {'type': 'ineq', number of result stocks <= 40)

I am not so familiar with the Scipy minimize, and I would appreciate if someone can help me.


texts: How I can compute Matthews Correlation Coefficient?

texts:
I want to calculate MCC, I have two CSV files: result.csv and true.csv

result.csv format:

                   id | prob_class1 | prob_class2 | prob_class3 | prob_class4
                    1 |0.34         | 0.4         | 0.32        |0.77
                    2 |0.84         | 0.4         | 0.2         |0.7

true.csv  format:

                     id | prob_class1 | prob_class2 | prob_class3 | prob_class4
                     1 |0            | 0           | 1           |0
                     2 |0            | 1           | 0           |0
                     3 |0            | 0           | 0           |1

result.csv is the output of the multi-classification problem, true.csv is the true result. How I can calculate the Matthews Correlation Coefficient?


texts: What are the possible approaches to fixing Overfitting on a CNN?

texts:
Currently I am trying to make a cnn that would allow for age detection on facial images. My dataset has the following shape where the images are grayscale.

(50000, 120, 120) - training
(2983, 120, 120) - testing

And my model currently looks like the following - I've been testing/trying different methods.

    model = Sequential()
    model.add(Conv2D(64, kernel_size=3, use_bias=False,
                     input_shape=(size, size, 1)))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(Conv2D(32, kernel_size=3, use_bias=False))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())

    model.add(Dense(128, use_bias=False))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(Dropout(0.5))
    model.add(Dense(10, activation='softmax'))

    #TODO: Add in a lower learning rate - 0.001
    adam = optimizers.adam(lr=0.01)
    model.compile(optimizer=adam, loss='categorical_crossentropy',
                  metrics=['accuracy'])
    model.fit(x_train, y_train, validation_data=(x_test, y_test),
              epochs=number_of_epochs, verbose=1)

After running my data on just 10 epochs I started to initially see decent values but at the end of the run my results were the following and it has me concerned that my model is definitely over fitting.

How many epochs: 10
Train on 50000 samples, validate on 2939 samples
Epoch 1/10
50000/50000 [==============================] - 144s 3ms/step - loss: 1.7640 - acc: 0.3625 - val_loss: 1.6128 - val_acc: 0.4100
Epoch 2/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.5815 - acc: 0.4059 - val_loss: 1.5682 - val_acc: 0.4059
Epoch 3/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.5026 - acc: 0.4264 - val_loss: 1.6673 - val_acc: 0.4158
Epoch 4/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.3996 - acc: 0.4641 - val_loss: 1.5618 - val_acc: 0.4209
Epoch 5/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.2478 - acc: 0.5226 - val_loss: 1.6530 - val_acc: 0.4066
Epoch 6/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.0619 - acc: 0.5954 - val_loss: 1.6661 - val_acc: 0.4086
Epoch 7/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.8695 - acc: 0.6750 - val_loss: 1.7392 - val_acc: 0.3770
Epoch 8/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.7054 - acc: 0.7368 - val_loss: 1.8634 - val_acc: 0.3743
Epoch 9/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.5876 - acc: 0.7848 - val_loss: 1.8785 - val_acc: 0.3767
Epoch 10/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.5012 - acc: 0.8194 - val_loss: 2.2673 - val_acc: 0.3981
Model Saved

I assume the issue might be related to the number of images I have for each output class, but other then that I am a bit stuck in moving forward. Is there something wrong in my understanding/implementation? Any advice or critique would be well appreciated this is more of a learning project for me.


texts: How to achieve causal deconvolution with Keras

texts:
Keras allows one to specify padding='causal' to achieve autoregressive connections in Conv1D layers. However, in deconvolution with Conv1DTranspose, padding is limited to same and valid.
How can one achieve causal deconvolution then?
Is there something I am missing?


texts: Transfer learning by concatenating the last classification layer

texts:
Before going into an obvious XY problem, I will explain you what I'm trying to do.

I'm training a simple MobileNet pre-trained with Imagenet for multiclass classification. What I do is freeze all the convolutional part, and then create a new Prediction Layer (conv2D 1x1xN where N is the number of classes).

Let's say I trained with 5 classes, and then I got a new set of images of 5 new classes. I want to keep the knowledge from the last training and be able to train only with the new classes.

What I do is to train a new model with the same frozen pre-trained weights, but only with the new 5 classes, then concatenate the weights from the old and the new model into a last layer that outputs for 10 classes.

The concatenation works perfectly, but when I run the evaluation I got a horrible accuracy (like if it was predicted randomly). For example, for 30 clasess trained in "sets" of 5, I got an accuracy of 0.03.

Is what I'm doing viable? A problem that I think it can be causing this is that the "labels" for every prediction are not kept in the same order or something, so that even if I copy the weights, the predictions may be unordered.


texts: Is there a loss function that measures the cross similarity between two 2D tensors?

texts:
Given two input tensors x1 and x2 with the shape [batch_size, hidden_size], let S be the matrix of similarity between all pairs (predict, target), where predict and target are dense vectors with the shape [hidden_size] and predict belongs to x1 and target belongs to x2.

Is there any loss function that is minimized as the values in the diagonal of S are close to 1 while the other values are close to -1?

That is, the similarity between the vectors from x1 and x2 of the same index must be greater than between vectors of different indexes.


texts: How modelling is affected by similar feature distributions across classes?

texts:
I have grouped my dataset according to labels (good and bad customers), in an attempt to test how each feature is distributed within.
Along this, I found some feature has almost exact distribution in both groups and
I'm wondering what insight can I learn from this? Can it mean that the feature isn't that significant to data prediction and can be dropped?


texts: My CNN produces volatile validation_loss and does not converge in python(Keras)

texts:
My model is experiencing wild and big fluctuations in the validation loss and does not converge.

I am doing an image recognition project with my three dogs i.e. classifying the dog in the image. Two dogs are very similar and the 3rd is very different. I took 10 minute videos of each dog, separately. Frames were extracted as images at each second.

This block of code is responsible for augmenting and creating the data to feed the model.

randomize = np.arange(len(imArr)) # imArr is the numpy array of all the images
np.random.shuffle(randomize) # Shuffle the images and labels
imArr = imArr[randomize]
imLab= imLab[randomize] # imLab is the array of labels of the images
lab = to_categorical(imLab, 3)

gen = ImageDataGenerator(zoom_range = 0.2,horizontal_flip = True , vertical_flip = True,validation_split = 0.25)
train_gen = gen.flow(imArr,lab,batch_size = 64, subset = 'training')
test_gen = gen.flow(imArr,lab,batch_size =64,subset = 'validation')

This picture is the result of the model below.

model = Sequential()

model.add(Conv2D(16, (11, 11),strides = 1, input_shape=(imgSize,imgSize,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3),strides = 2))
model.add(BatchNormalization(axis=-1))

model.add(Conv2D(32, (5, 5),strides = 1))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3),strides = 2))
model.add(BatchNormalization(axis=-1))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3),strides = 2))
model.add(BatchNormalization(axis=-1))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(BatchNormalization(axis=-1))
model.add(Dropout(0.3))

#Fully connected layer
model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(3))

model.add(Activation('softmax'))

sgd = SGD(lr=0.004)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

batch_size = 64
epochs = 100
model.fit_generator(train_gen, steps_per_epoch=(len(train_gen)), epochs=epochs, validation_data=test_gen, validation_steps=len(test_gen),shuffle = True)

Things I have tried.

High/low Learning rate  ( 0.01 -> 0.0001)
Increase Dropout to 0.5 in both Dense layers
Increase/Decrease size of both Dense Layers ( 128 min -> 4048 max)
Increased number of CNN layers
Introduced Momentum
Increased/Decreased Batch Size

Things I have not tried

I have not used any other loss or metric
I have not used any other optimiser.
Have not adjusted any parameters of the CNN layers

It seems that there is some form of randomness or too many parameters in my model. I am aware that it is currently overfitting, but that should not be the cause of the volatility(?).
I am not too worried about the performance of the model. I would like to achieve about a 70% accuracy. All I want to do now is to stabilise the validation accuracy and to converge.

Note:

At some epochs, the training loss is very low ( <0.1 ) but validation
loss is     very high ( > 3 ).
The videos are taken on different backgrounds, but +- the same amount on each background for each dog.
Some images are a bit blurry.


texts: Keras regularizers (kernel, bias and activity) vs tf.contrib.layers.apply_regularization

texts:
I have a DCGAN set up in tensorflow that is working well on the faces in the wild dataset. As an experiment, I tried using the same architecture in keras to better understand the difference in implementation. Performance so far is much worse in keras, but there are a few things I'm uncertain about how they translate, esp regularizer.

In tf, I apply the regularizer with

tf.contrib.layers.apply_regularization

Which is just applied to the whole network it seems. However in keras this must be specified in each layer, and there are three different kinds, kernel, bias and activity regularizer.

I'm trying to udnerstand how they differ, if tf regularizer applies one of the 3 keras types, and how to implement it the same way - do I call it in each convolutional layer including the final output layer? which of the 3 is equivalent of the default regularizer in tf?

Thanks


texts: Can all known ML algorithms be written as a sequence of matrix operations?

texts:
I keep hearing that machine learning is just linear algebra.

Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can be written as a sequence of basic matrix and vector operations, such as multiply, add, sum, etc?

Would writing the algorithm in this way be able to capture all aspects of the model, e.g. architecture, activation functions, learning rate, optimization method?

Can advanced features like convolution be expressed as basic matrix operations?

Is there any ML algorithm for which this isn't true?

Is matrix algebra Turing complete?


texts: How to deal with big output values after classification layer during training?

texts: In AI libraries such as, Tensorflow, Keras etc., how the big output numbers are dealt during training process? For example, for classification layer, it will have some outputs such as [23.4, -21254.3, 32123.4]. In next step, these numbers will go into softmax function which will take power of base e with each output numbers. However, this may result in extreme big numbers in case of extremely large positive number or extremely small negative number. So, how are extreme cases dealt during training process?


texts: Supervised Recommendation System trained on labeled phrase segments

texts:
I have a big collection of phrase segments (not whole ones) with user provided labels based on text similarity:

train_data (labeled phrase segments):
"has watched movie" => phraselabel1:"movie watch activity"
"watched with friends X,Y,Z movie" => phraselabel1:"movie watch activity"
"watched show" => phraselabel2: "show watch activity"
"is jobless" => phraselabel3: "lack of job"

input phrase segment:
"watched movie in the past" => ??

output:
recommend ordered set of phrase labels: phraselabel1 then phraselabel3 based on similarity criteria

I would like to build a (scalable) experimental recommendation system that finds the most similar phrase segments that already have phrase labels to the input unlabeled one and return the best labels based on the similarity. Labels will be added by users to subset of the data initially and the system will help future users with existing labels.

The number of phrase-labels will be way less than then total number of phrase segments but more granular than topics/tags

I was thinking of the following approaches:

Do TF-IDF (http://spark.apache.org/docs/latest/mllib-feature-extraction.html) between input phrase segment and all the labeled phrase segments. By finding the most similar I can derive the labels that are best to be suggested
Do TF-IDF between input phrase segment and all the phrase-labels.
Word2Vec?? I am not sure if it can be used for phrase segments as input and not just words..
Labeled LDA (Latent Dirichlet Allocation (https://github.com/myleott/JGibbLabeledLDA) although I don't think this is readily available in Spark
Neural Network (https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier) trained on the words of all input phrase segments as inputs and phraselabels as outputs using the user click feedback on the selected phraselabel for back-propagation training.

I would appreciate if you could shed some light on which approach (from above or a new one) is better based on the use case..

Thanks :)


texts: Classifier with a single 1 value per year

texts:
Let's say I want to find the probabilities of winning the best movie category in the Oscars. I know to following rules:

There is only 1 winner per year.
Logically, the sum of the predicted probabilities for each year should add up to be 1.

I can have a year feature in my classifier but that does not mean that 1 and 2 are enforced. If I use a simple classifier like logistic regression I also don't see how having the year as a feature is going to help.

My questions are:

Is there a way to enforce those rules in a model? Does it matter?
Should I just use a model without those rules and normalize the probabilities to 1 when I try to predict for a given year?