texts: My NN has less than 1% accuracy rate, what's wrong with my model/dataset?

texts:
Data info

X_train.shape

(20160, 44)

y_train.shape

(20160,)

#first row of the dataset
X_train.values[0]

array([93., 87., 85., 79., 78., 74., 64., 60., 60., 56., 51., 41., 37.,
       36.,  0., 36.,  0., 31.,  0.,  0., 32.,  0.,  0., 31.,  0., 23.,
       22., 17.,  0.,  0., 14.,  0., 12.,  8.,  5.,  1.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.])

y_train.values[0]

106.0

Neural Network

n_cols = len(X_train.columns)
es = EarlyStopping(patience = 5)
adam = Adam(lr = 0.0001)
model = Sequential()
model.add(Dense(n_cols, activation = 'relu', input_shape = (n_cols,)))
model.add(Dense(50, activation = 'relu'))
model.add(Dense(50, activation = 'relu'))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(optimizer = adam, loss = 'mean_squared_error', metrics = ['accuracy'])

history = model.fit(X_train, y_train, batch_size = X_train.shape[0], epochs = 100, validation_data = (X_test, y_test), callbacks = [es], verbose = 0)

The code above gives me a maximum accuracy of around 0.0025, I've also tried normalising X_train but same results.

A linear regression with the exact same data returns an R2 of 0.97


texts: Why is my U-matrix visually not separating the classes?

texts:
I am visualizing the U-matrix generated using a Self Organizing Map codebook to (visually) identify regions of similarity in the data.

Although I would like to use SOM to identify clusters in unlabeled data, I am running it first with some labeled benchmark data sets to make sure that my implementation is bug free.

For Iris data set, I can see a clear demarcation between the classes suggesting that the implementation is correct. The classes have been re-labeled as 1,2,3.

However, when I run the same code on another data set, I can see that SOM was able to map members of different classes together (all the 0s are together and 1s are together and there are two distinct regions), but there is no clear demarcation.

How can I improve the U-matrix or SOM to produce a clear demarcation between the classes?


texts: I am confused about how to pre-processed packet data. Does anybody have an idea of how to encode network ports and IP addresses? My goal is to model the data in a such way that I will get a good accuracy for my model.


texts: How regression algorithm works on categorical features

texts:
I have a dataset with features that most of them are nominal categorical features, I have converted my model to indicator values,

Original

F1,F2,L
1 ,1 ,50
2 ,3 ,30

After indicator values

F1-1,F1-2,F2-1,F2-3,L
1   ,0   ,1   ,0   ,50
0   ,1   ,0   ,1   ,30

I used different regression algorithm (Poisson, Bayesian, Decision Tree reg, Decision Forest reg, Boosted decision tree reg, linear regression, neural network), but all of then have low performance (r2 ~ 20-30)

Then I was thinking how regression can find values, then I found something interesting : relation of data with label
They are like below picture

But in most of the books and examples and samples suitable data for regression are like below

And this is the point I got confused!

So my question is how regression (or which algorithms) are suitable for predicting values in high categorical data


texts: Decision Tree Classifier with Majority vote (Is 50% a majority vote)

texts:
I have a homework question that requires me to create a 3 level tree (root, intermediate and leaf) with majority vote and expand the tree if 3 level tree is not possible. I have to use ID3 algorithm. Its a manual question rather than coding assignment and dataset contains 16, 4 features and a class label with T, F as their values.
I started calculating entropies and information gain for different nodes. When I complete the 3 level tree, I can assign the classes at the leaf with 50%, 100%, 100% and 75% majority vote or confidence. My questions are

is 50% really a majority vote?
Do I need to expand on that part of the tree that has 50% confidence
or I can just assign the classes arbitrarily?

Regards


texts: Dealing with a dataset where a subset of points live in a higher dimensional space

texts:
I have a classification problem where I am dealing with economic data in a high dimensional (~400) space which includes dates, addresses, salaries, and a number of other variables. Most of the variables are made up of discrete categories and so these end up being encoded into many dimensions of binary variables.

There is a subset of the data where observations are made in two separate locations instead of just one. As a result, many of the dimensions are repeated for this subset. Most of the repeated data results in extra dimensions of binary class variables which would just be zero for the compliment set (i.e. only one location) of observations. However, one dimension is a salary where there are real values for the 2-location subset but Nans for the remaining data. To illustrate:

Obs | pay_1 | loc_1_A | loc_1_B| ... | job_1_A | job_1_B | ... | pay_2 | loc_2_A | ...
======================================================================================
1   | 50k   |   1     |   0    |     |   1     |   0     |     |   40k |   1     |
2   | 40k   |   0     |   1    |     |   1     |   0     |     |   45k |   1     |
3   | 70k   |   1     |   0    |     |   0     |   1     |     |  NaN  |   0     |
4   | 50k   |   1     |   0    |     |   1     !   0     |     |  NaN  |   0     |

Question:

How to deal with data that has this systematically disjoint set of dimensions in which it lives?

I believe for many of the variables in the higher-dimensional subset it is no problem, since they will simply equal zero for any of the data points not in this subset. But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data ($pay_2$) to have values similar to a standard gaussian. Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.

I can think of a couple of approaches, one being to have a kind of non-linear "switch" variable that turns the higher dimensions on and off:

$$y = A * pay_1 + B * \text{loc_1_A} + ... + S * ( M * pay_2 + N * \text{loc_2_A} + ... )$$

though I don't see how I can implement this in a lot of standard methods.

Another option could be to just copy the $pay_1$ values into the $pay_2$ column for those rows where there is no existing $pay_2$ value. This would seem to me like it is imposing a lot of structure onto the data that is difficult to justify. Maybe if I add another binary "switch" variable on top of this, it can separate the classes in my data space?

I know this is probably a very problem/domain-specific issue but I would like to know if there are any standard treatments for data which has this kind of structure, as I'm sure it is a pretty common scenario. Any suggestions of ways to think about this or similar problems in other fields to look at?


texts: Can i access data on one spark cluster from another?

texts:
What if I want to sample a table over hive on Spark-cluster-1, but I'm logged in on Spark-cluster-2?

  Connecting to jdbc:hive2://spark.cluster.1:10000/default;principal=hive/_HOST@US.AD.companyname.com;ssl=true

This call returns error: "Error: Could not open client transport with JDBC Uri:" when I issue the call from spark.cluster.2 using this call:

  hive -e "select * FROM database.tablename where rand() <= 0.0001 order by rand()  limit 10"

What are the limitations to do this? I should be able to read a table even if I'm not logged-in to the cluster where the hive tables reside.

This doesn't make these data fluid. The current workaround is to copy them tables manually from one cluster to the other.

Is there a better way to do this?


texts: Pattern Recognition - Kernel Density estimators 2.5.1

texts:
Please refer page 122-123 of Pattern recognition and Machine Learning - Bishop. A few equations:

  Density estimate: $$ p(\mathbf{x}) = \frac{K}{NV} \tag{2.246}$$ where
  $K$ = #points in: $N$ regions of volume $V$ each.

  Kernel function: Number $K$ of points falling within a hypercube
  centered at origin: $$\begin{equation}k(\mathbf{u}) =\begin{cases} 1, \quad |u_i| \leq \frac{1}{2} \qquad i = 1,\dots,D,\\ 0, \quad otherwise  \end{cases} \tag {2.247} \end{equation}$$

  From 2.247, the quantity $k((\mathbf{x} - \mathbf{x_n})/h)$ will be
  one if the data point $\mathbf{x_n}$ lies inside a cube of side $h$
  centered on $\mathbf{x}$, and zero otherwise. The total number of data
  points lying inside this cube will therefore be: $$K = \sum_{n=1}^{N} k \Big(\frac{\mathbf{x} - \mathbf{x_n}}{h}\Big) \tag{2.248}$$
  Substituting this expression into (2.246) then gives the following
  result for the estimated density at $\mathbf{x}$ $$p(\mathbf{x}) = \frac{1}{N}\sum_{n=1}^{N} \frac{1}{h{^D}}k\Big( \frac{\mathbf{x} - \mathbf{x_n}}{h}\Big) \tag{2.249}$$

  where we have used $V$ = $h^D$ for the volume of hypercube of side $h$
  in $D$ dimensions. Using the symmetry function $k(\mathbf{u})$, we can
  now re-interpret this equation, not as a single cube centered on
  $\mathbf{x}$ but as the sum over $N$ cubes centered on $N$ data points
  $\mathbf{x_n}$

I am struggling to follow explanation of last paragraph.
Each point $\mathbf{x_n}$ is at the center of (respective) cube - total $N$ cubes. If so, then concept of boundary/ distance from a fixed point seems to be missing - ref. 2.247, 2.248 resulting all 1's. Which points will be zero, if any.

Just a rough sketch:

Left sketch represents 2.248 - imagine few points outside the cube (=0); points inside cube=1. Right sketch is my understanding of 2.249's explanation: 3 points, all at center of respective cube. On what basis will the data points be classified as 0 or 1?


texts: Machine learning algorithms and Computer Vision technologies for detecting 52 playing cards deck

texts:
https://en.wikipedia.org/wiki/Standard_52-card_deck
https://en.wikipedia.org/wiki/Playing_card
https://www.google.com/search?q=playing+cards&client=ms-android-lava&prmd=sinv&sxsrf=ALeKk02cmtbG25vrqk59BybOJWHF7PG3Aw:1600237953234&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjcuIj2hu3rAhXtzjgGHZegCj0Q_AUoAnoECAwQAg
https://www.google.com/search?q=playing+cards&client=ms-android-lava&prmd=sinv&sxsrf=ALeKk02cmtbG25vrqk59BybOJWHF7PG3Aw:1600237953234&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjcuIj2hu3rAhXtzjgGHZegCj0Q_AUoAnoECAwQAg
https://www.instagram.com/p/CFMBjc6J7PN/?utm_source=ig_web_button_share_sheet
Are there Machine learning algorithms & Computer vision technologies which can view the 52 playing card deck?
Input : 52 playing card deck images as .jpg,.gif,.tiff
Output : Detecting the playing card names.
For example : Spade Ace, Three diamonds, Jack of Hearts, King of Clubs.
All the playing cards names should be detected by inputting playing cards images.
If no, what are the limitations? Do the 52 playing card images have to be modified?
One can also add one more playing card as "Joker" to the existing 52 playing cards added as a image as input.


texts: Sourcing (discounted) products customers want

texts:
Goal: Generate a list of 100 products per vertical (e.g. fashion, electronics) that the teams should source, discount, and list on the website over a specific period. You may assume all customers are online only.

My thinking so far:

Predict the customers that will come to the website during the
specific period (time series). Only 30-40% customers return YOY.
Understand what they want (use search data, add to basket but didn't checkout etc). Potentially segment it further by looking at those customers who generate the highest revenue in general vs one off purchasers.
Further filter those products that these customers add (or take from 'viewed or saved' state to the 'checkout' state) once the product is on deal.
Potentially use clustering to recommend products similar to those from step 3 but that have never been on a deal?

I cannot influence the amount of discount. I can only influence what products we source. Therefore I want to source the ones with the highest potential of purchasers from the customer.

Any thoughts on the approach above?


texts: How to get a count the number of observations for each year with a Pandas datetime column?

texts:
I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year.

This is the column: df_raw['filed_date'] and the output is:

198894   2017-12-05
198895   2017-12-05
198896   2017-12-05
198897   2017-12-06
198898   2017-12-06
198899   2017-12-07
Name: filed_date, Length: 198900, dtype: datetime64[ns]

I couldn't find a method to just access the date and then daisy chain it to the value_counts() method. Any advice?


texts:
I want to find outliers in power consumption in real-time, at hourly rate, i.e., at the end of the hour, I should say whether power consumption in current hour was outlier/anomalous or not.

Approach:  Till now, I am done with following steps

Say I want to find whether power usage between 9 AM to 10 AM was anomalous? For this, I first find the usage of past n days during the same time interval, then I find the mean/median of all the previous usages
Now, I have usage of the current day and the mean/median usage of previous n days. Which statistical measure should I use to declare whether current day usage was anomalous or not?

Using above approach, for 24 hours of a specific (test) day and using past 10 days consumption, I have obtained results as:

Figure interpretation: Black line represents usage of current hour of current day; red and blue lines represent mean and median of past 10 days for same time interval

From the visual inspection, I can say that the usage between 07:10  - 08:00  and between 22:10 - 23:00 is anomalous as there is big difference between actual and previous mean/median usage. I don't know which statistical measure should I use to point out such anomalous instances automatically, using the discussed approach.


texts: ML recommendation system with items organized in a tree

texts:
I would like to develop a recommendation system (probably hybrid, user-based and feature-based) for items which are organized in a tree (there are categories, divided in sub-categories, divided in sub-sub-categories). The branches have different depths. An item is associated to a unique node of this tree, and it can be at any level. The tree has about 100 nodes, and it is probably the most important information I have about my items.

Here is what I found so far :

This is a very classic situations (any shop organizes its products in categories like "electronics"/"cloths"/... and in sub-categories like "phones"/"PC"/...) but I was surprised to find very limited information about it (but I maybe used non-optimal keywords).
There is some work about graphs (which may be a term similar to "tree"), but it seems to be a different problem (e.g. graphs of people in contact with each other like in Facebook)
Here is a 4 year old message with an answer about matrix factorisation which didn't really help me (I maybe didn't understand it) : Which algorithms should I use for recommendation system using a graph database?
Here is a publication by Alibaba where the tree seems to be only used to "break down" the problem in smaller pieces : https://medium.com/@alitech_2017/improving-recommender-system-with-tree-based-deep-model-e69f4a32d014

Would you have any advice ?

Thank you


texts: How many ways are there to check model overfitting?

texts:
I am running xgboost on a regression classification problem where the model is predicting a score of how likely a gene is to cause a disease from 0-1.
I try to avoid overfitting in all the ways I can think of and the mean output of nested cross-validation is r2 0.88, I'm not sure if I can trust this or if there are other ways I can see if this is overfitting. The output r2 on just training and testing non-nested is: Train r2: 0.971 Test r2: 0.868.
So far I:

Remove features with a correlation >0.9 and remove any features with >50% missing data (this is hard to strengthen, a lot of genetic features simply have missing data for a lot of under studied genes in biology)
Have no imputation to avoid imputation bias, and since xgboost accepts missing data.
Scale features with MinMaxScaler() in scikit-learn - recommended as a good starting point and most features don't have a normal distribution
Compare 2 feature selection methods (one using features xgboost deems important from SHAP values and one using Boruta, both give 0.87-0.88 r2 on average of the  10 nested CV k-folds and only remove 3-4 out of 57 features)
Use nested kfold cross validation with 10 kfolds

The only other area I'm aware of that I haven't really explored is projection techniques. I am not sure which method would be best for this (my features are all numeric but mixed continuous or discrete data types) such as between UMAP, PCA or partial least squares.
Are there any other ways I can investigate overfitting? I have a biology background so any resources on this would be useful and any help appreciated.
I have also more manually removed some minority example genes before training (e.g. removed training genes with a 0.9 score which make up only about 1/8 of the training dataset) to give the trained model to predict and view how the model generalises to this 'new' hard to predict genes - gives them a 0.6-0.7 score when they are actually 0.9:
y_pred =[0.69412696, 0.709764, 0.6366122]

y_true = [0.9, 0.9, 0.9]

r2_score(y_true, y_pred) #outputs 0.0

10-fold nested cv r2 results per fold:
 'test_r2': array([0.8484691 , 0.86808136, 0.91821645, 0.93616375, 0.94435934,
       0.82065733, 0.84856025, 0.8267642 , 0.84561417, 0.89567455]

Edit:
A few other things I've tried:

I think I've misused classification here (and removed tag accordingly), I use regression models and I don't have labels and only continuous scores so I don't get true positives, false positives etc. to be able to do ROC. I'm not sure what other metrics are good or better than R2 for regression that I can use.

I have tried applying imputation to compare other models (random forest, SVM, and logistic rgeression with elasticnet or lasso), all models perform notably lower than gradient boosting (0.59 average nested r2 is the highest with random forest) - but I was originally concerned with biased data from imputation, is imputation worth doing to counteract overfitting?

I use GridSearch in scikit-learn for all my models with nested cross-validation, I should have included this information originally as I have been trying to always do this.

I have a biology background, so not sure about best practices for machine learning, but from this I'm suspecting random forest is better and I should be trying to do a better parameter tuning than I currently do for it, and trusting that model's result on nested CV. Is this the best approach?
Also not sure if how I tune my random forest is reasonable, currently I use:
rfr = RandomForestRegressor(random_state=seed)
rfr_params={'n_estimators':[100, 500, 1000],
             'min_samples_split': [50, 100],
             'min_samples_leaf': [50, 100],}


texts: Why is $Y=\beta_0 x^{\beta_1} e$ a linear model? When we apply the transform, it becomes $lnY = ln\beta_0+\beta_1 lnx +lne$, and why is it still linear when the $\beta_0$ part is under ln?


texts: Regression in Python with many NaN values spread across all columns

texts:
I want to do a regression to predict "value" based on the other columns from below example table. The data was collected by single indicator and not across all data points, resulting in many NaN/blank values:

value     age     education     gender
32.3                            Male
31.8                            Female
32.8              High school
33.8              Technical school
26.4              College graduate
16.3     18 - 24
35.2     25 - 34
35.5     35 - 44

I converted categorical data by using dummy variables which resulted in below column examples. I guess that the quality of my model will be affected because I have only a single 1 by row and the rest is all 0.

value   18 - 24   25 - 34   35 - 44   College   High school
32.8       0         0         0        0          1
26.4       0         0         0        1          0
16.5       1         0         0        0          0

So my question is, what is the best way to clean and convert the data for given source data structure?


texts: Classifying Sequences Where Some Sequences in Both Classes

texts:
I am building a modified RNN (specifically a GRU) to classify sequences. These sequences are of variable length, and contain "states". Each point in the sequence is categorical and they look like [A,B,C] or [A,A,C,B], etc. Classification will be 1 or 0 depending on wether or not the user in the sequence performed an action.

I face an issue where some of these sequences appear in both classes. In addition, one class occurs less than 10% of the time.

In training on a downsampled dataset, the RNN can usually reach 75+% accuracy, with similar Specificity and Sensitivities. In Testing, the sensitivity falls dramatically to under 20%, with the specificity rising to 95%+.

This is less than desirable, and I believe the issue may stem from the sequences that appear in both classes. A sequence that is positive most of the time in training will usually be negative most of the time in testing (as the testing set is unbalanced). I assume that the net learns the distributions more than the sequences, which explains the stark contrast in results.

To people with experience working on these kind of imbalances and dataset issues, is my assumption correct? What advice do you have to handle data where the same sequence can appear in both classes?

As next steps, I am thinking of weighting the minority class, and training on a non-downsampled dataset to preserve the distribution of sequences in each class. I am also considering a "representative" training and testing set, where if sequence [A,B,A] appears in a 6:1 negative to positive ratio, I will downsample to the extent that I can preserve the majority. Hopefully that will lead to less imbalance.

There is also the option to add more "depth" to the training data. Each point in the sequence has multiple features (usually hierarchical) that add more specifics about each state, this may decrease the number of sequences that appear in both classes. However, I am hesitant to make things more complex if there is a chance the network isn't learning properly.


texts: Document classification: tf-idf prior to or after feature filtering?

texts:
I have a document classification project where I am getting site content and then assigning one of numerous labels to the website according to content.

I found out that tf-idf could be very useful for this. However, I was unsure as to when exactly to use it.

Assumming a website that is concerned with a specific topic makes repeated mention of it, this was my current process:

Retrieve site content, parse for plain text
Normalize and stem content
Tokenize into unigrams (maybe bigrams too)
Retrieve a count of each unigram for the given document, filtering low length and low occurrence words
Train a classifier such as NaiveBayes on the resulting set

My question is the following: Where would tf-idf fit in here? Before normalizing/stemming? After normalizing but before tokenizing? After tokenizing?

Any insight would be greatly appreciated.

Edit:

Upon closer inspection, I think I may have run into a misunderstanding at to how TF-IDF operates. At the above step 4 that I describe, would I have to feed the entirety of my data into TF-IDF at once? If, for example, my data is as follows:

[({tokenized_content_site1}, category_string_site1),
 ({tokenized_content_site2}, category_string_site2),
...
 ({tokenized_content_siten}, category_string_siten)}]

Here, the outermost structure is a list, containing tuples, containing a dictionary (or hashmap) and a string.

Would I have to feed the entirety of that data into the TF-IDF calculator at once to achieve the desired effect? Specifically, I have been looking at the scikit-learn TfidfVectorizer to do this, but I am a bit unsure as to its use as examples are pretty sparse.


texts: How to specify different columns stacked vertically within CSV using pandas?

texts:
I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the specific rows within that parameter with using df["crit_rating"][0:3], however i'm unable to use df["agility] to address as a seperate column. Is there a way to do this without having the make another CSV file or reformatting the file?
crit_rating, DPS, DPS-Error
-1250, 4439.51, 11.2203
-1225, 4438.28, 11.3488
-1200, 4439.3, 11.2504
-1175, 4442.01, 13.0366

agility, cDPS, cDPS-Error
-1250, 1400.14, 3.62795
-1225, 1473.09, 4.28675
-1200, 1547.47, 4.47641
-1175, 1620.25, 4.63525

Where I'd like it to be:
agility, DPS, DPS-Error, crit_rating, cDPS, cDPS-Error
-1250, 1400.14, 3.62795, -1250, 4439.51, 11.2203
-1225, 1473.09, 4.28675, -1225, 4438.28, 11.3488
-1200, 1547.47, 4.47641, -1200, 4439.3, 11.2504
-1175, 1620.25, 4.63525, -1175, 4442.01, 13.0366


texts: What makes the validation set a good representative of the test set?

texts:
I am developing a classification model using an imbalanced dataset. I am trying to use different sampling techniques to improve the model performance.
For my baseline model, I defined an AdaBoost model like so:

    from sklearn.model_selection import KFold
    kf = KFold(n_splits=5, shuffle=False)

    ada = AdaBoostClassifier(n_estimators=100, random_state=42)

    params = {
        'n_estimators': [50, 100, 200],
        'random_state': [42]
    }

    grid_ada = GridSearchCV(ada, param_grid=params, cv=kf, n_jobs=-1,
                           scoring='precision').fit(X_train, y_train)

    # The best precision score is
    grid_ada.best_score_

0.5294693068932419

    # look at all the validation scores
    grid_ada.cv_results_['mean_test_score']

array([0.51916435, 0.52946931, 0.48800155])

    # check the test scores are in line with what we expect from the CV scores
    precision_score(y_test, grid_ada.predict(X_test))

0.4423076923076923
In this case, I am not able to determine if my validation result (~53%) was a good representative of my test result (~44%). And if it isn't, why is that the case
I suppose my question can be split into 3 parts:

When do we determine that a validation set is a good representative of a test set? Should the difference between the two results be between a certain range?
What are some of the reasons for large discrepancies between the validation and test set results? I know from a previous question that data leakage from training set into validation set by upsampling the data before splitting it can cause this. But are there any other obvious reasons?
Does class imbalance influence the reliability of the validation results? So should I be using StratifiedKFold as the scikit learn documentation states:

Some classification problems can exhibit a large imbalance in the distribution of the target classes: for instance there could be several times more negative samples than positive samples. In such cases it is recommended to use stratified sampling as implemented in StratifiedKFold and StratifiedShuffleSplit to ensure that relative class frequencies is approximately preserved in each train and validation fold.

UPDATE:
I have taken two additional steps that have made my validation set more representative of my test set:

I am now using the StratifiedKFold for the cross validation, like so:

    from sklearn.model_selection import StratifiedKFold
    skf = StratifiedKFold(n_splits=5, shuffle=False)

Regarding the initial split of the data between and train and test split, I am now using the stratify option in this method because I expect that future data this model will receive to make predictions will have a similar, imbalanced distribution between classes, so stratification by class percentages make sense:

    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,
                            shuffle=True, test_size=0.2, random_state=42)


texts: How to arrange the dataset/images for CNN+LSTM

texts:
I am working on an image classification problem using Transfer Learning with Resnet50 as base model (in Keras) (For example Class A and Class B).

There is a time factor involved in this classification. For example, I need sufficient evidence to make transition from one class to another.
So one of the thought came to my mind to make smooth transition is adding LSTM layer to the CNN layer (CNN+LSTM).

Need your help in understanding below queries.

Is there any way that I can add LSTM layer to the transfer learning process (assuming the CNN layer weights are not trainable)
How do I need to prepare the dataset (image frames). For example I have 10 videos each for class A and Class B. Do I need to keep the images in sequential order as it is in video. (As of now for the normal image classification, I have shuffled the image frames)
Any thought on building my own CNN + LSTM model.

A reference link would be great for 2 and 3.

Whether the image frames need to be updated as it is in the video ?

Any updates on this please.

Thank you,
Deep Guy


texts:
Actually when Im importing wordcloud module after installed on python and jupyter notebook, they said that there is no module named 'wordcloud'.
I need answer for that, please!

pip install wordcloud
from wordcloud import WordCloud


texts: Unstable results in test mode with fractional max pooling in PyTorch

texts:
I make some variants of ResNet, originally found in TorchVision, modify them, train them and so on. What I have found is that even in .eval() mode, even if I load state right before evaluation, I receive different results. The code looks like

...
imageData = imageDataset.getImage(imageNum)
imageData = np.expand_dims(imageData, 0)
pytImageData = torch.from_numpy(imageData).cuda()
...
self.loadState(self.curEpochNum)
model.eval()
self.pytOptimizer.zero_grad()
with torch.set_grad_enabled(False):
    activations = model.forward(pytImageData).cpu().numpy()

    self.loadState(self.curEpochNum)
    activations2 = model.forward(pytImageData).cpu().numpy()
    diff = activations2 - activations
    print('diff', diff.min(), diff.max())

Difference is quite high. I have found this during investigation of occlusion heatmaps, they were quite noisy (and I would even say gave strange results, but maybe this is just one more mystery of neural nets, with different nature).

ImageNet-1000 dataset,
torch version 1.5.0.dev20200128,
torchvision version 0.6.0.dev20200128

Another question appears given that this is caused by fractional max pooling: how to make it stable? As I suppose, only random mode is implemented in PyTorch, so random positions are selected for each convolution at each run. But the authour also invented pseudorandom mode. Is it possible for example to turn it on somehow?


texts: Why is input preprocessing in VGG16 in Keras not 1/255.0

texts:
I am just trying to use pre-trained vgg16 to make prediction in Keras like this.

from scipy import ndimage
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input

im = scipy.misc.imread("cat_dog/validation/cats/cat.1362.jpg").astype(np.float32)
im = scipy.misc.imresize(im, (224, 224)).astype(np.float32)

#im /= 255.0
#im = im - np.mean(im, axis=2, keepdims=True)

im = np.expand_dims(im, axis=0)
im = preprocess_input(im)

out = vgg16_model.predict(im)
np.argmax(out)

It seemed that im /= 255.0 give very bad prediction. I commented it out and it started making good prediction. I also added preprocess_input(...) but that doesn't seem to affect prediction for the few random trials I did.

The question is that according to this great blog:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

under "Using the bottleneck features of a pre-trained network: 90% accuracy in a minute", pre-trained VGG16 is in a transfer learning context. And if you look at this gist, you see this line of code:

def save_bottlebeck_features():
    datagen = ImageDataGenerator(rescale=1. / 255)
    etc.

The preprocessing of input seemed to be 1/255.0 during caching of features from the last conv layer. This is sort of puzzling. I further also looked up how preprocess_input(...) is defined in the code I have, and found for 'tf':

x /= 127.5
x -= 1.

Which you can check here.


texts: How to use neural networks with large and variable number of inputs?

texts:
I'm new to machine learning, but I have an interesting problem. I have a large sample of people and visited sites. Some people have indicated gender, age, and other parameters. Now I want to restore these parameters to each user.

Which way do I look for? Which algorithm is suitable to solve this problem? I'm familiar with Neural Networks (supervised learning), but it seems they don't fit.


texts:
I am learning python and trying myself out at mashine learning. I am reproducing a super simple example - based on the infamous iris dataset. Here it goes:

from sklearn import datasets
iris = datasets.load_iris()

X = iris.data
y = iris.target

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .5)

from sklearn import tree
nordan_tree = tree.DecisionTreeClassifier()

nordan_tree.fit(X_train, y_train)

from sklearn.metrics import accuracy_score

I get the following error message:

Traceback (most recent call last):
  File "tree3.py", line 17, in <module>
    print(accuracy_score(y_test, predictions))
NameError: name 'predictions' is not defined

I don't get it. As far as I understand, predictions is the vector containing all the predictions produced with  DecisionTreeClassifier? What am I doing wrong?


texts:
I received a dataset for analysis that had ~100 numeric columns with anonymous column names($X1$, $X2$, $X3$, etc...) and asked to do a binary classification.  My resulting classification algorithm using a SVM had good accuracy (> 95%), but I was unable to do much in the way of feature engineering or feature generation other than the standard scaling, null-value replacement, etc, since I had no intuition about the columns.
Is there any standard logic as to how to do some sort of automated feature generation, i.e. some simple mathematical combinations of various columns to create new, useful features?  Does this sort of thing have any mathematical basis for linear or tree-based models?  Or is feature engineering only really meaningful when one has intuition based on the column names...


texts: word based learning similar to what is done in mnist

texts:
I am fairly new to this field and I have a use case where I would like to classify the type of a cloth by its attribute information(like descriptions of its length, width, height, cloth type etc etc). If you look at the famous Fashion MNIST dataset, they use images to classify the type of cloth into 10 known labels. Similarly, I would like to use lot of words that describe the clothing(referring to its label) instead of having images.
Is this possible? Could some one advice on what machine learning approach can I take do this?


texts: How can I split an image into rectangles?

texts:
I have a labelled form to which people will add their name and a series of numbers. They will then take a picture of the form. Like so:

I can get decent results by simply sending this to AWS Rekognition

but I feel like it would do even better if I was able to split the image into parts before sending it to the OCR API. What techniques can I use to accomplish this?

I can modify the form for better results if that helps. Maybe make borders more or less dark? Whatever is needed.


texts: Can regression trees predict continuously?

texts:
Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $f$ wherever I want.

Are regression trees capable of finding a smooth model of the function (hence a tiny change in the input should only give a tiny change in the output)?

From what I've read in Lecture 10: Regression Trees it seems to me that regression trees basically put the function values into bins:

  For classic regression trees, the model in each cell is just a constant estimate of
  Y .

As they write "classic" I guess there is a variant where the cells do something more interesting?


texts: why does performance of machine learning models plateau after certain amount of training data

texts:
I'm referring to the below image which I came across.

The explanation seemed intuitive at first but I don't think I understand how it works. The image says that the performance of traditional machine learning algorithms plateau after a certain amount of data while that of deep learning algorithms gets better with more data. Does it have to with the feature learning where deep learning methods automatically learn the important features as opposed to the manual feature selection for machine learning models? If so, can you please elaborate on that.


texts: Log analysis dataset with labeled cybersecurity issues

texts:
I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but there is a preference on existence of known cybersecurity issues in the data.
Currently all I was able to find log datasets(HDFS, BGL) that had anomalies which were not cybersecurity issues but rather execution flow errors. Also I have found numerous amounts of network data such as in https://vizsec.org/data/, but they contain network traffic instead of logs. Also, I have found log datasets that actually had cybersecurity issues but the quantity of them were too little to train a model on.
It would also be helpful to know, how is it possible to generate such a dataset in large quantities.


texts: PicklingError in pyspark (PicklingError: Can't pickle : attribute lookup Person on __main__ failed)

texts:
I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11)

import pickle
class Person:
  def __init__(self, name, age):
    self.name = name
    self.age = age
p1 = Person("John", 36)
pickle.dump(p1,open('d.pkl','wb'))```

PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed


texts: I tried loading my saved .h5 model and predicting with that model, i'm getting error list index out of range

texts:
I saved my keras model into .h5 format. Again I've loaded that .h5 file into my colab and tried to predict with that model.
model.save("/content/drive/My Drive/Datasets/sentiment_analysis.h5")
from keras.models import load_model
loaded_model = load_model("/content/drive/MyDrive/Datasets/sentiment_analysis.h5")
loaded_model.predict("i love machine learning and google")

It's giving error list index out of range.
IndexError: list index out of range


texts: I have a data with continous variables and categorical variables. I am using Random Forest and have made my continues variables Gaussian by transformation and have standardized it. Should categorical variables be done the same?


texts: Is rolling of biased dice random phenomenon?

texts:
Random phenomenon is a situation in which we dont know what the out come is going to come.
Rolling of unbiased dice is a random phenomenon since we dont know what number is going to come. We can only say every number has
1/6 probability.
Rolling a biased coin on other hand also we dont know which number is going to come. We only know every number does not have 1/6 probability.
So does that mean rolling of biased dice is a random phenomenon ?


texts: Where is the error in the following code?

texts:
I am trying to analyze the movie database using python, downloaded from imdb. While trying to generate some plots, I am running into errors which confuses me.
I am trying to generate a matrix of small figures which can show me any hidden pattern etc. Here is the code:

fig, axes = plt.subplots(nrows=4, ncols=6, figsize=(12, 8),
                         tight_layout=True)

bins = np.arange(1950,2012,3)
for ax, genre in zip(axes.ravel(), movieGenre):
    ax.hist(movieDF[movieDF['%s'%genre]==1].year, bins=bins, histtype='stepfilled', normed=True, color='r', alpha=.3, ec='None')
    ax.hist(movieDF.year, bins=bins, histtype='stepfilled', ec='None', normed=True, zorder=0, color='grey')
    ax.annotate(genre, xy=(1955, 3e-2), fontsize=14)
    ax.xaxis.set_ticks(np.arange(1950, 2013, 30))
    ax.set_yticks([])
    ax.set_xlabel('Year')

The first hist isn't working, but the second one is working when I am commenting out the first one. Here is the traceback:

 KeyError                                  Traceback (most recent call last)
<ipython-input-158-c2e7c2737372> in <module>()
      4 bins = np.arange(1950,2012,3)
      5 for ax, genre in zip(axes.ravel(), movieGenre):
----> 6     ax.hist(movieDF[movieDF['%s'%genre]==1].year, bins=bins, histtype='stepfilled', normed=True, color='r', alpha=.3, ec='None')
      7     ax.hist(movieDF.year, bins=bins, histtype='stepfilled', ec='None', normed=True, zorder=0, color='grey')
      8     ax.annotate(genre, xy=(1955, 3e-2), fontsize=14)

/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/matplotlib/axes.pyc in hist(self, x, bins, range, normed, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, **kwargs)
   8247         # Massage 'x' for processing.
   8248         # NOTE: Be sure any changes here is also done below to 'weights'
-> 8249         if isinstance(x, np.ndarray) or not iterable(x[0]):
   8250             # TODO: support masked arrays;
   8251             x = np.asarray(x)

/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/series.pyc in __getitem__(self, key)
    477     def __getitem__(self, key):
    478         try:
--> 479             result = self.index.get_value(self, key)
    480
    481             if not np.isscalar(result):

/Users/dt/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/index.pyc in get_value(self, series, key)
   1169
   1170         try:
-> 1171             return self._engine.get_value(s, k)
   1172         except KeyError as e1:
   1173             if len(self) > 0 and self.inferred_type == 'integer':

KeyError: 0

Here is the first few columns of the data:

     imdbID     title                      rating    vote      runtime  year    genre
0   tt0111161   The Shawshank Redemption    9.3      1,439,277  142    1994 [Crime, Drama]
1   tt0468569   The Dark Knight             9.0      1,410,124  152    2008 [Action, Crime, Drama]
2   tt1375666   Inception                   8.8      1,209,159  148    2010 [Action, Mystery, Sci-Fi, Thriller]
3   tt0137523   Fight Club                  8.9      1,123,462  139    1999 [Drama]
4   tt0110912   Pulp Fiction                8.9      1,117,193  154    1994 [Crime, Drama]

movieGenre is basically collecting all the different genres from 'genre' column with duplicates removed: movieGenre = set(movieDF.genre.sum()) . I then added a single column to movieDF data frame for each genre such that if a particular movie belong to that genre, then that cell is True otherwise it is False.  So for example, for the movie Inception, the Action column is marked True but Crime column is marked False and so forth.

Thanks.


texts: SRGAN Generator Architecture: Why is it possible to do this elementwise sum?

texts:
Consider the first residual block. Its first convolution layer takes in inputs:

THe PRELU's output
64 filters(64 outputs) each one being 3*3 with a stride of (1 ; 1)

So I think that the output of this convolution will have a different shape than the PRELU's output's one.

Then there is a second convolution layer in this same first residual block. So the shape will be even more different!

The first residual blocks ends with an elementwise sum... Which has these problematic inputs:

PRELU's output
The Conv2D, BN, PRELU, Conv2D, BN output

The problem is that, as explained above, the PRELU's output will have a shape quite different than "the Conv2D, BN, PRELU, Conv2D, BN output"'s shape.

So how is it theoretically possible (then technically, using Keras) to do this elementwise sum?

Edit: by "shape", I mean "width, height and eventually depth".


texts: Is generalizing a model, then removing the generalization good for FFNNs?

texts: If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen data. But my question is, once the model gives fairly good results, isn't it advisable to remove the reguarizations then train the model again for some time, so that its predictions are more accurate?


texts: Extract features from Decision tree leaf nodes

texts:
Recently came across a coursera course on "How to win Kaggle competitions" where they explain how we can engineer a categorical feature from each leaf node of the decision tree.

[Video Link][1]

I cannot understand this concept. Any suggestion or pointers towards understanding  this will be great.

For example assume the following random training data:

Gender  Age Sample_Ftre

 M      25   1.5
 M      26   1.5
 F      28   1.5
 F      27   1.5
 M      26   1.5

Can anyone explain what will be the value of new engineered_feature from the decision tree and how to calculate it.


texts: while re-training a pre-trained model, I'm facing this issue RuntimeError: You must compile your model before using it

texts:
model summary:

RuntimeError: You must compile your model before using it.

It says that the model needs to be compiled. But as far i know, if i compile a model, all the previous trained data will be lost, and the model will be trained from scratch. I don't want this to happen because i want to use this model for further re-training purposes. Can you help me in this.

Can I re-train the model without compiling it and ,save it and use this model for future training ?


texts: Automatic caching (validation) system for pipelines?

texts:
The vast majority of my DS projects begin with the creation of a simple pipeline to

read or convert the original files/db

filter, extract and clean some dataset

which has as a result a dataset I can use to compute features and train/validate/test my model(s) in other pipelines.
For efficiency reasons, I cache the result of this dataset locally. That can be in the simplest case, for instance to run a first analysis, a .pkl file containing a pandas dataframe; or it can be data stored in a local database. This data is then typically analyzed in my notebooks.
Now, in the course of a project it can be that either the original data structure or some script used in the pipeline itself changes. Then, the entire pipeline needs to be re-run because the cached data is invalid.
Do you know of a tool that allows you to check on this? Ideally, a notebook extension that warns you if the cached data became invalid.


texts: Normalized Euclidean Distance versus cross correlation?

texts:
Normalized Euclidean Distance and Normalized Cross - Correlation can both be used as a metric of distance between vectors.

What is the difference between these metrics?  It seems to me that they are the same, although I have not seen this explicitly stated in any textbook or literature.

thank you.


texts:
I've just realized my prediction approach for LSTM might not be correct.

I am trying to predict character by character, by reading over the book. The way I've approached the problem is as follows:

   b                                    c                d                e
   ^     carry cell state forward       ^                ^                ^
LSTM_t0  ------------------------->  LSTM_t1  ----->  LSTM_t2  ----->  LSTM_t3
   ^                                    ^                ^                ^
   a                                    b                c                d

This means I have 4 timesteps, at at each one I feed next letter into LSTM, expecting it to immediately predict the next letter.

Should I instead do this:

ignore          ignore           ignore             e
  ^               ^                ^                ^
LSTM_t0  ---->  LSTM_t1  ----->  LSTM_t2  ----->  LSTM_t3
  ^               ^                ^                ^
  a               b                c                d

In the first case, I am able to get 4 loss-values, but in the second example, I only have 1 source of gradient, at _t3

My main concern is in first example, I demand LSTM to make prediction of 'b' and 'c' without supplying it enough previous context.  It's fine for 'd' and 'e', but asking for answer at timestep 0 and 1 is a bit unfair?

What would be best for this particular example?


texts: problems during training a MLP type of network

texts:
I trained a neural network model, a MLP type of network, where the first several layers are 1-D convolution for processing sequence type of input.

However, the training process looks like as follows, where the orange line represents the validation loss and the blue line represents the training loss. The validation loss is large compared to the training loss and the training loss also  stops decreasing after the first several iterations. Are there any generic guidance to improve the performance? I have about 1 million training traces, and the number of parameters of the network is about 140K.


texts: How to make model for Multivariate timeseries using tensorflow probability structural model?

texts:
I have done modelling for Univariate timeseries but while using multivariate time series ( independent features) not able to achieve result. please let me know if anybody have used .

please find reference link


texts: Hive query to get all rows where a particular column value lies in a particular precentile

texts:
I am trying to filter my rows in hive table named id_counts based on percentile values.
Lets considers the following table.

+------+----------+
|  id  | quantity |
+------+----------+
| a01  |      234 |
| a02  |      345 |
| a03  |       23 |
+------+----------+

now lets say I want to get the rows which have quantity in the 90th percentile then what query should i give.
I tried the following:

select * from id_counts having quantity>=  percentile(quantity, 0.9);

But it gives the error:
FAILED: SemanticException [Error 10025]: Line 1:54 Expression not in GROUP BY key '0.9'

Update:
I am able to resolve this using the following query but is there a simpler method?

select * from id_count a cross join
    (select percentile(quantity, 0.9) as top_ones
     from id_count) aa
where a.quantity >= aa.top_ones;

A simpler query would be appreciated. Is it possible that no join is required?


texts:
I am new to modeling, and I am practicing building a logistic regression model. I would like to create a confusion matrix, but my code doesn't seem to work.

Here is the code for the model (which works):

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

#train, test = train_test_split(hp[age], test_size=0.3)

#from sklearn import preprocessing
X = hp['age'].values.reshape((32561,1))
#X = hp[['age','hours-per-week']].values
y = hp['evalinvest'].values

LogReg = LogisticRegression()
LogReg.fit(X,y)
print(LogReg.score(X,y))

0.916710174749

Here is where I am having diffculty:

# Confusion Matrix
import numpy as np
from sklearn.metrics import *
CM = confusion_matrix(X,y)
print ("\n\nConfusion matrix:\n", CM)

It runs and outputs results, but I don't feel like it is correct.

Confusion matrix:

 [[  0   0   0 ...,   0   0   0]
 [  0   0   0 ...,   0   0   0]
 [385  10   0 ...,   0   0   0]
 ...,
 [  1   0   0 ...,   0   0   0]
 [  3   0   0 ...,   0   0   0]
 [ 33  10   0 ...,   0   0   0]]

Then, when I run the following code, it doesn't work:

tn, fp, fn, tp = CM.ravel()
print ("\nTP, TN, FP, FN:", tp, ",", tn, ",", fp, ",", fn)

error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-68-dca3ebbdc69a> in <module>()
----> 1 tn, fp, fn, tp = CM.ravel()
      2 print ("\nTP, TN, FP, FN:", tp, ",", tn, ",", fp, ",", fn)

ValueError: too many values to unpack (expected 4)


texts: Micro Average vs Macro Average for Class Imbalance

texts:
I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional neural network model for classification of the data. For evaluation I'm looking at the AUC and ROC curves. Because I have three classes I have to either use micro- or macro-average.
To calculate the micro- and macro-averaged AUC and ROC curve, I use the approach described here: https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
The micro-averaged AUC / ROC is calculated by considering each element of the label indicator matrix as a binary prediction and the macro-averaged AUC / ROC is calculated by calculating metrics for each label, and find their unweighted mean. In my case micro-averaged AUC is usually higher than macro-averaged AUC.
If we look at the sklearn.metrics.roc_auc_score method it is written for average='macro' that

This does not take label imbalance into account.

I'm not sure if for micro-average, they use the same approach as it is described in the link above.
Is it better to use for dataset with class imbalance micro-average or macro-average? That means which metric is not affected by class imbalance?  In my case micro-averaged AUC (0.85) is higher than macro-averaged AUC (0.79). When I look at the confusion matrix, the majority class is very well predicted (because the network probably learns to predict the majority class) but the minority classes are poorly predicted (almost as many false negatives as true positives). So, overall the AUC should not be that high I think.


texts: Training Image classifier on tiny dataset

texts: If we have a tiny dataset in terms of images per class where we only have one image available for a given class e.g. a Flag, will data augmentation strategy work or what should be the strategy of training in this case?


texts: Automatically clear Rstudio temp files when manually stopping background "job"

texts: When I manually stop a local background job (that is running a loop over many rstan analyses) on Rstudio, Rstudio/R treat the job as if it's failed (rather than purposely stopped), and I'm stuck with 10's of GB of files in my "C:\Users[username]\AppData\Local\Temp" folder that I must manually remove. Is there some way to have Rstudio automatically remove these files only when a job is manually stopped (as opposed to failing)?


texts: Data scaling for convolutional neural networks and other issues

texts:
My project involves deep learning on physiological data gathered from a wearable device.
I aim to evaluate the potential CNN usefulness in classifying data collected from the wearable.
The wearable collects 6 channels, each on a different scale and sampled from different distributions.
The class variable is binary. One category makes up around 3/4 of the labels, and the other - the remaining 1/4.
My dataset it quite small - 10s of  subjects with ~30 files each. The files each contain ~15k timepoints, so to enrich the dataset, I segmented the data into 1k chunks. To enrich the data even more, I also created chunks from shifted windows. In other words, each chunk is followed by another chunk with a 50% overlap, resulting in a 3D NumPy array that has 2n-1 arrays of 1k timepoints and 6 channels.
After creating these chunks, I shuffled them randomly and ran them through a simple CNN. Then, I normalized the data within each chunk and channel - once using simple Z-score transform, and then using MinMax scaling.
The network I built in TF2 is very simple, consisting of the following layers:

Conv2D(10, (1, 3), activation = 'relu' , input_shape = (1024, 6, 1);
Note regarding this layer - the convolution occurs over data channels, but I also tried using
convolution over the rows (the temporal dimension), and using a conventional 3x3 kernel too
MaxPooling2D(pool_size=(1,3))
Dropout(0.2)
Flatten()
Dense(100, activation='relu')
Dense(1, activation='sigmoid')])

I'm using Adam (default settings), with binary cross-entropy as the loss function. Eighty % of the data went into training, 20% to the test set.
I encountered 2 problems:

Overfitting when learning and testing on the normalized/standardized data
Stagnation of both the training and validation accuracy when using the raw (untransformed) data

I'm really a novice in deep learning, however from my viewpoint - the network isn't learning.
I tried modifying the network by playing with the dropout proportion, adding batch normalization when working with untransformed data, using other optimizers and learning rates, and adding more dense layers. The bottom line is that nothing seems to work.
My questions are, therefore:

Should I have scaled the entire dataset prior to segmentation?
Is adding additional convolution layers useful in this case?

I'd appreciate any answer and viewpoints.
Thanks for the patience and time.
EDIT 2020-02-05: There's a mistake in the 4th paragraph; I've transformed the data before running it through the CNN, not after. Sorry for that.


texts: How to train or approach the image datasets with different resolution in Deep Learning

texts:
Image classification:

I am having a data set of image collection more than 10k but even though all are the same image but taken in different sizes (pixels into pixels) some are in square and some are in a rectangle. When I am trying to make them similar, I have planned to zoom to the image exactly where my learning is concentrated but I don't know how to do this.

Note these are color images.

I don't whether this is the good approach or any approach is appreciable.


texts: split text data to train and test according to its word distribution

texts:
I have a list of sentences as a dataset for a classification task and a dictionary object that contains the frequency of each word in the dataset. I want to split the data into two lists of train set and test set such that the distribution of words frequency in both lists be the same.
How can I do this in python?


texts: Shapley contribution when coalition is 0

texts:
I am exploring Shapley for channel attribution based on [here][1]
Consider C1, C2, C3, C4 as 4 channels in question.
Some of the coalition does not have value, such as
(C1, C2) -> 20
(C1, C3, C4) -> 10
(C1, C2, C3, C4) -> 0

The reason being there is no conversion that went through all 4 channels.
How does Shapley works in this case ?
According to https://en.wikipedia.org/wiki/Shapley_value
At some point, there will be a marginal contribution of for C2 as
v(C1, C2, C3, C4) - v(C1, C3, C4) < 0

Is this still valid ? To interpret this, can we say C2 has a negative contribution to this coalition. But with the similar logic, since v(all Ci) = 0 , contribution of all C1, C3, C4 will also be negative in this case.

One axiom of the definition is marginal
sum(marginal v(Ci)) = v(C1, C2, C3, C4) = 0

However, when I tried Shapley, v(Ci) > 0 . How can they sum up to be = 0

Continuing Q2, I notice I don't have
sum(marginal v(Ci)) = v(C1, C2, C3, C4) = 0

.
[1]: https://medium.com/analytics-vidhya/the-shapley-value-approach-to-multi-touch-attribution-marketing-model-e345b35f3359


texts:
What are .nii files, and how is data stored in them? I have some of these and I want to know how can I create 3D image of MRI scan from them.

I can load the file in my python script using nibabel. Where to go from here next?


texts:
I have this below architecture I would like to understand below question
This is my code and my image size is 50X50

 model.add(Conv2D(16, (3,3), input_shape=(image_x, image_y, 1), activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Conv2D(32, (3,3), activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Conv2D(64, (3,3), activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Flatten())
        model.add(Dense(128, activation='relu'))
        model.add(Dropout(0.2))
        model.add(Dense(num_of_classes, activation='softmax'))`
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

My question are :
1]  dense_1 :  Flatten has 1600 input and out will be 128 so how it will select 128 out put from 1600 input?
2] In which layer lost function and optimizor will be apply.


texts: Class distribution discrepancy training/validation. Loss now uninterpretable?

texts:
I have a 3-class image classification problem. The classes are highly unbalanced (with about 98% of the images beloning to one class). To counteract this unbalanced data, I weight the losses by using the inverse of the class probabilities (1/(Proportion of class)). This means that the loss gets multiplied a great lot for the minority classes. This does help in counteracting the class imbalance.

However, for this weighted loss I use the proportions in the training dataset. In the validation dataset, the distribution is somewhat different (96.66%, 2.59%, 0.75% in val set as opposed to 98.28%, 0.98%, 0.73% in the training set). This means that if I use the weights based on the training set, mainly misclassified class 1 images gets punished a lot more than they should according to the distribution in the validation set.

The reason for this distribution discrepancy is that I can't just do a random split on my images, since they are taken by some 10 cameras all taking pictures of the same respective static scenery (highways). This means that I have to hold out all images of about 2/3 cameras as validation and use the other cameras for training, otherwise the model will be too optimistic, since validation images come from the same camera as the train images.

This makes the weighted loss uninterpretable for the validation data. Is there a way to counteract this? For instance, would giving separate weights for the train/validation sets (based on their respective distributions) help?


texts:
I have a dataset which contains various columns: numerical and categorical.

Dataset here:

I was able to process the categorical data using .astype('category') and cat.codes features in Pandas dataframe as explained here in Approach #2.

def process_categorical(self, dataset):
        """

        :param dataset:
        :return:
        """
        # Label Encoding.
        for categorical_feature in LABEL_ENCODED_FEATURES:
            categorical_feature = _to_string(categorical_feature)
            dataset[categorical_feature] = dataset[categorical_feature].astype('category')
            dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes
        # Drop previous values.
        dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)
        # Rename back categories. {'to_user_cat': 'to_user' ... }
        dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)
        return dataset

I'm able to train my model and test data.
When I generate my training dataset where categorical features were encoded. (Using .cat.codes), this numerical data is different than my predictions. Example:

dataset['user_agent_numerical'].value_counts()

10    1002
11     850
1      288
8       18
7       17
2       16
6       14
3        5
0        4
9        2
5        2
4        2

Which corresponds to: dataset.groupby(['user_agent']).size()

user_agent
24gt2wreg24h                4
Avaya one-X Deskphone     288
Deskphone                  16
FreePBX 1.8                 5
HiDude UA v3.81             2
SimpleSIP V4.3              2
Twilio Gateway             14
caller                     17
eyeBeam release 3006o      18
friendly-scanner            2
pplsip                   1002
sipcli/v1.8               850

When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)

ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,contact_user,callid,content_type,user_agent,source_ip,source_port,destination_port,contact_ip,contact_port
sip:789011972592277524@13.57.9.131,789011972592277524,13.57.9.131,78901113579131,13.57.9.131,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,163.172.120.42,60993,5060,212.129.10.158,60993

I need to do pre-processing, My prediction data contains pplsip as user-agent hence it ends up with 0, instead of 10.

How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value.

User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported here.

Complete code here.


texts: Understanding of naive bayes: computing the conditional probabilities

texts:
For a task on sentiment analysis, suppose we have some classes represented by $c$ and features $i$.

We can represent the conditional probability of each class as: $$P(c | w_i) = \frac{P(w_i|c) \cdot P(c)}{P(w_i)}$$
where $w_i$ represents each feature and $c$ is the class we have. Then empirically, we can represent $$P(w_i|c) = \frac{n_{ci}}{n_c}$$
$$P(w_i) = \frac{n_{i}}{n}$$
Our priors for each classes are then given by: $$P(c) = \frac{n_c}{n}$$ where:

$n$ is the total number of features in all classes.

$n_{ci}$ represents the number of counts of that feature $i$ in class $c$.

$n_c$ is the total number of features for the class, and

$n_i$ is the total number of features for all classes.

Is my understanding of the above correct? So given these $P(c|w_i)$ probabilities for each word, I'm the naive bayes assumption is that the words are independent, so I simply multiply each word in a document for a certain class, i.e. to compute $\prod P(c|w_i), i \in N$ where $N$ is the number of words in the document. Is this correct?

To actually compute the conditional probability numerically, would it suffice to do the following:

$$P(c | w_i) = \frac{P(w_i|c) \cdot P(c)}{P(w_i)} = \frac{n_{ci}}{n_c} \cdot \frac{n_c}{n}\cdot \frac{n}{n_i} = \frac{n_{ci}}{n_i}$$

The last part of the equation looks a bit suspicious to me as it seems way too simple to compute for a rather complex probability.


texts: Using tensorflow object detection in another model

texts:
I am trying to use Tensorflow (tf) object detection API models in another custom model I built. Specifically, I am trying to do:

jointly train tf object detection models Y with another model X.
in a parallel experiment, just train model X while obtaining tf object detection model Y predictions and incorporating it into X (in some way).

I have gone through the official tf object detection API docs and scoured the net trying to find good examples where tf object detection API was customized for cases beyond just object detection. I haven't found any. Any help or links would be appreciated.


texts: Using scikit-learn instead of mlr -- what would the code look like?

texts:
So I would like to test out scikit-learn and would like to know how the code below (using mlr) would be written in scikit-learn.

    task = makeRegrTask(data = dat, target = "medv")
    makeLearner("regr.randomForest")
    mod = train(lrn, task)
    X = dat[which(names(dat) != "medv")]
    model = Predictor$new(mod, data = X, y = dat$medv)

Any help is greatly appreciated.


texts:
I am working on a classification problem. I am using pre-trained GloVe word embedding as input I wanted to know whether adding a dropout after the embedding layer makes sense at all? I have seen a lot of notebooks on Kaggle where dropout layers are added after the embedding layer and were wondering why this is done.
model= Sequential()
model.add(Embedding(vocab_size+1,embedding_dim,input_length=max_len, weights=[embeddings_matrix],trainable=False))
model.add(Dropout(0.3))
model.add(Bidirectional(LSTM(64,return_sequences= True)))
model.add(Bidirectional(LSTM(32)))
model.add(Dense(32,activation='relu'))
model.add(Dense(1,activation='sigmoid'))


texts: Implementing Connectivity: "IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed"

texts:
I am trying to implement connectivity as a feature within my code, but am unsure of how to fix this error code. Here is my code up until the point of the error.
import mne
import matplotlib.pyplot as plt
import numpy as np

from mne.time_frequency import psd_welch
from mne.connectivity import spectral_connectivity

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, plot_confusion_matrix
from sklearn.metrics import confusion_matrix
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import KFold, cross_val_predict

#Importing data
pre = mne.io.read_raw_eeglab('Subject31pre.set')
prop = mne.io.read_raw_eeglab('Subject33prop.set')

#Create events in regular intervals, used as markers to generate epochs
preEvents = mne.make_fixed_length_events(
    pre, id=1, start=0, stop=None, duration=1.0, first_samp=True, overlap=0.0)
propEvents = mne.make_fixed_length_events(
    prop, id=2, start=0, stop=None, duration=1.0, first_samp=True, overlap=0.0)

#Plotting of Power Spectral Density
pre.plot(duration=20, n_channels=64)
pre.plot_psd()
prop.plot(duration=20, n_channels=64)
prop.plot_psd()

#Epoch parameters
event_id = None
tmin = 0
tmax = 1
baseline = None
picks = None
reject = None

#Definition of epochs, for both pre and prop data
preEpochs = mne.Epochs(pre, preEvents, event_id, tmin, tmax, proj=True, baseline=baseline, picks=picks, reject=reject, preload=True)
print(preEpochs)
propEpochs = mne.Epochs(prop, propEvents, event_id, tmin, tmax, proj=True, baseline=baseline, picks=picks, reject=reject, preload=True)
print(propEpochs)

#Merge all epochs into one variable
epochsAll = mne.concatenate_epochs([preEpochs, propEpochs])
print(type(epochsAll))

def eeg_connectivity(epochs):
    FREQ_BANDS = {"delta": [0.5, 4.5],
              "theta": [4.5, 8.5],
              "alpha": [8.5, 11.5],
              "sigma": [11.5, 15.5],
              "beta": [15.5, 30]}
    print("inside: ",type(epochs))

    sfreq = epochs.info['sfreq']

    con, freqs, times, n_epochs, n_tapers = \
        spectral_connectivity(epochs, method='coh', mode='multitaper', sfreq=sfreq,
        fmin=0.5, fmax=30, faverage=True, n_jobs=1)

    freqsNd = np.array(freqs)

    X = []
    for fmin, fmax in FREQ_BANDS.values():
        con_band = con[:, :, (freqsNd >= fmin) & (freqsNd < fmax)].mean(axis=-1)
        X.append(con_band.reshape(len(con), -1))  # -1

    return np.concatenate(X, axis=-1)

And here is the error received:
  File "", line 71, in eeg_connectivity
    con_band = con[:, :, (freqsNd >= fmin) & (freqsNd < fmax)].mean(axis=-1)
IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

Thank you.


texts: how to calculate similarity between users based on movie ratings

texts: Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A user who happened to watch the same movies rated them as 8,2,3 and an another user of the same kind rated those movies as 7,6,6 and some other user only watched first two movies and he rated them as 8,5. Now the question is to which user the main user is close. I tried to come up with functions, but they are prone to fail. Can you help me?


texts:
I am trying to retrain inception final layer on new set of images. I am using docker TensorFlow image on Windows environment. Below are the steps that I am following.

Install docker toolbox for windows.
Pulling the tensorflow docker image.

docker run -it gcr.io/tensorflow/tensorflow:latest-devel

Linking tensorflow to the folder on host machine where new set of images are stored.

docker run -it -v "$(pwd)":/tf_files/samples gcr.io/tensorflow/tensorflow:latest-devel

tf_files/samples contains the set of images on which I want to retrain the model.
cd /tensorflow

git pull
Then I am running the below script.

python tensorflow/examples/image_retraining/retrain.py \
--bottleneck_dir=/tf_files/bottlenecks \
--how_many_training_steps 500 \
--model_dir=/tf_files/inception \
--output_graph=/tf_files/retrained_graph.pb \
--output_labels=/tf_files/retrained_labels.txt \
--image_dir /tf_files/samples

Inception model is being downloaded properly but after that I am getting an error which says:

  tensorflow.python.framework.errors_impl.PermissionDeniedError: /tf_files/samples
  /AppData/Local/Application Data; Operation not permitted

Below is the exact error that I am getting.

root@54808ee651a1:/tensorflow# python tensorflow/examples/image_retraining/retra
in.py \
> --bottleneck_dir=/tf_files/bottlenecks \
> --how_many_training_steps 500 \
> --model_dir=/tf_files/inception \
> --output_graph=/tf_files/retrained_graph.pb \
> --output_labels=/tf_files/retrained_labels.txt \
> --image_dir /tf_files/samples
Traceback (most recent call last):
  File "tensorflow/examples/image_retraining/retrain.py", line 1326, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "tensorflow/examples/image_retraining/retrain.py", line 988, in main
    FLAGS.validation_percentage)
  File "tensorflow/examples/image_retraining/retrain.py", line 144, in create_image_lists
    sub_dirs = [x[0] for x in gfile.Walk(image_dir)]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 518, in walk
    for subitem in walk(os.path.join(top, subdir), in_order):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 518, in walk
    for subitem in walk(os.path.join(top, subdir), in_order):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 518, in walk
    for subitem in walk(os.path.join(top, subdir), in_order):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 499, in walk
    listing = list_directory(top)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 478, in list_directory
    compat.as_bytes(dirname), status)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.PermissionDeniedError: /tf_files/samples
/AppData/Local/Application Data; Operation not permitted


texts: Uniformity of color and texture in an image

texts:
I am new to the field of deep learning and have a problem in determining whether two images have uniform color and texture. For example, I have a

Master image -

Now, with respect to this image i need to determine whether the following images have uniform texture and color distributions -

image 1 -

image 2 -

image 3 -

I need to develop an algorithm which will evaluate these 3 images with the master image. The algorithm should approve the image 1 and reject image2 because of its color. And reject image 3 because of color and texture.

My approach for the problem was directly analyzing image for texture detection. I found that Local Binary Patterns method was good among all texture recognition methods (but I am not sure). I used its skimage implementation with opencv in python and found that the method worked.

from skimage import feature
import numpy as np
import cv2
import matplotlib.pyplot as plt

class LocalBinaryPatterns:
    def __init__(self, numPoints, radius):
        # store the number of points and radius
        self.numPoints = numPoints
        self.radius = radius

    def describe(self, image, eps=1e-7):
        # compute the Local Binary Pattern representation
        # of the image, and then use the LBP representation
        # to build the histogram of patterns
        lbp = feature.local_binary_pattern(image, self.numPoints,
            self.radius, method="uniform")
        (hist, _) = np.histogram(lbp.ravel(),
            bins=np.arange(0, self.numPoints + 3),
            range=(0, self.numPoints + 2))

        # normalize the histogram
        hist = hist.astype("float")
        hist /= (hist.sum() + eps)

        # return the histogram of Local Binary Patterns
        return hist

desc = LocalBinaryPatterns(24, 8)

image = cv2.imread("main.png")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
hist = desc.describe(gray)

plt.plot(hist,'b-')
plt.ylabel('Feature Vectors')
plt.show()

It detected the features and made a histogram of feature vectors. I plotted the histogram using matplotlib and clearly found that image 1 and image 2 texture features were almost similar to the master image. And image 3 texture features were not matching.

Then I started analyzing images for their color. I plotted the color histograms using opencv as -

import cv2
from matplotlib import pyplot as plt

def draw_image_histogram(image, channels, color='k'):
    hist = cv2.calcHist([image], channels, None, [256], [0, 256])
    plt.plot(hist, color=color)
    plt.xlim([0, 256])

def show_color_histogram(image):
    for i, col in enumerate(['b', 'g', 'r']):
        draw_image_histogram(image, [i], color=col)
    plt.show()

show_color_histogram(cv2.imread("test1.jpg"))

I found that color histogram of image 1 matched with master image. And color histograms of image 2 and 3 did not matched. In this way I figured out that image 1 was matching and image 2 and 3 were not.

But, I this is pretty simple approach and I have no idea about the false positives it will match. Moreover I don't know the approach for the problem is the best one.

I also want this to be done by a single and robust algorithm like CNN (but should not be computationally too expensive). But I have no experience with CNNs. So should I train a CNN with master images?. Please point me in the right direction. I also came across LBCNNs, can they solve the problem?. And what can be other better approaches.


texts: Which model to use for multiclass audio classification?

texts: I am working on a project wherein I want to classify Tabla taalas(patterns) and I didn't find any dataset regarding it. I am recording them myself and I've ~500 data samples recorded. What model shall I use to classify the patterns if I have less than 500 samples and 6 classes?


texts: Deep Q Learning - How is the ground truth obtained?

texts:
I am new to reinforcement learning so I apologize for the wrong use of terms, if any. In SARSA, the value of a state-action pair is updated after the robot takes an action following its internal policy to move into a new state. The policy(action) for each state is then updated at the end of each episode.

In deep Q-Learning, the goal is now to estimate the value of each action from the current state. The input is thus a one-hot vector whose dimension is equal to the number of states and the output a vector containing the scores for each action.

If my explanation is correct, then my question is, how do we train the deep net? How do we get the ground truths? The robot can only, at each episode, move in one direction from one state. That would mean that the vector containing the scores for the action is zero everywhere except for one entry...?


texts: DICE loss too low but no overlap between prediction and label

texts:
I am trying to achieve the segmentation of the bone on the cross sectional area of MRI images with the Unet I found here. The label is a binary png image which I intend to compare to my prediction. For the frames, I wrote my own data loader for my DICOM MRI files. There is a class imbalance, since the bone is a small area on the image
I am using the following dice loss in keras
def DiceLoss(targets, inputs, smooth=1e-6):

#flatten label and prediction tensors inputs = K.flatten(inputs) targets = K.flatten(targets)

intersection = K.sum(K.dot(targets, inputs)) dice = (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth) return 1 - dice

I have seen it in kaggle among other similar implementations in Keras that I found out there.
But the DICE loss is always too small (10^-4) even though there is no overlap between the prediction and the label. Why is this? should I first convert my prediction to a binary mask? if so, How can I do this in keras? I tried to convert targets to a numpy array by doing targets.numpy() and then apply a threshold, but it throws a "tensor does not have attribute numpy" error. Do you have any other idea?


texts: Machine learning method to predict event date

texts:
Let's say I have a big dataset consisting of variables including but not limited to the start/end date of loans, their notional amount, a loan prepayment indicator etc.

My goal is to create a model that will be trained on past data in order to predict the prepayment date of current loans and I was wondering which ML method would be most suitable for this case. My first thought was to handle this as a classification problem, using interval dates to predict a prepayment date interval, but I believe that there should be a more robust & sophisticated approach to this.


texts: Giving higher priority to certain inputs in SKLearn Random Forest

texts:
Here is the sample data I have:

Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)

Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)

Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)

Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)

I am using Scikit Learn Random Forest Classifier.
I want the Classifier to give higher priority to certain inputs.
For Example: 100x priority to Tag 1: 10x priority to Tag 2: 1x priority to Tag 3.

Does Scikit-Learn have any parameters where I can specify this?

[Since the purpose of the Random Forest is to learn the priority of the inputs, is my question nonsensical?]

Appreciate your insight and support.


texts:
I am trying to put numeric data into fixed number of buckets using Python/R.

I have data in key:value format {1 : 12.3, 2 : 4.7, 3 : 7.4, 4 : 15.9, ......, 50 : 24.1}, which is device_id:data_usages I need to bucket based on value into nine buckets (1,5,25,50,150,250,1000,5000,10000), So later I can see which data points are in which bucket.

What function can do this in Python OR R?


texts:
I have a basic question regarding convolutional neural network. Assume I have a set of 1000 RGB images and I train a CNN from this set. I can obviously split each of my RGB images into 3 different greyscale images, each representing red, green, and blue array - thus creating 3000 greyscale images.

My question is: if I train a CNN from those 3000 greyscale images, would I get the same parameters from the first CNN? More precisely, how does CNN behave when we feed it with RGB images? Does it extract the combined features from the RGB images or does it merely learn the features in each R,G,B channel separately?


texts: Is GridSearchCV computing SVC with rbf kernel and different degrees?

texts:
I'm running a GridSearchCV with a OneVsRestClasssifer using SVC as an estimator. This is the aspect of my Pipeline and GridSearchCV parameters:

pipeline = Pipeline([
    ('clf', OneVsRestClassifier(SVC(verbose=True), n_jobs=1)),
    ])

parameters = {
    "clf__estimator__C": [0.1, 1],
    "clf__estimator__kernel": ['poly', 'rbf'],
    "clf__estimator__degree": [2, 3],
}

grid_search_tune = GridSearchCV(pipeline, parameters, cv=2, n_jobs=8, verbose=10)
grid_search_tune.fit(train_x, train_y)

According to the documentation of SVC the degree parameter is only used by the poly kernel:

http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

  degree : int, optional (default=3)

  Degree of the polynomial kernel
  function (‘poly’). Ignored by all other kernels.

but when I see the output of my GridSearchCV it seems it's computing a different run for each SVC configuration with a rbf kernel and different values for the degree parameter.

[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=2
[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=3
[CV] clf__estimator__kernel=poly, clf__estimator__C=0.1, clf__estimator__degree=3
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=3
[CV] clf__estimator__kernel=rbf, clf__estimator__C=0.1, clf__estimator__degree=3

Shouldn't all values of degree be ignored, when the kernel is set to rbf?


texts: Can i find similar players using a clustering method like the k-mean algorithm?

texts:
I am working on a data mining project on NBA data. I want to make a recommendation system similar to the google one, where you search for players and you get recommendation for similar players. I intend to make such system based on their stats. My data looks like this

14  Tony Allen  709     357     15386   2178    4580    0.476   102     363     0.281   2076    4217    0.492   0.487   1220    1701    0.717   804     1567    2371    942     995     299     964     1675    5678

15  Al-Farouq Aminu     431     228     9988    1119    2592    0.432   188     607     0.31    931     1985    0.469   0.468   530     731     0.725   618     1724    2342    488     413     245     540     813     2956

One approach I have in mind is using k-means clustering, but I am not sure how I will use the clusters to find similar players. Each cluster will have hundreds of players, I will need further filtering.

Does anybody have any suggestions/tips for handling this problem?


texts: Dimension Reduction - After or Before Train-Test Split

texts:
Should one apply dimensionality reduction methods to the data set before or after train-test splitting?
Anyway, in case of training a model with preprocessing by dim-red, one should apply the same dim-red to the future instances to be predicted, right? But how could you reduce the dimensionality of one single instance in the same way?


texts:
I'm trying to use a Neural Network in order to predict two different values from an input that contains 24 different features. The results I've gotten so far are not good enough, so any suggestions would be appreciated since I've been stuck for some time. This is what I've done so far:
Input data:
I have an input that contains 24 different features (the total dataset has around 150,000 instances). So I've tried to standarize my input, normalize it, log transform it, and use PCA in order to reduce the dimensionality of the problem. Out of this, PCA has proven to be the best solution (using the first 5 principal components).
To make sure that input is significant, I have done a quick fit using a Random Forest Regressor and a Extra Tress Regressor to calculate the importance of each feature (after having performed PCA). And compared to a random feature, all the features that are being used seem to be significant enough for the model.
Neural Network
For the neural network, I have tried a lot of things arriving to the following architecture:
initializer_sm = tf.keras.initializers.GlorotNormal()

model_nn = keras.Sequential([
    keras.layers.Dense(128,input_dim = xnn_train.shape[1], activation='selu', kernel_initializer=initializer),
    keras.layers.Dense(256, activation='selu', kernel_initializer=initializer),
    keras.layers.Dense(256, activation='selu', kernel_initializer=initializer),

    keras.layers.Dense(2, activation = 'softplus', kernel_initializer=initializer_sm)
    ])

def custom_loss(y_true,y_pred):

    return (K.abs(y_true - y_pred)/y_true)*100

epochs = 100
opt = keras.optimizers.Nadam()
model_nn.compile(optimizer = opt, loss = custom_loss)
history_nn = model_nn.fit(xnn_train, ynn_train, epochs = epochs, batch_size = 1024, validation_data = (xnn_val,ynn_val))

With the following results:

And these are the histogram for each output predicted w.r.t the real values:

I have also tried to implement 1D convolutial layers before the architecture shown before, but the only effect that they have is that the loss function converges faster.


texts: How to improve LSTM accuracy on multiclass text classification?

texts:
So, I'm trying to build a LSTM model to classify multiclass text label. The goal is to make a prediction about user rating (1, 2, 3, 4, 5) based on their review.
My hyperparameter is like this:
# set hyperparameter
vocab_size = 5000 # make the top list of words (common words)
embedding_dim = 32
max_length = 354
oov_tok = '<OOV>' # OOV = Out of Vocabulary

Model:
model = Sequential()
model.add(Embedding(vocab_size, embedding_dim))
model.add(Dropout(0.5))
model.add(Bidirectional(LSTM(embedding_dim)))
model.add(Dense(1, activation='softmax'))
model.summary()

Compile:
model.compile(
    loss='categorical_crossentropy',
    optimizer='Adam',
    metrics=['accuracy']
)

Then, the result I get:

I only get 50% accuracy and I think it's not a good result. Any idea how can I improve the accuracy? Here's the collab: Collab's


texts: Back-propagation through max pooling layers

texts:
I have a small sub-question to this question.

I understand that when back-propagating through a max pooling layer the gradient is routed back in a way that the neuron in the previous layer which was selected as max gets all the gradient. What I'm not a 100% sure of is how the gradient in the next layer gets routed back to the pooling layer.

So the first question is if I have a pooling layer connected to a fully connected layer - like the image below.

When computing the gradient for the cyan "neuron" of the pooling layer do I sum all the gradients from the FC layer neurons? If this is correct then every "neuron" of the pooling layer has the same gradient?

For example if the first neuron of FC layer has a gradient of 2, second has a gradient of 3, and third a gradient of 6. What are the gradients of the blue and purple "neurons" in the pooling layer and why?

And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then? See the example below.

For the topmost rightmost "neuron" of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?

How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?

Please do not post a bunch of equations and tell me that my answer is right in there because I've been trying to wrap my head around equations and I still don't understand it perfectly that's why I'm asking this question in a simple way.


texts: Python Function to Incorporate Data Transform

texts:
I am new to python and I have developed a model and like to create a function for API use.

For example: predict(1,2,1) <-- call the function

if the first argument is 1 and it has 7 categories, the first 7 data input would be (1,0,0,0,0,0,0) If the second argument is 2 and it has 5 categories, the input includes the first 7 is (1,0,0,0,0,0,0,1,0,0,0) If the third argument is 1 and it has 3 categories, the input includes the first and second is (1,0,0,0,0,0,0,0,1,0,0,0,1,0,0)

How do I write that in a python function for api use? Thanks in advance.

import pandas as pd
import pickle

# load model
model = pickle.load(open('model.pkl','rb'))

def predict(Project_Sector, IncidentType, Parent_Child_Type):

    features = ['Project_Sector', 'IncidentType', 'Parent_Child_Type',]

    s = predict("Project_Sector", "IncidentType", "Parent_Child_Type")

    data_transf = pd.get_dummies(s,columns=features)

    # predictions
    result = model.predict(data_transf)

    # send back to browser
    output = {'results': int(result[0])}

    # return data
    return jsonify(results=output)


texts: Testable hypotheses construction; minimum predictive strength vs. significance

texts:
Is this null hypothesis TESTABLE?

Research Question: "Can a predictive model utilizing logistic regression be built which predicts at least one customer will churn in 90 days, and this individual prediction will be at a minimum of 70% confidence, using a chosen set of independent variables?"
Null Hypothesis: "A predictive model utilizing logistic regression cannot  predicts at least one customer will churn in 90 days, with this individual prediction being at a minimum of 70% confidence, using a chosen set of independent variables."

If so, how do you test the "70% confidence" of the particular churn event that allows you to reject the null hypothesis, at the p < 0.05 level?  I'm not sure the confidence of an individual prediction can be done, but that may depend on the modeling technique.  It may be that you drop the p < 0.05, and just use the validity of the full model (R-square and Pearson chi-square with logistic regression).

Or, is this the only way to write the null hypothesis, and the strength of the model must be left to the discussion section?

"There is no significant relationship between the full model using the chosen independent variables and customer churn."  (With r-square and chi-square showing the significance of the full model.)


texts:
I would need to analyse the structure of texts like this:

******VIRUS ALERT****** ******VIRUS ALERT****** ******VIRUS ALERT ******

There is NEW VIRUS rapidly affecting computers on the internet. This new virus is insidious, in that it transmitted as a USENET message. Usenet is the "news group" area on the internet that users can openly discuss and exchange information on a wide variety of topics.

What makes this virus DOUBLY DANGEROUS, is that it is disguised as a common chain letter. Chain letters have been passed across usenet almost since it's beginning. Lately, a common chain letter subject is MAKE MONEY FAST.

The Make Money Fast (MMF) chain is read by thousands of people daily. It is also known as: "Easy Cash", "Make Cash Fast", "Turn 5$ into $50,000" and many others. They are all basically the same scheme, in which the reader send $1 to each of the 5 people at the bottom of the list, then moves his name onto the list.

The MMF Virus, as it has been doubed, rides along on these chain letters as a "hidden binary attachment". Since most news reader programs (computer programs used to read USENET messages) will automatically decode and store binary attachments, there is NO SAFE WAY to protect yourself from infection.

The virus attackes your system the next time you run your news reader. Though the virus is transmitted during a normal usenet session, your NEXT usenet session will probably be your last for a while. As a hidden attachment, it is automatically activated with your news reader, and very quickly destroys your partition table. Generally, this is not even noticed until the next time you try to run ANY program.

The next thing the virus does is to place your micro processor into an nth-complexity infinate binary loop, quickly destroying it. This will appear at first as a normal "lock-up" but will quickley wipe out the delicate circuitry in your system.

At this point, your ONLY hope is to NOT DOWNLOAD ANY MESSAGES that have a subject similar to above. Please, FORWARD this message to ANYONE you know that reads usenet news.

In particular, I would be interested in grammar structure of each sentence (subject, verbs, object, and so on); specific pattern at the beginning of the text and at the end; and the sentiment.

Could you please give me any tips on which Python libraries I could use to do so?

Thanks


texts: Import csv file contents into pyspark dataframes

texts:
How can I import a .csv file into pyspark dataframes? I even tried to read csv file in Pandas and then convert it to a spark dataframe using createDataFrame, but it is still showing some error. Can someone guide me through this? Also, please tell me how can I import an xlsx file?
I'm trying to import csv content into pandas dataframes and then convert it into spark data frames, but it is showing the error:

"Py4JJavaError" An error occurred while calling o28.applySchemaToPythonRDD. : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

My code is:

from pyspark import SparkContext
from pyspark.sql import SQLContext
import pandas as pd
sqlc=SQLContext(sc)
df=pd.read_csv(r'D:\BestBuy\train.csv')
sdf=sqlc.createDataFrame(df)


texts: Differences between Test & Score results and calculated by Confusion Matrix (accuracy, sensitivity and specificity)

texts:
We are using Orange and have 2 files (training and testing). We apply different Learners (kNN, AdaBoost...) and get the Evaluation Results. But we have some doubts about some options in the Test & Score window.

When we apply the option "Test on train data" does it generate for each Learner a model and then applies it to the training population?

And the same about the "Test on test data", does it generate the model with the training data and this option applies it to the testing population?

Because when we get the confusion matrices and we calculate accuracy, sensitivity and specificity the values differ from the ones in the Evaluation Results.


texts: perform cluster on a multiple dimensional data in R

texts: I have a data set which has 2488 samples and each sample has 13 features.Now I want to perform cluster on this data set in R but I found k-means method usually for two dimensions data.So can any one help me? Many thanks!


texts: How to choose the features for a neural network?

texts: I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to test the network with the new feature and see the results, but is there a method to test if the feature IS UNLIKELY helpful? Like correlation measures etc?