texts: Sklearn classification report is not printing the micro avg score for multi class classification model

texts:
There are 6 class labels encoded as 0,1,2,3,4,5

While executing  classification report score it outputs accuracy,macro avg,weighted avg  .The micro average score  is missing in the output . Im not sure why micro average score is not getting printed . What should i do to get micro average score as well ?

print(classification_report(y_test, best_preds,labels=[0,1,2,3,4,5]))

          precision    recall  f1-score   support
       0       0.65      0.76      0.70        46
       1       0.74      0.56      0.64        41
       2       0.60      0.68      0.64        41
       3       0.65      0.59      0.62        41
       4       0.75      0.79      0.77        61
       5       0.76      0.70      0.73        40

accuracy                           0.69       270
macro avg      0.69      0.68      0.68       270
weighted avg   0.69      0.69      0.69       270

I went through the sklearn documentation and i changed the class labels from 0-5 to 1-6 to just simply see what scores are printed by classification report and to my suprise the micro average score was printed in the output but  the entire score in this classification report is wrong as the class labels are 0-5 and not 1-6 that is  why there is a warning printed in the output as well .

print(classification_report(y_test, best_preds,labels=[1,2,3,4,5,6]))

          precision    recall  f1-score   support
       1       0.74      0.56      0.64        41
       2       0.60      0.68      0.64        41
       3       0.65      0.59      0.62        41
       4       0.75      0.79      0.77        61
       5       0.76      0.70      0.73        40
       6       0.00      0.00      0.00         0

micro avg       0.70      0.67      0.69       224
macro avg       0.58      0.55      0.56       224
weighted avg    0.70      0.67      0.69       224

  UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
    'precision', 'predicted', average, warn_for)
  UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.
    'recall', 'true', average, warn_for)


texts: Why running the same code on the same data gives a different result every time?

texts:
I am using Keras in Jupyter Notebook.

I understood that for the same results, the random numbers should be produced from the same seed each time.

So, in the first of all my codes, I set random.seed as 1234 in a cell.

np.random.seed(1234)

Then other cells are the code for my model and the fit and evaluate code. But each time that I run the model cells, the loss values are different!

Why does it happen? How can I solve it?


texts: Grouping already clustered data (with a pre-defined x and y)

texts:
I have an already clustered data set (I wanna keep my x and y), where there's clearly a small group of elements in the middle that don't follow the expected patterns.

I can select them manually, but I wonder if there's a way of automating the selection part of these elements, efficiently.

Something like using just the grouping part of a clustering algorithm, I've been trying it with a threshold, but it doesn't produce good results in cases that won't form a circular cluster.


texts: How to use a ragged tensor with a convolutional layer?

texts:
I have textual data of various lengths for which ragged tensors seems well suited. For instance my data could look as follows :

x = tf.ragged.constant([[1,2,3,4,5,6,7], [5,6,1,2]])

I would like provide this ragged tensor to a model composed by some convolutional filters, let's say at least one filter as follows :

model = Sequential([Embedding(alphabet_size, embedding_dim),
                    Conv1D(filters=10, kernel_size=10),
                    GlobalMaxPooling1D()])

I tried to use tf.ragged.map_flat_values however I am not sure that it does what I would like, i.e : embedding each text line of the batch, convolving it, and then taking the max over each text line.

Is there a workaround to make this model work on (very) variable lengths texts (except of course using 3d tensors batches)?


texts:
What are the source and target functions in igraph package in R?  so that source(s->t) = s and target(s->t) = t

I am working on this directed graph example:

require(igraph, quietly=TRUE)
bsk <-read.table("http://www.dimiter.eu/Data_files/edgesdata3.txt", sep='\t', dec=',', header=T)

bsk.network <- graph.data.frame(bsk, directed=T)

edit: I am trying to write a function like this:

f_source <- function(str,network){
 result <- head_of(network, E(network)[str]);
}

f_source(AA%->%DD,bsk.network)

The problem is the string. It includes quotations while E(network)[str] does not work with quotations.


texts: What would be a good randomization environment for data science?

texts:
I would like to know if there are any best practices to optimize random environment. Currently I use this simple structure in my config :
from numpy.random import Generator, PCG64
rng = Generator(PCG64(42))
np.random.seed(42)

I use the rng generator for all general purposes (draw following a certain distribution, permutation of index, synthetic datapoints, etc.) and use the legacy np.random.seed to set the random state of scipy for the rvs method of scipy.stats generators.
I read somewhere in the sklearn doc (warning section here) that the sklearn.model_selection module uses the same global seed, that would the global seed set with np.random.seed isn't it ?
If you have a better understanding of how scipy and sklearn refers to the global seed and what would be a good default randomization setup, it would be very usefull. Thanks


texts: I have a csv file with time as the datetime format and want to edit the same column into miliseconds, how do I change that?


texts:
I am new to ML and I created a super basic logistic regression example with 4 points on the $x$ line that belong to two classes:

points = [[1, 1]]
points = points + [[2, 0]]
points = points + [[1.5, 1]]
points = points + [[2.5, 0]]

data   = np.array([x[:-1] for x in points])
target = [int(x[-1]) for x in points]

alg = LogisticRegression()
alg.fit(data, target)
print (alg.coef_)
print (alg.intercept_)

print (alg.predict_proba(1))
print (alg.predict_proba(1.5))
print (alg.predict_proba(2))
print (alg.predict_proba(2.5))

I expected the model to understand that anything greater than 1.5 belongs to class 0, and anything smaller than 2 belongs to class 0. However, everything is predicted to be in class 0. Also, in this case , what is the meaning for the coef and intercept? More precisely, is there a way to deduce from the coef and intercept where the model thinks the points switch from class 0 to class 1?


texts: Apply Custom Loss in Keras by getting batch_size

texts:
I have a python package/function which takes two numpy arrays and returns a value. I want to use this value as my loss in keras. How do I infer batch size to compute average loss (while using egar execution). Using the below implementation gives me error

 __index__ returned non-int (type NoneType)

import myloss
def customLoss(yt,yp):
   loss=0
   for i in range(yp.shape[0]):  #get batch size; yp is of shape (batch,200,1)
      loss = loss + myloss(yt[i,:],yp[i,:])
   return loss/yp.shape[0]

model.compile(optimizer=tf.keras.optimizers.Adam(), loss=customLoss)


texts: Does switching the positive and negative case make a difference?

texts: I believe that in most cases, the positive case is arbitrarily defined to be the case with a lower frequency. If I instead define the negative case as the one with the lower frequency, what differences would this entail for the model?


texts: Paper about AE-CNN is unclear. Deriving layers of dense blocks?

texts:
I am implementing the algorithm called Automatically Evolving CNN (AE-CNN).

Some things aren't specified which makes it a bit hard to understand what the paper actually means to say. In the chapter 3.2 Encoding Strategy it says:

  ...Note that the number of convolutional layers
  in a DB is known because it can be derived by the spatial
  sizes of input and output as well as k. ...

By DB it means a single dense block from the DenseNet algorithm. K means the K parameter for the dense block. This is the growth rate (how many layers each conv layer adds). Later on, the paper restricts the allowed values for K to just 12, 20, and 40.

I have two questions about this statement:

Given the shape of the input- and output tensors and a growth rate, how would one derive the number of layers?

The block cannot change the width and height of the feature maps (except for not padding but this is not the case) nor can it change the depth of a tensor. The depth of the output tensor can only be equal to the chosen K (I believe that the other feature maps must be ignored after the last conv layer).

The statement is only talking about single DBs but, later on, the paper says that it groups multiple DBs into units (DBU) and that the input- and output tensor shapes are only specified per units. This is contradictory. How should I interpret this? Does the statement makes sense if it would talk about DBUs instead?

Note: The paper isn't specific about it but I interpreted the paper to say that the depth of either the input tensor or output tensor (but not both) should be evolved.

Edit: A follow-up question: would it make sense to evolve this parameter (the number of layers in a DB or DBU)?


texts:
I was asked this during an interview that I did not crack.
You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one day. Task 1: Using TV instantaneous power consumption data, identify the times when the TV is “ON”. Notice that some TVs may have standby modes. Task 2: Using all the data given, design a classifier to identify times when the TV is “ON”. You may also want to train and test your designed classifier. The trained classifier should not take TV instantaneous power as input. Please provide a script which attempts task 1 and 2. Please do not provide a script without explanations.
Additional information: Data format in the attached .csv file: “House” column indicates which house it is, ranging from 1 to 3. “Time” column indicates time stamps. “TV” column indicates TV instantaneous power consumption. “Agg” column indicates total (aggregated) instantaneous power consumption. Sampling rate is 1/60 Hz.
My submission is here.
I did not receive any feedback from the employer and I don't know what went wrong. Please let me know your thoughts on what could have been done better.


texts: How can I detect significant changes in my data?


texts: Final layer of neural network responsible for overfitting

texts:
I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in the training set. The dataset is structured, ie, not images. My original features were highly right skewed; I do my best to transform these into nicer distributions by either taking logs or categorising some of them. I use an architecture of 512->128->128->1, with relu activations in every layer except the last. My loss function is sigmoid cross entropy.

The validation set contains 10 million rows. Initially the validation error goes down, but then starts to go up again after a couple of epochs. On analysing the gradients and weights of each layer, I see that the overfitting coincides with the weights on the final layer only getting larger and larger. The final layer seems to go into overdrive while the rest of the network seems to do very little learning.

I can solve the overfitting problem by using l2 regularisation, but this hurts the validation error. I have yet to find a beta regularisation parameter which doesn't hurt the best validation error I've seen. Dropout makes things even worse.

Granted, the classification problem is very difficult, with probably a very weak signal, but I find that gradient boosted trees are able to generalize much better than a simple, say, 64x64 multi-layer perceptron (the log loss on the training set is the same for both network and gradient boosted tree).

Are there any words of wisdom on how to make this network generalize better given that I've already tried:

dropout of varying degrees
l1/l2/group lasso regularization
adding noise to inputs
adding noise to gradients and weights
feature-engineering so as to remove/re-represent highly skewed features
batch normalization
using a lower learning rate on the final layer
simply using a smaller network (this is the best solution I've found)

to some or all layers. All methods hurt the validation error so much that the performance is nowhere near how well the tree model does. I would have given up by now were it not for the fact that the tree model is able to do so much better out of sample, but the training log loss for both is the same.


texts: Assign a unique cluster based on a dataframe column with KMeans Algorithm

texts:
I have the following df

    x1      x2    x3   x4
   1000   5000    0.8  restaurant1
   2000   7000   0.75  restaurant1
    500   1000    0.5  restaurant2
    700   1400    0.6  restaurant2
   1000   5000    0.8  restaurant2
    100    600    0.9  restaurant3
    200   1200    0.9  restaurant3
     50   1000    0.9  restaurant3

applying a Kmeans Algorithm for 2 clusters what happens is that y:

    x1      x2    x3   x4             Y
   1000   5000    0.8  restaurant1    1
   2000   7000   0.75  restaurant1    1
    500   1000    0.5  restaurant2    2
    700   1400    0.6  restaurant2    2
   1000   5000    0.8  restaurant2    1
    100    600    0.9  restaurant3    2
    200   1200    0.9  restaurant3    2
     50   1000    0.9  restaurant3    2

Possible Desired Outputs:

     x1      x2    x3   x4             Y
    1000   5000    0.8  restaurant1    1
    2000   7000   0.75  restaurant1    1
     500   1000    0.5  restaurant2    2
     700   1400    0.6  restaurant2    2
    1000   5000    0.8  restaurant2    2
     100    600    0.9  restaurant3    2
     200   1200    0.9  restaurant3    2
      50   1000    0.9  restaurant3    2

or

     x1      x2    x3   x4             Y
    1000   5000    0.8  restaurant1    1
    2000   7000   0.75  restaurant1    1
     500   1000    0.5  restaurant2    1
     700   1400    0.6  restaurant2    1
    1000   5000    0.8  restaurant2    1
     100    600    0.9  restaurant3    2
     200   1200    0.9  restaurant3    2
      50   1000    0.9  restaurant3    2

I would like to set this boundary: a restaurant must belong to 1 and only 1 cluster.

I understand why there is this output, but how could I avoid and fix it?

Below the code that I used in my notebook:

  #Converting float64 to numpy array
  x1=df['x1'].to_numpy()
  x2=df['x2'].to_numpy()
  x3=(df['x5']/df['x2']).to_numpy()
  x4=df_joint_raw['x4'].cat.codes.to_numpy()

  X=np.stack((x1,x2,x3,x4),axis=1)
  #Getting clusters
  y_pred=KMeans(n_clusters=2, random_state=0).fit_predict(X)


texts: How to set class-weight for imbalanced classes in KerasClassifier while it is used inside the GridSearchCV?

texts:
Could you please let me know how to set class-weight for imbalanced classes in KerasClassifier while it is used inside the GridSearchCV?
# Use scikit-learn to grid search the batch size and epochs
from collections import Counter
from sklearn.model_selection import train_test_split,StratifiedKFold,learning_curve,validation_curve,GridSearchCV
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
import numpy as np
from sklearn.model_selection import GridSearchCV
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.metrics import classification_report
import pandas as pd
from sklearn.pipeline import Pipeline

# Function to create model, required for KerasClassifier
def create_model():
    # create model
    model = Sequential()
    model.add(Dense(12, input_dim=20, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)
# load dataset
X, y = make_classification(n_classes=2, class_sep=2,weights=[0.95, 0.05], n_informative=3, n_redundant=2, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)
print('Original dataset shape {}'.format(Counter(y)))

ln = X.shape

X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)
st=StandardScaler()

# create model
model = KerasClassifier(build_fn=create_model, verbose=0)
pipeline = Pipeline(steps=[('scaler', st),
                               ('clf', model )])
# define the grid search parameters
batch_size = [20, 40, 60, 80, 100]
epochs = [ 50, 100]
param_grid = dict(clf__batch_size=batch_size, clf__epochs=epochs)
cv = StratifiedKFold(n_splits=5, random_state=42)

grid = GridSearchCV(estimator=pipeline, param_grid=param_grid,cv=cv,scoring="f1")
grid_result = grid.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
# Predictions
ypred = grid_result.predict(X_train)
print(classification_report(y_train, ypred))
print('######################')
ypred2 = grid_result.predict(X_test)
print(classification_report(y_test, ypred2))


texts: What is an appropriate machine learning technique to analyse development of status over time?

texts:
I have a dataset as follows (not the actual data, but representative):

User   Age  Salary  Status_2000  Status_2001  Status_2002  Status_2003  ... Status_2019
John   30   10000   A            A            A            B            ...  A
Mary   25   20000   A            B            B            B            ...  C
Jacob  35   30000   E            F            F            F            ...  A
Jenny  28   22000   A            F            F            A            ...  F

I have over 50,000 rows and I want to perform an analyse the development patterns of the Status from 2000 to 2019. However, the Status are encoded with letters from A to J. I would like to ask whether there are any machine learning methods or clustering methods which could help in this analysis? I initially thought of K-Means but I'm not sure if that is appropriate even with Label Encoding.

In the same dataset, there are also the following columns:

User   Age  Salary  2000_%  2001_%  2002_%  2003_%  ... 2019_%
John   30   10000   0%      0%      20%     25%     ... 100%
Mary   25   20000   0%      10%     40%     60%     ... 80%
Jacob  35   30000   0%      80%     100%    80%     ... 80%
Jenny  28   22000   0%      0%      0%      20%     ... 60%

where the data represents % of the maximum amount from 2000 to 2019, and I want to perform an analysis on the development of the %. Are there any techniques suitable for such analysis?

Thank you all for your help in advance!


texts: Accessing directory of small files as one file

texts:
I have a job that results in a directory of part- files. I'd like to read it as if it were one file. Specifically, I'd like to read it that way over a web interface.

How can I do either one of these things? Is there a hadoop component which makes this easy?


texts: Get insights from Random forest::Variable Importance analysis

texts:
I run variable importance on my Panel data (TV viewing over specific period) which consists of the old-Panel (Panel 0) and the new panel (Panel 1)

I am interested in understanding the differences in viewing based on the panel, as well as demographics like region. I used Avg daily total viewing in minutes as target variable and select age_group, region and big cities as predictor variables in R ( group by/repeat by panel ( 0 and 1) )

I visualized the result in R , in each chart’s Y-Axis is the prediction probability, X-Axis shows each variable’s values. How can we interpret the outcome to turn it to actionable insight?

In other words, what is the difference between plotting mean(prediction) vs. plotting mean(actual)? And how to make conclusion from it?


texts: Extracting possible relationships and insights from a large dataset

texts:
I have a dataset with about 26'000 variables and 33'000 continuous observations. My task is to extract any interesting insights/possible relationships between various variables.
I am looking for any Python library or workflow suggestion that can speed up this process.
One thing I already have in mind is looping through the variables and doing correlation and linear regression tests. Then conducting a deeper analysis on the variables with the best results, but I was wondering if this is an efficient approach or if there are any better methods.


texts:
I am a little bit confused.

When using mini-batches, it is a good idea to shuffle. This will not work if the training examples are dependent on each other, e.g. 5 minute voltage measurement data, where the algorithm should classify each 5 minute increment as a 1 or a 0 and  information about the previous or next measurement is required in order to correctly classify the current measurement (e.g. prediction of last training example, difference between previous and current voltage measurement, difference between current and next voltage measurement, etc.).

So, is it not recommended to use mini-batches at all for this kind of data? Or am I missing something...

Also, would this be possible with a typical logistic regression or neural network, or would it require something like an RNN (long short-term memory).


texts:
I'm new to ML. I'm taking over a Classification project which involves analyzing data for customers which returned a product and I need to determine the return reason (~10 categories).
This data was captured at the counter, and could include words like: LGTM (Looks good to me) NFF (No fault found), etc.

I have a training set of 1000 records and when using Google Prediction API I get a "classificationAccuracy" value of "0.82" and 10 labels.

Questions:
1. Any recommended API to analyze this type of data?.
2. What is a good  "classificationAccuracy" value?

Thank you


texts:
I was reading an article called "Ensemble learning in recommender systems: combining multiple user interactions for ranking personalization" where they explain a method they use called "BPR Optimization Criterion" to develop their own algorithm, and also they explain that BPR's authors apply a technique to calculate the order of relative item-item relevance for the same user to obtain a personalized ranking.

Basically, the first matrix contains only positive interaction of the user and the item, '?' is used for unknown interaction that may be positive, negative or not interaction at all. Then they compare each pair of items for each user, so if the user has interaction with the item i and has not interacted with the item j, then the user prefer i over j. But when they express that on the secondary matrix, it's all the opposite, it appears a minus sign for interaction i1,j2 and i1,j3.

So I wonder if the image is wrong or I'm missing something. Thanks.


texts: Statistics + Computer Science = Data Science?

texts:
i want to become a data scientist. I studied applied statistics (actuarial science), so i have a great statistical background (regression, stochastic process, time series, just for mention a few). But now,  I am going to do a master degree in Computer Science focus in Intelligent Systems.

Here is my study plan:

Machine learning
Advanced machine learning
Data mining
Fuzzy logic
Recommendation Systems
Distributed Data Systems
Cloud Computing
Knowledge discovery
Business Intelligence
Information retrieval
Text mining

At the end, with all my statistical  and  computer science knowledge, can i call myself a data scientist? , or am i wrong?

Thanks for the answers.


texts: What does the number after a machine learning model name mean?

texts:
I'm not sure if this is off-topic, but I'm posting here anyway.

So I saw lots of machine learning models have like an ID after their names, for example, resnet101, resnet152, densenet201 etc. What exactly do those numbers 101, 152 and 201 mean? And how it's determined?


texts: How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python?

texts: I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using python? BeautifulSoup takes a link and extract some text from the link rather than searching the relevant links by itself based on some keywords.


texts: in NLP academic paper, would it be okay to refer the "token embeddings" as "tokens"?

texts:
I am writing a paper in Natural Language Processing (NLP), and I just have a quick question about terminology.

In language models like Transformers, "token" refers to individual word in a text sequence, whereas there is a special term "token embedding" to refer to the embedding that results after token gets passed through the initial embedding layer.

Would it be problematic if I just refer a "token embedding" as a "token"?

(e.g. "interaction between hidden embeddings and token embeddings" ---> "interaction between hidden embeddings and tokens")

I am trying to accommodate the different terminologies, but my sentences are getting really wordy...

Thank you,


texts: How to create a matrix from two given vectors in R in RStudio?

texts:
Suppose, $c(1, 2, 3, 4)$ and $c(2, 4, 5, 6)$ are two vectors. Then in R or RStudio,

  How to create a $4\times 2$ matrix from these two vectors?

Also,

  how to add another vector $c(8, 9, 10, 11)$ as a column to the previous matrix?


texts: Combining parameters for Douglas-Peucker Simplification

texts:
NOTE : I'm not sure if this is the right forum for this question. if not, please advice.

Context : I am collecting a huge amount of data using an android app that is placed on a vehicle. I collect the data at ~1second intervals for about 2 hours, which gives almost 7200 data sets. These are the parameters :

Timestamp (milliseconds)
Latitude
Longitude
Speed
Acceleration

Now I was looking at ways to simplify this data, as processing and rendering these many date points, especially on mobile devices is not a good idea.

EDIT : as answer to Spacedman's comment : I want to simplify the data because most of it is redundant. The data is collected every 1 second regardless of whether the values are constant or changing. Accuracy is not crucial as its only used for displaying visual graphs on a website, drawing a polyline on a map. etc. So I want to keep only the minimum necessary data points required to reproduce the graph/line.

While searching, I came across the Ramer–Douglas–Peucker algorithm and also found a library that implements this.

The issue : I am a bit confused as to how to simplify this data. I could :

Somehow simply the entire thing by considering each data as a 5D point
, or
Generate three sets of arrays, namely : [Lat, Long, time], [Speed, time] and [acceleration, time].

So my question is :

Are there any advantages/disadvantages of using either of the two approaches?

I was thinking that - as each metric will have different patterns of variations, will combining them reduce the efficiency of the whole simplification process?

Or is it best to keep the metrics separate, so that each will be simplified to its maximum efficiency?

I am a Java/Obj-C developer, normally active on SO, and I am not an expert on these things, So I would like to know what you guys think.

Thanks in advance


texts: How to implement Classification and Anomaly detection (C++)

texts:
I am creating a system using C++(DX11) and i'm reading raw data into my program, i want to classify what the 3D data-set i'm reading in is and detect any anomalies it may have when compared to a database of the same types of item. I've not really done much in regards to machine learning aside from simple KMeans and linear regression solutions, what would be the best approach to implementing something like classification and anomaly detection in my program and what background reading/research might be required?

Currently, the program can display the raw data as a 3D image.


texts: Using DNN as the objective function for a multi-objective optimization algorithm

texts:
When creating a multi-objective optimisation/MCDM algorithm such as NSGA-ii, does it make sense to use a deep neural network trained on a supervised tabular regression prediction task, in place of a simple equation for the objective function?

Is possible or advantageous to replace a nonlinear equation with model.predict() function in Keras to be able to model more complex objective functions?

I am using pymoo with nsga-ii


texts: The images generated by generator has no labels, then how do Discriminator loss is generated on the basis of classification of generator generated images.


texts: How to merge multiple filters of a layer to a single filter in convolutional neural network?

texts:
In a convolutional neural network (CNN), the layer weights are learnt such that they extract meaningful features from the data. For each layer, can we merge multiple filters into a single filter after the CNN is trained?
Example: Suppose the first layer of a CNN has N (3x3) filters. After training, how can we merge some filters out of these into a single filter and discard the individual filters, so that the merged filter will help to extract the combined features. This might need retraining the network after merging is done. This might help to reduce the number of filters in the network and may speed up inference.
Can anyone suggest some techniques using which filters can be merged?
Thanks.


texts: What is the difference between symmetric bipartite graphs and a complete bipartite graph?

texts:
I am studying Restricted Boltzmann Machines (RBMs), and it is described as a symmetrical bipartite graph. Link

How is this different from a Complete bipartite graph? They seem to be the same to me, which is why I'm curious to why there is such a clear difference in terminology.


texts: Expected 2D array, got scalar array instead: array=11

texts:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.metrics import r2_score

# veri yukleme
veriler = pd.read_csv(r'C:\Users\k\Desktop\maaslar_yeni.csv')
# x burada bagımsız degısken y ise bagımlı degiskendir.
x = veriler.iloc[:,2:3]
y = veriler.iloc[:,5:]
X=x.values.reshape(-1,1)
Y=y.values.reshape(-1,1)

#linear regression
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(X,Y)

import statsmodels.api as sm
model=sm.OLS(lin_reg.predict(X),X)
print(model.fit().summary())

In a tutorial, the instructor just changed x and by making new X,Y. Then he tries to use LinearRegression as you see in the picture for learning the R value. But when I try this I see the above error. How can I solve this?


texts:
I have data set of cicids2017  which in total has 2830743 rows and 79 columns. Since my local jupyter was crashing a lot, SO I am using Google Colab.
But Google Colab is also crashing if some big operation occurs, or even when I try to plot. For Example,
df['Destination Port'].plot.hist(figsize=(12,6))

This will crash my Colab server.
I am looking for some library that can perform EDA automatically, and handle such big data. Thanks.


texts: copy data between two dataframes in python

texts:
I have two different dataframes, both containing a common element (identifier). I want to copy data from df1 for each identifier in df2 (each identifier can repeat maximum 5 times in df1)
df1 ---data vertically
id_column         value1
identifier1       some data1
identifier1       some data2
identifier1       some data3
identifier2       some data2
identifier2       some data1
identifier3       some data3

While df2 looks like this and this is how the information should be copied from df1:
df2 ---data horizontally
id_column        new1          new2           new3

identifier1      some data1      some data2       some data3
identifier2      some data2      some data1
identifier3      some data3

Is there a way to do this with python?
Many thanks for any help


texts: How to deal with Missing Not at Random Data for k-means clustering?

texts:
I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer.
This field has value 0 when it's inferred that the customer is not a homeowner at all (they are more likely a renter, live with relatives, etc).
I'm struggling to think of a good way to treat this value.

Does it make sense to keep this value as 0? Then my understanding is that the algorithm will interpret this as someone who doesn't own property is closely related to someone who owns an extremely low value property which doesn't seem intuitively right.

Is there a better way of dealing with this?


texts: Learning Resources for Recommendation system

texts:
Beginner here:

Could you please suggest some of the learning resources (books/youtube/articles) for beginners who want to build a recommendation system for their organization.

Have no clue about it except for Amazon and Netflix example.


texts:
This question uses the following code:
Xtrain = np.random.rand(400,1)
ytrain = f(Xtrain)
Xval = np.random.rand(200,1)
yval = f(Xval)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    #tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(1, activation='relu')
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.MeanSquaredError()
             )

model.fit(Xtrain, ytrain, epochs=500, verbose=0)

Using the command model.layers[0].get_weights() I get the following output:
[array([[-0.43412966, -0.51346564, -0.14263666,  0.8693182 , -0.4930619 ,
          1.249465  , -0.3924656 , -0.48984256, -0.55827504,  0.11134321]],
       dtype=float32),
 array([ 0.        ,  0.        ,  0.        ,  0.34663308,  0.        ,
         0.36201355,  0.        ,  0.        ,  0.        , -0.11139664],
       dtype=float32)]

And using model.layers[1].get_weights() I get this:
[array([[-0.04339373],
        [ 0.19533908],
        [-0.2295354 ],
        [ 0.903574  ],
        [-0.17581558],
        [ 0.7272965 ],
        [-0.69347996],
        [ 0.02008992],
        [-0.30351916],
        [-0.29846227]], dtype=float32),
 array([0.29466572], dtype=float32)]

I don't understand why the outgoing weights from layer 0 (second array in the model.layers[0].get_weights() list) don't match the incoming weights for layer 1 (first array in the model.layers[1].get_weights() list). And why is there an outgoing weight from layer 1? Isn't that supposed to be the final layer?


texts:
I know how Information Gain and Gini Index work in General.

I have problem figuring out how to apply these techniques in NLP and text feature extraction.

Can someone show me an example of how to implement these techniques on NLP.

Thanks


texts: Does the performance of neural networks depend on the method used to unroll weights ?

texts:
Lets say we have weights(theta1 and theta2) of neural net as:

theta1 =[1, 2, 3]
theta2= [4, 5, 6]

If we unroll these weights into a single dimension array in matlab/octave ,we get:

theta = [theta1(:);theta2(:)]
%theta = [1, 4, 2, 5, 3, 6]

If we unroll these weights in a slight different way , for example consider this python code:

theta = np.array(theta1,theta2)
  theta = theta.ravel()
  #theta = [1,2,3,4,5,6]

I have implemented a neural network using gradient descent in octave and it worked fine but the same implementation is not working in python(% accuracy = 10%). The only thing different the python implementation is the way the weights are unrolled.
So, does the performance of neural network really depend on the way the weights are unrolled ?


texts: Two datasets and two separate objectives: Is it okay to combine datasets to train for different classifications?

texts:
Some background:
I'm currently working on a research project where we have a given training set of ~6k records of economic published papers from a different research group and a separately data-scraped set of ~267k records that we want to actually classify.
The initial goal of the project is to classify the scraped data into two categories. One of the features we use for this are the economic fields where the paper comes from, noted by JEL codes. JEL codes are categorized first by letter (A-Z) as outlined here, then split into a numerical subcategory.
The fields are given in the given training set, but we decided to shift into a different method of classification that deviates from how the original authors have using JEL codes (which are economic journal codes that are tagged when a paper is published).
So my goal (or what I want to do) now, is to use a k-means++ semi-supervised algorithm to cluster the records of each individual paper into approximately 17 to 20 different JEL categories possible, where the "supervised" data will come from papers that have JEL codes that are only from one JEL category or letter.
For example, a paper with A1, A21, and A2 will be labeled as 'A' by default, while a paper with A1, B1, and Z1 will be considered an unlabeled record since it has JEL codes with multiple letters.
After I have the fields, I wish to use these features for the binary classification later in the process, which I have already coded with the split train and larger 'test' mentioned above.
QUESTION:
So in implementing this, would it be incorrect to merge the two datasets solely for the clustering portion? I would like to have more data since the training dataset doesn't cover all of the different JEL categories that the test dataset has.
In theory, I feel like it wouldn't affect the second binary classification part, but I do understand that usually training and testing sets should remain separate.
[Edit]If I can't I plan to use the clustering algorithm to output centroids for the training set, then use some hyperparameter & distance to centroid to classify (although I've read that classifying with clustering isn't the best solution). Otherwise, I would use the centroids from the full dataset (combined) to classify the fields using hyperparameters & distance to the centroid. (like k-nearest neighbors.)
If anyone has other suggestions as to how I should approach this, it is also welcome!


texts: Using recurrent neural networks for modeling errors in IMUs

texts:
Inertial measurement units (IMU), usually composed of accelerometers and gyroscopes; are well known to have inherent errors in their data, originating from bias, random walk noise, temperature dependence etc. creating a highly non-linear dependence. Typically, extended Kalman filters are used to estimate and remove these errors for stable measurement of orientations and angular velocities: but even this is not entirely accurate, as some higher order errors are ignored or approximated, and the fact that the Markov assumption ignores the effect of values older than the immediately previous value while predicting a future value. The highest levels of accuracy in IMUs are usually obtained after a rigorous factory calibration (which, in turn, makes the good ones very expensive).

In a scenario like this, how applicable would recurrent neural networks be in modeling these errors? Assuming my 'training data' consists of the accelerometer and gyroscope values, which can be fused to obtain a noisy orientation estimate, and at the same time, a much more precise orientation estimate coming from another sensor (for example, a very accurate GPS): would it be possible to replace the functionality of the Kalman filter by an RNN for error estimation and sensor output prediction?


texts: What would I prefer - an over-fitted model or a less accurate model?

texts:
Let's say we have two models trained. And let's say we are looking for good accuracy.
The first has an accuracy of 100% on training set and 84% on test set. Clearly over-fitted.
The second has an accuracy of 83% on training set and 83% on test set.

On the one hand, model #1 is over-fitted but on the other hand it still yields better performance on an unseen test set than the good general model in #2.

Which model would you choose to use in production? The First or the Second and why?


texts: Does running the script of train_test_split several times with varying test size in sklearn cause data leakage?

texts:
I ran a script of ridge and lasso regression twice with and without pca. Both times i got an okay R^2. but when i changed the train_test_split test size from 20 to 30%. My model started to over fit. So i was wondering whether this is because of changing the test size or something else.

Thank you in advance,


texts: Isn't the optimizer network in deepminds learning to learn a DRQN?

texts:
In the paper "Learning to learn by gradient descent by gradient descent" they describe an RNN which learns gradient transformation to learn an optimizer.

The optimizer network directly interacts with the environment to take actions,

$\theta_{t+1} = \theta_t + g_t(∇f(\theta_t), \phi).$ (Equation 1 from the paper)

and hence feels like a reinforcement learning problem in continuous action space. The formulation of optimization equation looks like what one would  typical do in a supervised learning problem,

$L(\phi) = E_f[f(\theta^*(f,\phi))]$ (Equation 2 from the paper)

Is this an indirect formulation of policy gradient?


texts: Object identifaction with trained CV model

texts:
I am rather new to Computer Vision and Neural Networks, but I already build the following setup:

I have a robot which crawls through the grass with a live video feed.
I build an OpenCV filtering and contour recognition routine, which can identify objects or regions in the video-feed, that might qualify to be Clover. I save those snippets as small images to train a model.
I gathered and pre-classified >6.000 images manually into "Clover" and "Grass".
I trained a Tensorflow model and it has now an accuracy of 90% for identifying the above.
When Clover is found, I mark the region and use it for object tracking.

So far so good. Now it is all about performance and quality. The issues are:

Performance: In the desired use-case, I fire up the contour identification task every x seconds on the live-video, which gathers the contour images. I then run them through the model prediction. The latter process is rather quick, because I limit the amount of contours returned from the contour identification.
The contour identification with filtering is rather slow though. During this time, unhandled frames stack up in the Queue. I do have to queue these frames to rule out video stuttering, which will cause the object tracking to loose focus.
Quality: I do not have a problem with the quality of image classification, I rather have a problem with the contour-identification via filtering. This method does not indentify all regions that might contain Clover and can be classified by the model, as it is realy hard to calibrate the filter parameters to work with very crowded and similar colored background in different light-scenarios (clover on grass, all green, sometimes cloudy sometimes direct sunlight). I can only mark ~50% of the interesting regions with this approach, although the classification would be able to tell me if the additional regions are Clover or not.

I read multiple times, that e.g. the YOLOv?-Models (which I was not able to set up and test) do both things at the same time: object identification and classification. As I understand YOLOv? is more a proconfigured toolset with underlying model(s), than a model alone, there might be a technique used in there to do the object identification (not classification) task. Maybe in an all in one approach.

So I hope there is a method, where I can use the pretrained model or other functions of Tensorflow to do the actual object identification. And I hope, that these methods combined with the classification has in the end a higher performance and outputs a higher amount of identified Clover than my current cascade.

So the question is: is there a way or a method, where I can use a pretrained model for the object isolation/identification inside of a crowded image? Or has this job always to be done via filtering?

Thanks in advance,
Seb


texts: Can numerical discrete finite data be always treated also as categorical?

texts:
In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or infinite.

I want to establish if a numerical, discrete and finite data can also be treated as categorical data.

I know that it depends on 'the meaning' of the data and requires some common sense analysis but I want to establish if the following statement is always true:

  "Numerical, discrete and finite data can also be a categorical data"

In the classification of data the numerical data is said to have 'mathematical meaning as a measure of something'. But 'technically', without assessing the meaning of the data, it does also make them capable of being a categorical data (ordinal or not), if we strip it from mathematical meaning.

Example can be a following array of items:

Energy

15
15
20
25
25

Every observation has 'Energy' characteristic it can be treated as mathematical discrete and finite numerical value which can be a measurement of energy an item has. But also it can be treated as a category: two items are in 15 category, one in category 20 and two in category 25.

Thanks for confirming this.


texts:
The derivation of the normal equation can be noted
$\theta  = (X^TX)^{-1}(X^T)y$, where $X^{-1}$ is the inverse of $X$ and can also be written $inv(X)$.

But why can't we write $inv(X^T X)$ as $inv(X)inv(X^T)$, and also use the definition of inverse $inv(X^T)(X^T) = I$?

Then $\theta  = inv(X^T X)(X^T)y = inv(X)inv(X^T)(X^T)y = inv(X)Iy = inv(X)y$

Why isn't the final result of normal equation in regression in machine learning written as  $\theta = inv(X) y$ after simplifying?


texts: Understanding and find the best eps value for DBSCAN

texts:
I'm trying to run the DBSCAN algorithm on this .csv. In the first part of my program I load it and plot the data inside it to check its distribution. This is the first part of the code:
import csv
import sys
import os
from os.path import join
from sklearn.cluster import DBSCAN
import matplotlib.pyplot as plt
import numpy as np

def load_data(file_path, file_name):
   with open(join(file_path, file_name)) as csv_file:
       data_file = csv.reader(csv_file,delimiter=',')
       temp1 = next(data_file)
       n_samples = int(temp1[0])
       print("n_samples=")
       print(n_samples)
       n_features = int(temp1[1])
       temp2 = next(data_file)
       feature_names = np.array(temp2[:n_features])

       data_list = [iter for iter in data_file]

       data = np.asarray(data_list, dtype=np.float64)

   return(data,feature_names,n_samples,n_features)

# --- Main program ---

file_path="Datasets/"
file_name3="CURE-complete.csv"
data3,feature_names3,n_samples3,n_features3 = load_data(file_path, file_name3)
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111)
fig.subplots_adjust(top=1)
ax.set_title('Dataset n. 3 of data points')
ax.set_xlabel(feature_names3[0])
ax.set_ylabel(feature_names3[1])
plt.plot(data3[:,0], data3[:,1], '.', markersize=1.2, markeredgecolor = 'blue')
plt.show()

This is how the data are in represented:

Since I need to know how many clusters should I split the dataset into, I used the Means algorithm which returned 4 (this is the elbow of the diagram, the best value):
sse = {}
for k in range(1, 11):
    kmeans = KMeans(n_clusters=k, random_state=0).fit(data3)
    sse[k] = kmeans.inertia_

kn = KneeLocator(
    list(sse.keys()),
    list(sse.values()),
    curve='convex',
    direction='decreasing',
    interp_method='polynomial',
)

plt.figure()
plt.plot(list(sse.keys()), list(sse.values()))
plt.vlines(kn.knee, plt.ylim()[0], plt.ylim()[1], linestyles='dashed')
plt.xlabel("Number of clusters")
plt.ylabel("SSE")
plt.show()

This is the diagram:

Then, I run the DBSCAN algorithm with the following parameter: min_sample = 10 (fixed value, I need to use it) and eps = 2. This is the code:
np.random.seed(5)
dbscan2 = DBSCAN(eps=2, min_samples=10).fit(data3)
fig = plt.figure(figsize=(20,10))

ax = fig.add_subplot(111)
fig.subplots_adjust(top=1)
ax.set_title('Clustered points in dataset n. 3')

ax.set_xlabel('x')
ax.set_ylabel('y')

# set the list of colors to be selected when plotting the different clusters
color=['b','g','r','c','m','y','k','w']

# number of clusters given by the diagram
k=4

#plot the dataset
for clu in range(k):
    # collect the sequence of cooordinates of the points in each given cluster (determined by clu)
    data_list_x = [data3[i,0] for i in range(n_samples3) if dbscan2.labels_[i]==clu]
    data_list_y = [data3[i,1] for i in range(n_samples3) if dbscan2.labels_[i]==clu]
    plt.scatter(data_list_x, data_list_y, s=10, edgecolors='none', c=color[clu], alpha=0.5)

plt.show()

This is the result:

Is my process incorrect? I do not understand why the plots looks so different. I would expect a digram like to the first one but with different color for each cluster.


texts: Remove rows that are too much alike not to be duplicates

texts:
I have a dataset of real estate advertisements. Several of the lines are about the same real estate property so it's full of duplicates that aren't exactly the same. What would be the best methods to remove rows that are too much alike not to be duplicates?

It looks like this :

    ID  URL CRAWL_SOURCE    PROPERTY_TYPE   NEW_BUILD   DESCRIPTION IMAGES  SURFACE LAND_SURFACE    BALCONY_SURFACE ... DEALER_NAME DEALER_TYPE CITY_ID CITY    ZIP_CODE    DEPT_CODE   PUBLICATION_START_DATE  PUBLICATION_END_DATE    LAST_CRAWL_DATE LAST_PRICE_DECREASE_DATE
0   22c05930-0eb5-11e7-b53d-bbead8ba43fe    http://www.avendrealouer.fr/location/levallois...   A_VENDRE_A_LOUER    APARTMENT   False   Au rez de chaussÃ©e d'un bel immeuble rÃ©cent,...   ["https://cf-medias.avendrealouer.fr/image/_87...   72.0    NaN NaN ... Lamirand Et Associes    AGENCY  54178039    Levallois-Perret    92300.0 92  2017-03-22T04:07:56.095 NaN 2017-04-21T18:52:35.733 NaN
1   8d092fa0-bb99-11e8-a7c9-852783b5a69d    https://www.bienici.com/annonce/ag440414-16547...   BIEN_ICI    APARTMENT   False   Je vous propose un appartement dans la rue Col...   ["http://photos.ubiflow.net/440414/165474561/p...   48.0    NaN NaN ... Proprietes Privees  MANDATARY   54178039    Levallois-Perret    92300.0 92  2018-09-18T11:04:44.461 NaN 2019-06-06T10:08:10.89  2018-09-25

So far I tried to compare the description :

df['is_duplicated'] = df.duplicated(['DESCRIPTION'])

And to compare the array of photos :

def image_similarity(imageAurls,imageBurls):
    imageAurls = ast.literal_eval(imageAurls)
    imageBurls = ast.literal_eval(imageBurls)
    for urlA in imageAurls:
        responseA = requests.get(urlA)
        imgA = Image.open(BytesIO(responseA.content))
        print(imgA)
        for urlB in imageBurls:
            responseB = requests.get(urlB)
            imgB = Image.open(BytesIO(responseB.content))
            hash0 = imagehash.average_hash(imgA)
            hash1 = imagehash.average_hash(imgB)
            cutoff = 5

            if hash0 - hash1 < cutoff:
                print(urlA)
                print(urlB)
                return('similar')
        return('not similar')

df['NextImage'] = df['IMAGES'][df['IMAGES'].index - 1]
df['IsSimilar'] = df.apply(lambda x: image_similarity(x['IMAGES'], x['NextImage']), axis=1)


texts: Interpreting the Root Mean Squared Error (RMSE)!

texts:
I read all about pros and cons of RMSE vs. other absolute errors namely mean absolute error (MAE). See the the following references:

MAE and RMSE — Which Metric is Better?
What's the bottom line? How to compare models
Or this nice blogpost, or this question in stats.stackexchange containing interesting responses, and this one in datascience.stackexchange

Still I can not get my head around something about RMSE:

Scenario: Let's say we have a regressor for predicting house prices with a MAE of 20.5\$ and a RMSE of 24.5\$. Based on MAE, I can certainly interpret that the average difference between the predicted and the actual price is 20.5\$. How can I interpret RMSE? Can we still safely say the predicted and the actual price are off by 24.5\$ at the same time base on RMSE (upper-bound of prediction error)?

In the first medium post, it says:

  RMSE does not describe average error alone and has other implications
  that are more difficult to tease out and understand.

It confuses me a little. And I could not find any reliable reference to also clearly state that one can safely interpret RSME as one does MAE. Is RMSE is simply a only mathematically more convenient for optimization etc., and we are better off with MAE for the interpretation?

Any detailed explanation is highly appreciated.


texts: Why Logistic regression into Spark Mllib does not use Maximum likelihood estimation?

texts:
During comparison estimates/ coefficients in 'R'& Spark Mllib of Logistic regression, It has been observed that estimates are not same.

On further investigation, I found that R & Mllib has different implementations for Logistic regression.

  R's glm is return a maximum likelihood estimate of the model while
  Spark's LogisticRegressionWithLBFGS is return a regularized model
  estimate.

Maximum likelihood estimates seems more efficient, as per literature available.

I am curious to know, why Spark Mllib developers has not chosen 'Maximum likelihood estimation' technique?


texts: How to properly split train and test to measure performances?

texts:
Have specific question about how to measure a model's performances:

Is it correct to have common samples between the train and the test sets?
Is it correct to have duplicates samples in the test set?


texts:
Path length between positions can be logarithmic when using dilated convolutions, left-padding for text

Could anyone elaborate on the above statement quoted from CNN and RNN comparisons ?


texts: How to compare genetic profiles or vcf files in Python?

texts:
I have hundreds of vcf file where each vcf file contains genome profile for a tissue.
A portion of the vcf file is as follows:

I can read each vcf file into a dataframe. So it would be hundreds of dataframes. Each vcf file/dataframe contains hundreds of columns and 40/50 thousands rows.
I want to see the difference in ALT column for each profile (vcf files/ dataframes) on CHROM, POS, ID and REF columns.
What would be the best way to compare these dataframes/vcf files to see any similarity on ALT column?
Thanks in advance.


texts: Keras Binary Classification val_acc won't go past ~67; Full data and code included

texts:
I'm working on a binary classification in Keras with a Tensorflow backend. No matter how much I tweak, I can't seem to get my model past a val_acc of 67%. Is there something I'm missing, or is this just simply as accurate as I can get with my data?

Link to the data I am using

My Code

Load and Balance dataset to a 1:1, and create validation data.

from sklearn.utils import resample

raw_data = pd.read_csv('Data.csv')

df_majority = raw_data[raw_data['RESULT']==0].iloc[1:-2,0:3].dropna()
df_minority = raw_data[raw_data['RESULT']==1].iloc[1:-2,0:3].dropna()

print(raw_data['RESULT'].value_counts())

df_majority_downsampled = resample(df_majority,
                          replace=False,
                 n_samples=raw_data['RESULT'].value_counts()[1],
                             random_state=123)

# Combine minority class with downsampled majority class
df_downsampled = pd.concat([df_majority_downsampled,df_minority])

# Display new class counts
print(df_downsampled['RESULT'].value_counts())
print(numpy.unique(df_downsampled['RESULT']))

X = df_downsampled.iloc[1:-2,0:2].dropna()
Y = df_downsampled.iloc[1:-2,2:3].dropna()

X, XTest, Y, YTest = train_test_split(X, Y, test_size = 0.3, random_state = 0)

print(YTest['RESULT'].value_counts()) #Just a double check to make

Create Model

def create_model(activation):
    model = Sequential()

    model.add(Dense(128,activation=activation,input_dim=2))
    model.add(BatchNormalization())

    model.add(Dense(64,activation=activation))
    model.add(BatchNormalization())
    model.add(Dropout(0.2))

    model.add(Dense(32,activation=activation))
    model.add(BatchNormalization())
    model.add(Dropout(0.2))

    model.add(Dense(16,activation=activation))
    model.add(BatchNormalization())
    model.add(Dropout(0.2))

    model.add(Dense(8,activation=activation))
    model.add(BatchNormalization())
    model.add(Dropout(0.2))

    model.add(Dense(4,activation=activation))
    model.add(BatchNormalization())
    model.add(Dropout(0.2))

    model.add(Dense(2,activation=activation))
    model.add(BatchNormalization())
    model.add(Dropout(0.2))

    model.add(Dense(1,activation='sigmoid'))
    # load weights
    model.load_weights("weights.best.hdf5")
    model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.0001),metrics=['accuracy'])

return model
model = create_model('relu')

filepath       ="weights.best.hdf5"
checkpoint     = ModelCheckpoint(filepath,monitor='val_acc',verbose=1,save_best_only=True,mode='max')
callbacks_list = [checkpoint]

history = model.fit(X,Y,epochs=2000,batch_size=32, shuffle = True,validation_data = (XTest,YTest), verbose = 0,
     callbacks=callbacks_list)

Predict and get score

from sklearn.metrics import roc_auc_score

predict = model.predict_classes(X)
print(numpy.unique(predict))

#for index,val in enumerate(predict):
#print("Predicted: %s, actual: %s, for val %s"(val[0],Y.iloc[index].values,X.iloc[index].values))

predict = [val[0] for val in predict]
print("ras score: ",roc_auc_score(Y,predict))

predict = model.predict(numpy.array([0.0235,0.5]).reshape(-1,2))
print(predict[0][0])

Result using current model

Sok suggestion of placing Dropout before Dense: val_acc =0.65990


texts: is there a way to customize my loss function to increase recall in one class only?

texts:
First, thank you.

I'm using Tensorflow 1.13 with Python 3. I'm trying to create a CNN classifier on data with 4 classes, one-hot encoded. However, the insights I need it to produce are really around spotting either classes 0 or 3. If it tells me a data row is of class 1 or 2, that's of little use to my goals. Output from sklearn classification_report is below and I want to increase recall on classes 0 and 3 during model training. Precision would be nice, too.

               precision   recall   f1-score  support

          0       0.13      0.42      0.20      1484
          1       0.15      0.74      0.24       696
          2       0.81      0.34      0.48     12325
          3       0.13      0.23      0.17      1495

avg / total       0.65      0.35      0.41     16000

I'm relatively new to both python and TF, having gone through several tutorials, and am just now starting to build my own models. I've not really been able to stretch myself out past capabilities discussed in any of the learning courses I've watched. I know that Keras, for example, has custom loss function capability, but I'm not sure I understand how to integrate it into what I've already done. Relevant bits from my existing project are below.

# my current loss function, what I want to
# replace because just plain accuracy isn't
# terribly helpful
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_true,logits=y_pred))

# setting up the trainer
optimizer = tf.train.AdamOptimizer(learning_rate=0.0001)
train = optimizer.minimize(cross_entropy)

# run it
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    for i in range(steps):
        batch_x, batch_y = dataHelper.next_batch(100)

        sess.run(train,feed_dict={x:batch_x,y_true:batch_y,hold_prob:0.5})
        if (i + 1) % 100 == 0:
            print('on step {}'.format(i))
            preds_decoded = tf.argmax(y_pred,1)
            trues_decoded = tf.argmax(y_true,1)
            final_predictions, true_values = sess.run([preds_decoded, trues_decoded],
                                    feed_dict={x:price_data_only_test,y_true:test_labels,hold_prob:1.0})
            print(classification_report(y_true=true_values,y_pred=final_predictions))
```


texts: Time series forecasting with constraints

texts: I want to predict the passenger flow volume of an airline route, which subjects to supply capacity constraints of the route (i.e., the passenger flow volume should not be higher than the supply capacity). Are there any algorithms that could be used to do this kind of forecasting problem with constraints?


texts:
Say you had a set of users, tens of thousands.  You have time series of each of their behaviors using this app.  How might you use these time series to predict future behavior of new users?

The intuitive solution is to feature engineer behavior of users and use that to train a model, such as average weekly minutes in app and things like that.  My issue with this approach is that you lose a great deal of information.

What I'm wondering is if there's a better technique for aggregating a large set of time series data to build a predictive model.  Perhaps an LSTM would work but that seems like it wouldn't capture the nuances of the dataset and I don't believe they're typically used to aggregate predictions on a variety of individual samples.


texts:
I have a task where the input is a parsed document (i.e., full text in 1 string or tokens) and I need to classify parts of the text into say 5 classes (i.e., 5 tokens from the entire text are labeled into 5 different classes).
Example:
Document #1: "... cat ..." (the token "cat" belongs to class "0" which is animals)
Document #2: "... fish ..." (the token "fish" belongs to class "0" which is animals)
It is important to note that at inference time, I have the entire document (in text), and so most of the tokens from it do not belong to any of the classes.
What would be a good approach to this task? I thought about a simple classification problem where I take the labeled tokens from each document and input it into an RNN classifier, but that ignores the rest of the document and at test time irrelevant tokens can have larger probabilities than the labeled tokens.
I also had an idea inspired by YOLO, and maybe apply a 1D CNN object detector (with the respective number of classes) on the entire text. Is this reasonable?
Thanks.


texts:
I have not been able to find an answer, so if it is out there, please let me know.

I would like to calculate the amount of time, that a uC needs to give me an Output of an CNN. Therefore, I would like to know, how many computations does a convolution neural network do?

I understand how many parameters it has and how many calculations it does (if every convolution layer has 1 filter).

My CNN has the following design:

sample_length = 24
cnn = Sequential()
cnn.add(Conv1D(8, 3, activation='relu', input_shape=(sample_length, 1)))
cnn.add(Conv1D(7, 3, activation='relu'))
cnn.add(MaxPooling1D(2))
cnn.add(GlobalAveragePooling1D())
cnn.add(Dropout(0.5))
cnn.add(Dense(2,activation='softmax'))

The input layer has 24 neurons and the first convolution layer has 8 filters with a Kernel size of 3. So, I need (24 - 3 + 1) dot products of  size 3 and (24 - 3 + 1) additions for the bias for one filter. That gives me 176 dot products with size 3 and 22 additions of 2 numbers. The output now has shape (22, 8).

Let us assume, the next layer has just 1 filter with Kernel size of 3. How does the filter slide over this layer to get an output size of (20,1)? Does it have to compute a dot product for the first 3 neurons of each "dimension" and then add them together?


texts: When do we say that the dataset is not classifiable?

texts:
I have many times analysed a dataset on which I could not really do any sort of classification. To see whether I can get a classifier I have usually used the following steps:

Generate box plots of label against numerical values.
Reduce the dimensionality to 2 or 3 to see if classes are separable, also tried LDA sometimes.
Forcefully try to fit SVMs and Random Forests and look at feature-importance to see if the features make any sense or not.
Try to change the balance of classes and techniques like under-sampling and over-sampling to check if class imbalance might be an issue.

There are many other approaches I can think of, but have not tried. Sometimes I know that these features are not good and not at all related to the label we are trying to predict. I then use that business intuition to end the exercise, concluding that we need better features or totally different labels.

My question is how does a Data Scientist report that the classification can not be done with these features. Is there any statistical way to report this or fitting the data in different algorithms first and looking at validation metric is the best option?


texts:
Can anyone recommend a tool to quickly label several hundred images as an input for classification?
I have ~500 microscopy images of cells. I would like to assign categories such as 'healthy', 'dead', 'sick' manually for a training set and save those to a csv file.
Basically, the same as described in this question, except I do not have proprietary images, so maybe that opens up additional possibilities?


texts: Is copying parameter considered as plagarism?

texts:
So my friends and i are writing a kaggle assignment and the base code is written by me.

One of my friend use my base code(feature engineering, labeling, etc.) and put it into a loop to find the best parameter.

And the resulting parameter has 7 decimals accuracy which outperforms every other people's result.

So if we copying his parameter is it consider as cheating?


texts:
Assume we train a KMeans model using data X. This will give a set of centroids that can be used to cluster data X* using a Nearest Centroid Classifier.

If we use a density-based model such as DBSCAN to train a model using data X, how can we use it to classify future data X*? Is KNN suitable?

Thanks!


texts: how to apply feature selection on LSTM-RNN?

texts: am doing my research using lstm-rnn algorithm. i have time-series and non time-series features. how to apply lstm on my dataset? and also how to apply feature selection mechanism to select features?


texts:
I am exploring the use of lime for spam/ham categorisation.
specifically I have a data frame having list of messages.
I would need to identify which messages are spam and which ones are ham by using a set of words (100).
I would need to find to test the accuracy of the model. I found some articles on towardsdatasciene and medium that helped me a bit, but I would need a really small example on what I would need (already labelled messages? If yes, I should manually label them first, or should I use some algorithm to label them as spam/ham? How could I predict a new message?...).

Any explanation and comment would be greatly appreciated!


texts: What to consider before learning a new language for data analysis

texts:
I'm currently in the very early stages of preparing a new research-project (still at the funding-application stage), and expect that data-analysis and especially visualisation tools will play a role in this project.

In view of this I face the following dilemma: Should I learn Python to be able to use its extensive scientific libraries (Pandas, Numpy, Scipy, ...), or should I just dive into similar packages of a language I'm already acquainted with (Racket, or to a lesser extent Scala)?

(Ideally I would learn Python in parallel with using statistical libraries in Racket, but I'm not sure I'll have time for both)

I'm not looking for an answer to this dilemma, but rather for feedback on my different considerations:

My current position is as follows:

In favour of Python:

Extensively used libraries
Widely used (may be decisive in case of collaboration with others)
A lot of online material to start learning it
Conferences that are specifically dedicated to Scientific Computing with Python
Learning Python won't be a waste of time anyway

In favour of a language I already know:

It's a way to deepen my knowledge of one language rather than getting superficial knowledge of one more language (under the motto: you should at least know one language really well)
It is feasible. Both Racket and Scala have good mathematics and statistics libraries
I can start right away with learning what I need to know rather than first having to learn the basics

Two concrete questions:

What am I forgetting?
How big of a nuisance could the Python 2 vs 3 issue be?


texts: Which type of classification best suits this scatter plot?

texts:
The following image shows a scatter plot of my data. The Y axis points are the labels, labeled from 1 to 6 and X-axis are dimensionally reduced values of all my features. I reduced them for better visualization.

I have used decision Trees and Random forests both give a score of 0.74 and 0.79 respectively.

I have also tried SVM and Naive Bayes just for seeing the score they gave 0.37 and 0.33 respectively.

I was using 75% to train and the rest to test.

Which is the best classification algorithm for this data? Multiclass Classification


texts: Calculate whether datapoints are part of a larger distribution

texts:
I have some normally distributed variables (~800) and some variables that are in some way special (~30). I need to find out whether the special ones can be considered normal members of the distribution.

I drew a histogram of the full distribution with the special members as scatter. In addition to that, the normal distribution of all variables and that of the special members. The plot shows these graphics for the strongest difference between the calculated distributions. Green is the normal distribution of all members, orange that of the special ones.

How do I calculate the likelihood of whether the special members are part of the larger ensemble, i.e. they don't stand out?


texts: Training an acoustic model for a speech-to-text engine

texts: What are the steps for training an acoustic model? The format of the data (the audio) includes its length and other characteristics. If anyone could provide a simple example of how to train an acoustic model, it would be greatly appreciated.


texts: What is the name for this chart which splits a quantity by allocation to a class hierarchy?

texts:
I need to create this type of chart, where you can see a quantity is split into a hierarchy of classes or taxonomy. In the example below the quantity is a household monthly income, and the classes are different monthly expenditures, classified broadly to the left, and progressively split into more detailed sub-classes.

However, I don't know what it is called and thus cannot find which tool to use.


texts: Multivariate non-negative, discrete time series forecasting with neural networks

texts:
I have a data set obtained from the event log of a bunch of access points in a building. At the moment I am interested in forecasting, for each access point, the number of connected devices at each access point on a given point in time. The features in this data set are timestamp, containing the datetime and AP1, ... , AP9, containing the (relative) number of connected devices for each access point. An example:

> synced_logs[500:515,]
              timestamp 1 2  3  4  5  6  7  8  9
500 2017-09-25 12:55:00 1 0  1 -1 -1  0  1 -1  0
501 2017-09-25 12:56:00 1 0  1 -1 -1  0  1 -1  0
502 2017-09-25 12:57:00 1 0  1 -1 -1  0  1 -1  0
503 2017-09-25 12:58:00 1 0 -1 -1  1  0  1 -1  0
504 2017-09-25 12:59:00 1 0 -1  1  1  0 -1 -1  0
505 2017-09-25 13:00:00 1 0  1  1  1  0 -1 -1  0
506 2017-09-25 13:01:00 1 0  0  1 -1  1 -1 -1  0
507 2017-09-25 13:02:00 1 0  0  1 -1  0 -1 -1  0
508 2017-09-25 13:03:00 1 0 -2  1 -1 -1  3  3  0
509 2017-09-25 13:04:00 1 0  1  0 -1 -1  1 -2  0
510 2017-09-25 13:05:00 1 0  0  0 -1  1 -1  2  0
511 2017-09-25 13:06:00 1 0 -1  1 -1 -1  1 -2  1
512 2017-09-25 13:07:00 1 0 -2 -2 -1  3  1 -2  1
513 2017-09-25 13:08:00 1 0 -2 -2 -1  1 -1 -2 -1
514 2017-09-25 13:09:00 1 0  1 -2 -1  1  0 -2 -1
515 2017-09-25 13:10:00 1 0  1 -2 -1  1 -2 -2 -1

(the exact number of already connected devices at the first measurement is not known, so the counts are all relative). The data set spans a period of a week and contains 9803 observations (at the minute level). The data set was transformed from a asynchronous event log to a synchronous time series format. I plan on adding lagged variables to this data set if needed (based on partial auto- and cross-correlation statistics).

I want to try to model the counts using neural networks, but have a few questions:

Do neural networks play nice with non-negative, discrete data?
Are certain network topologies preferred over others in a case like this? E.g. LSTMs?
Is it better to model a multivariate outcome (i.e. a vector a counts) or is it better to fit a model for each access point?


texts: Learning resources for data science to win political campaigns?

texts: Does anyone know, where I can learn about applying data science to win a political campaign? I know the Obama campaign had 12 data scientists in 2008 and 165 data scientists in 2012. In 2012, they ran over 65,000 simulations every night, for 14 months. They correctly predicted every state within 0.5% and Florida within 0.05%. How did they do this? And where can I find the data they used?


texts: Efficient algorithm to compute the ROC curve for a classifier consisting of an ensemble of disjoint classifiers

texts:
Suppose I have classifiers C_1 ... C_n that are disjoint in the sense that no two will return true on the same input (e.g. the nodes in a decision tree).  I want to build a new classifier that is the union of some subset of these (e.g. I want to decide on which leaves of a decision tree to give a positive classification).  Of course, in doing so there will be a trade off between sensitivity and positive predictive value.  So I would like to see a ROC curve.  In principle I could do this by enumerating all subsets of the classifiers and computing the resulting sensitivity and PPV.  However, this is prohibitively expensive if n is more than 30 or so.  On the other hand, there are almost certainly some combinations that are not Pareto optimal, so there might be some branch and bound strategy, or something, that avoids most of the computation in many cases.

I would like advice about whether this approach is likely to be fruitful and whether there is any work or if you have any ideas about efficiently computing the ROC curve in the situation above.


texts: Keras model.predict giving different shape from training label array

texts:
I'm using the following code to try and learn tensorflow. I've clearly specified the shapes of the training and validation X and y arrays.
import numpy as np
import tensorflow as tf

f = lambda x: 2*x
Xtrain = np.random.rand(400,1)
ytrain = f(Xtrain)
Xval = np.random.rand(200,1)
yval = f(Xval)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu')
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.MeanSquaredError()
             )

model.fit(Xtrain, ytrain, epochs=50, verbose=0)

When I run yval.shape, model.predict(Xval).shape, I get the output ((200, 1), (200, 10)). I'm not able to understand where these extra 9 dimensions are coming from. Even the Xval prediction should have the shape (200, 1).


texts: How Autograph is able remove unnecessary operations?

texts: Maybe my question is more related to computer science or graph theory, but when I started to read Tensorflow's documentation, I've learned that Autograph is able to avoid unnecessary operations to be made. However, why is this not be possible to do in Eager mode? For instance, isn't constant folding something that can normally done by compilers? Why expressing operations in a graph could help on identifying unecessary operations?


texts:
I have to make prediction on streaming data for which I have trained the model offline.

Is it a common practice to use scaling when we have to train the model offline and have to predict the on streaming data, also how to scale the streaming data?


texts:
I'm making a model that predicts delays in flight takeoff times using the day of the week and month. Each data entry is of the form [0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0], with the first 12 digits being for the month and the last 7 for the day of the week. The labels are integers, either positive or negative, that represent the number of hours delayed.

This is my code:

#
#
# This model uses day and month as a predictor of delay.
# https://www.kaggle.com/usdot/flight-delays#flights.csv
#
#

# TensorFlow
import tensorflow as tf
from tensorflow import keras
from keras.models import load_model
from keras.models import Sequential
from keras.layers import Dense, Activation
from sklearn.preprocessing import minmax_scale

# Helper libraries
import numpy as np
import csv

# Get features and labels
train_features = np.load("train_features.npy")[:10000, :]
train_labels = np.load("train_labels.npy")[:10000]
print(train_labels)
test_features = np.load("test_features.npy")[:1000, :]
test_labels = np.load("test_labels.npy")[:1000]

# Risk Assessment Model
model = Sequential()
model.add(Dense(6, activation="relu", input_dim=19))
model.add(Dense(1, activation="sigmoid"))

model.compile(optimizer="rmsprop", loss="mean_squared_logarithmic_error", metrics=["accuracy"])

model.fit(
    train_features,
    train_labels,
    epochs=100,
    verbose=1,
    validation_data=(test_features, test_labels),
)

model.save("time_model.h5")

The model has terrible accuracy, and I'm having trouble understanding why. I don't know what kind of activation functions I should be using, and I'm feeling completely lost. Thanks in advance for your help.


texts: Preprocess image data to classify objects based on shape

texts:
Currently I'm trying to build a neural network that is able to classify different types of bottles on an image solely based on the shape. The bottles have no label and at first I only used beer and wine bottles.
My goal is to compare a neural network that uses feature extraction to one that uses the raw images.

Example input image:

https://i.gyazo.com/c22151a4bee6146cc689fcb8358483dd.jpg

Using openCV to process the images, I tried to obtain only the edges of the bottles and classify the types based on the shape.
Currently, I am receiving the following result for the images:

https://i.gyazo.com/8554b06ebf6e8bd520d11222f70c77c5.png

In the next step, I cropped out each bottle in the image and split them into a 2d array of tiles (normally I try with 30x50 but in this image I show it with 10x20)

https://i.gyazo.com/28fb1bc0fa8d61b185ed2245db222cc8.png

For every tile I execute PCA to extract the eigenvectors. In the end I receive a matrix of 10x20 tiles, where each of them contains the direction of the line.
As you can see, the (0.0, 1.0) tuples are the vertical lines, and the (1.0, 0.0) tuples are the horizontal lines.

https://i.gyazo.com/bcd291535ea01b6392ab6aef6747de9d.png

  Sorry for the long post, here is my question:

Is it possible to use this data do classify between wine and beer bottles? The shape of the neck of the bottle is different but is it representative for building a neural network?
I tried to train a model by calculating the gradient (arctan from the tuple) but i didn't manage to get some successful results. Does anyone have a idea how to pre-process the data for a neural network?

Is it better to use the cropped out images of the bottles as image data for the neural network?

I am thankful for any help as this is my first machine learning project and I would really like to get it working.


texts: In doc2vec, how to model correctly when many documents share the same label?

texts:
Of all the examples I've found for doc2vec training, the documents are uniquely labeled. What happens when many documents share the same label?

TaggedDocument of gensim accepts a list labels for the same text. It implies we can have multiple labels for the same text. However, it's not clear to me if it's a good practice to have fragmented texts under the same label. You can still train and get the embeddings. But are they good?

For example, the question I am posting here has a title, a detailed description, and a list of tags. How do I model it for doc2vec to find similar questions?

Note that some of the tags are not in the title nor the description. What's the best way to include them in the doc2vec trainings. Shuffle them and concatenate with title and description? Or keep them as separate entries under the same label?


texts:
Orange has a hyperspectral dataset called "liver cirrhosis" and you can visualize the hyperspectral image using the hyperspectral image widget. However, I would like to perform various clustering methods e.g. k-means on the spectra at each pixel and then display those clusters on the hyperspectral image widget like in this photo

does anyone know how to do this kind of analysis?


texts: Decomposing R^2 into independent variables

texts:
Consider a linear regression model:
$$y = β_0 + β_1X_1 + β_2X_2 + ... + β_kX_k + ε$$
where $R^2 = 1 - (SSR/SST)$.

I would like to determine the contribution of a factor $i$ (call it $R^2_i$) into the total $R^2$ such that $R^2_1$ + $R^2_2$ + ... +$R^2_k$ = $R^2$, because I want to know the impact of each factor into the total variance.

One methodology, called Shapley-Owen Decomposition, exists that performs that. The problem is that it is very computational  heavy because it takes $2^k$ number of computations for each factor. It can be done pretty fast on Python when $k$ is small, but when $k$ is large, it is impossible.

My personal application is to decompose the $R^2$ into factors when $k$ is like 15-20 AND I want to do it using rolling-basis, because I want to see how the contributions change over time (which naturally means more computations).

I am wondering what efficient methodologies are out there to achieve my goal. A reference to academic paper/application would be appreciated. Thank you very much.


texts: Check the Reliability of Questionnaire using spss

texts:
I need to check reliability but i have category variable (nominal/ordinal) .
I choose analyze > scale> put all variable .
my question:
can i use this method for category variables and i have result as follow
Case Processing Summary
N
Cases   Valid   6
Excluded 96
Total  102
is there is wrong for the result , why excluded is 96 ?
Reliability Statistics
Cronbach's Alpha    N of Items
.840                   36
also, I have 37 variables , one of them is string. that is the reason if n item 36.

I analyze questionnaire and I need slandered example to make my output of analyzing is professional .


texts: Trainable parameters explode in CNN , how to control them and yet get desired accuracy

texts:
I am building a CNN mode using CSV of size (79999, 11)
Here is my model below. When I do a model summary it gives me close to two billion trainable parameters , which my laptop or even AWS Ec2 is not able to handle. What am I doing wrong ? New to ML pls help. I want trainable parameters to be something, which my 4 core 12 GB i5 can handle. Right now my Kernel just hangs.
Thanks --
Code :

seed=42;tf.reset_default_graph(); tf.set_random_seed(seed); np.random.seed(seed)

model = Sequential()
K.set_image_dim_ordering('th')  # th = (channels, rows, cols) ; tf = (rows, cols, channels)
#model.add(Convolution2D(30, 5, 5, border_mode= 'valid' ,input_shape = (1, 14, 5), activation= 'relu' ))
#model.add(Convolution2D(30, 5, 5, border_mode= 'valid' ,input_shape = image_shape, activation= 'relu' ))
#model.add(Conv2D(30, (5, 5), border_mode= 'valid' ,input_shape = image_shape, activation= 'relu', padding="valid" ))
#xxmodel.add(Conv2D(30, (5, 5), input_shape = (num_of_images, num_colors_per_pixel, rows_per_image, cols_per_image), activation= 'relu', padding="valid" ))
model.add(Conv2D(15, (5, 5), input_shape = image_shape, activation= 'relu', padding="valid" ))
model.add(MaxPooling2D(pool_size=(1, 1)))
#model.add(Convolution2D(20, 3, 3, activation= 'relu' ))
model.add(Conv2D(10, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(1, 1)))
model.add(Dropout(0.2))
model.add(Flatten())

model.add(Dense(8, activation= 'relu' ))
#model.add(Dense(25, activation= 'relu' ))
model.add(Dense(1, activation= 'softmax' ))

model.compile(loss= 'binary_crossentropy' , optimizer= 'adam' ,
              metrics=[ 'accuracy' ])
print ('model.input_shape  ', model.input_shape)
model.summary()
Result :
model.input_shape   (None, 1, 500, 559932)
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 15, 496, 559928)   390
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 15, 496, 559928)   0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 10, 494, 559926)   1360
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 10, 494, 559926)   0
_________________________________________________________________
dropout_1 (Dropout)          (None, 10, 494, 559926)   0
_________________________________________________________________
flatten_1 (Flatten)          (None, 2766034440)        0
_________________________________________________________________
dense_1 (Dense)              (None, 8)                 2212827552
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 9
=================================================================
Total params: 22,128,277,287
Trainable params: 22,128,277,287
Non-trainable params: 0


texts: Build train data set for natural language text classification?

texts:
I have extracted ~550 video scripts (subtitles) from 11 free courses on the Coursera platform. I have pre-processed them in terms of punctuation removal, stop words removal, tokenization, stemming and lemmatization. Now, I've been advised that for my task I can attempt to use a simple Bag of Words. However I am not sure how exactly would that help me towards classifying my text into one out of six categories. The categories are related to the intent a video material was created with and more precisely, which part explains a concept, which part discusses an example, which part gives practical advice etc. Below are my categories:

ConceptDescription-> Explanation of the Main concept(s)
ConceptMention-> Mentioning of a concept, related to the main concept
Methodology / Technique-> To achieve something, what should one do
Summary-> Summary of the discussed material or of the whole course
Application-> Practical advise for the concept
Example-> Concept example

By manually reading several files from 3 of the courses, I created a dictionary, containing spoken language words, that may help me identify which class a specific sentence/paragraph falls into. However I do NOT have a train dataset for a classifier. So my idea was to use that dictionary to label my data, e.g. sentence 1 as Summary, sentence 4 as ConceptDescription and sentence 12 as Example and then marking sentences 2 and 3 the same as 1, sentence 5-11 like sentence 4 etc.

My question is, is this idea too lame? And is there a way to create at least an average quality training dataset in a way that is not manual? Or if manual check is the only option, is there an option where I would need to do manual labeling on only a small fraction of the files, say 50 out of 550 and classification would still produce average to good result? I don't aim at perfect result, but I aim at something less time-consuming due to limited time.

I also played with tf-idf which outputs terms, but of course, not really what I need, so that was a bit random.

Thanks in advance for your help. Any specific ideas and algorithms would be very welcome.


texts: How to find coreset of a given dataset in python?

texts:
I am trying to implement the core-means algorithm, which is basically k-means using coreset. I have searched up and down but could not find any libraries or modules which could help me with this.

The paper I am following talks about building a coreset using grids, something like a quadtree where you keep dividing the point space in 4 equal parts to find heavy points which makes a coreset.

Any help would be greatly appreciated in getting me started as to how should I go about implementing this and if there are any libraries/modules available to help me do so.

PS: I'm new to python and machine learning and Hi StackExchange!


texts: How to Improve Low Accuracy Keras Model Design?

texts:
I am trying to train a system that looks at some data points and predicts the quantity of surfers on a surf break. I have labeled the pattern for the past 2 months and I have 1500+ training examples with observations every 15 minutes during the day (excluding night time)

My data is shown below (*kooks = surfers):

I am using Keras and here's the code:

*I am removing the month feature from the input matrix before processing it. I also did the preprocessing.MinMaxScaler() drill.

model = Sequential()
model.add(Dense(64, input_dim=6, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='relu'))

model.compile(optimizer='adam',loss='mse', metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=10)
history = model.fit(X, y, validation_split=0.33, epochs=200, batch_size=15, verbose=0, callbacks=[early_stopping_monitor])

The results I am getting are extremely poor:

Test score: 0.015
Test accuracy: 0.12

I have tried multiple optimizers and multiple activation functions, but haven't landed at a satisfactory model yet.

I have a couple of suspicions:

The data is not really predictable, as the system is getting confused as some times many of the features are the same (see lines 0 and 1), but the expected output is completely different.
The design of the model is not well suited (I don't really know how to design the hidden layer dimensions)
The loss function, optimizer and/or activation functions on each layer (including the output layer) are not well suited.

Am I doing something really wrong or this is just the nature of the beast? Any thoughts / advices?


texts: How to visualize convolutional network focus on autoencoder?

texts:
I'm trying to create some cool (and at least somewhat meaningful) visualizations of my CNN that I am using to autoencode images, as I use the encoded middle layer to do similarity comparisons.
I.e. I want a view of the input image but with some sort of "heat map" or something overlayed that shows what the CNN is "focusing on", if that makes sense. Some sort of gradient magnitude perhaps, I saw this on a different CNN project, e.g. analogous to the examples on this site.
How could I go about doing this for my autoencoder? Network is coded in Python with Keras, and I've reproduced the code below:
img = Input(shape=(IMG_W, IMG_H, IMG_C), name='input_img_')
    network = Convolution2D(CONV_0, 3, 3, activation='elu', border_mode='same', name='conv_0_e')(img)
    network = MaxPooling2D((2, 2), border_mode='same')(network)
    network = Convolution2D(CONV_1, 3, 3, activation='elu', border_mode='same', name='conv_1_e')(network)
    network = MaxPooling2D((2, 2), border_mode='same')(network)
    network = Convolution2D(CONV_2, 3, 3, activation='elu', border_mode='same', name='conv_2_e')(network)
    network = MaxPooling2D((2, 2), border_mode='same')(network)
    network = Convolution2D(CONV_3, 3, 3, activation='elu', border_mode='same', name='conv_3_e')(network)
    encoded = MaxPooling2D((2, 2), border_mode='same', name='encoded')(network)

    network = Convolution2D(CONV_1, 3, 3, activation='elu', border_mode='same', name='conv_0_d')(encoded)
    network = UpSampling2D((2, 2))(network)
    network = Convolution2D(CONV_1, 3, 3, activation='elu', border_mode='same', name='conv_1_d')(network)
    network = UpSampling2D((2, 2))(network)
    network = Convolution2D(CONV_2, 3, 3, activation='elu', border_mode='same', name='conv_2_d')(network)
    network = UpSampling2D((2, 2))(network)
    network = Convolution2D(IMG_C, 3, 3, activation='elu', border_mode='same', name='conv_decoded')(network)
    decoded = UpSampling2D((2, 2))(network)

model_inputs = img
    model_outputs = decoded
    model = Model(input=model_inputs, output=model_outputs)

    model.compile(loss='binary_crossentropy', optimizer='adadelta')

This question is a little more open-ended, I don't think there's a "right answer" per se, but I'm very open to creative ideas of how I could visualize what the network is doing. Hate to put it this way, but I'm looking for something that I would show a layman and they'd be like "wow look at that AI go" lol. But seriously that's what I'm looking for here. Maybe a collaged image of feature maps? Let me know what you think, and I'd really appreciate any sample code!


texts: How to adjust/smooth a certain number using constants or rules

texts:
Hi, I am handling a dataset with a customer purchase history.
The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage.

There are two things I'm trying to figure out:

How to penalize the small purchase count when it comes to a coupon purchase proportion.

I added a field 'mod_cpn_ratio' because when i calculate the simple proportion: (cpn_ord_cnt/total_ord_cnt), it did not do a good job for the customers with the smaller number of purchases, eg. (1 coupon purchase / 2 entire purchases) = 0.50% vs (50 / 100) = 0.50%

So for the 'mod_cpn_ratio' I added a constant '10' to the denominator to penalize those who bought less. I wonder if this was a fair way to handle this problem, or if there are better ways to handle it(possibly not just constant but dynamic number)

Rank customers with total_ord_cnt, but give some extra credit for those who made more purchases without coupons.

When two customers have the same number of 'total_ord_cnt', I would like to rank higher the one purchased more without coupons.

Any advice would be appreciated.

Thanks!


texts: Deciding what type of model to use for predicting the bottom decile of student grades

texts:
I have a large dataset which includes 36 variables (in %iles) to describe a student, and then the output is the students grades as a %ile. I am trying to predict, using the 36 variables, whether a student will fail out, ie. be in the bottom 2%ile.

My first attempt was a neural network with binary classification, forcing the model to predict that the student is either in the bottom 2%ile, or not. Having read a few posts on FHarrell.com, I was thinking that a binary logistic regression that outputs the probability of being in the bottom percentiles is better?

Or alternatively, should I just do a non-binary linear regression to directly predict the students %ile? The reason I thought that this wouldn't be as useful for my purposes is because I don't care about making predictions between any of the other percentiles. For example, I don't care about distinguishing between an A and a B student, I am just looking to identify F students. Therefore I would be losing accuracy in the bottom 2%ile and gaining it in the other irrelevant %iles?

Finally, I think that the neural network makes sense given that there is interaction between the variables. But if there is another model that I should consider please let me know.


texts: I am trying to do a grid searching using the methodology that mentioned in this post. However, I found that XGBClassifier().fit() is using much more memory than xgboost.train. Does anyone know why? Is this related to sparse matrix?


texts:
I am trying to train an AlexNet image model on the RVL-CDIP Dataset. The dataset consists of 320,000 training images, 40,000 validation images, and 40,000 test images.
Since the dataset is huge I started training on 500 (per class) sample from the training set. The result is below:
we can see from the graph above that the validation loss started decreasing at a much slower rate around epoch 20 while training loss continued decreasing the same. This means our model started overfitting the data? I assume that this is probably because the data i have in the training set is not enough to get better results on the validation set? (validation data is also a 500 (per class) sample from the whole validation set)
is it a correct approach to train the model on a small sample (eg. 500 images per class), save the model, load the saved model weights and then train again with a larger sample (eg 1000 images)? My intution is that this way the model would have new data every new run that helps it to learn more about the validation set. And if this approach is correct, when training the model for the second time with a larger sample, should the training sample include images (some or all) that were trained in the first model?
You can find the full code with results here


texts: Suggestion for stacked modelling in machine learning

texts: I have built several models on the training dataset and i am not happy with the results and I wish to club them all together and generate a new model, so here is my idea as i already have the results for the existing models i would like to create a new dataset with the existing model results as separate features on top of the original feature dataset applying a clustering to filter some data in the original dataset and would like to train the model across all the same models and get the result, Would that be called as stacked modelling?