texts: Keras multi-label time-series classification considering time-series as an input image vector

texts:
I am trying to build a multi-class classifier using Keras. I am not quite sure I have implemented it correctly.
Data is like this

label time-series variables [0:25728}

index 0  1   2   3   4            25728
  0   1  2.5 3.2 1.6 1.05 ........ 2.54
  1   5  3.2 1.6 1.5 1.49 ........ 1.41
  2   1  2.3 3.2 1.5 1.52 ........ 2.11
  3   3  0.2 3.1 1.5 1.89 ........ 0.81
  4   8  1.2 1.1 0.2 1.19 ........ 3.71
  .   5  .    .   .   .   ........   .
  .   7  .    .   .   .   ........   .
1323  5  .    .   .   .   ........   .

Here is the code.
I split data by 68 % then reshaping 1D array to a 2D array.
as 384*67 = 25728
So forming an image of vector 384 by 67 for one label

def readucr(filename):
data = np.loadtxt(filename, delimiter=',')
Y = data[:, 0]
X = data[:, 1:]
return X, Y

x_train, a = readucr(path+'p2_TRAIN')
x_test, b = readucr(path+'p2_TEST')
df_train_y = pd.read_csv(path+'p2_TRAIN',header=None)
df_test_y = pd.read_csv(path+'p2_TEST',header=None)

x_train = x_train[:,0:25728]
x_test = x_test[:,0:25728]

scaler = MinMaxScaler(feature_range=(0, 1))
x_train = scaler.fit_transform(x_train)
x_test = scaler.fit_transform(x_test)

x_train =x_train.reshape(x_train.shape[0],384,67)
x_test =x_test.reshape(x_test.shape[0],384,67)

train_label_y = df_train_y[0].values
test_label_y = df_test_y[0].values
batch_size = min(x_train.shape[0] / 10, 10)

y_train = np_utils.to_categorical(train_label_y)
y_test = np_utils.to_categorical(test_label_y)

x_train = x_train.reshape(x_train.shape + (1,))
x_test = x_test.reshape(x_test.shape + (1,))

input_shape = x_train.shape[1:]
model = Sequential()

model.add(Conv2D(32, kernel_size=(3, 3), padding='same',
                 input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, kernel_size=(3, 3), padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, kernel_size=(3, 3), padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(8, kernel_size=(3, 3), padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(GlobalAveragePooling2D())
model.add(Dense(9, activation='softmax'))

optimizer = keras.optimizers.Adam()
model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

hist = model.fit(x_train, y_train, batch_size=batch_size, epochs=nb_epochs, verbose=1)
score = model.evaluate(x_test, Y_test)
print("Accuracy: %.2f%%" % (score[1] * 100))

It gives 96.16% accuracy but I don't believe it is true.
I want to predict the labels.

How can I predict labels?
What I am doing wrong?

Please help! Thank you.


texts: Questions on ensemble technique in machine learning

texts:
I am studying the ensemble machine learning and when I read some articles online, I encountered 2 questions.

1.

In this article, it mentions

  Instead, model 2 may have a better overall performance on all the data
  points, but it has worse performance on the very set of points where
  model 1 is better. The idea is to combine these two models where they
  perform the best. This is why creating out-of-sample predictions have
  a higher chance of capturing distinct regions where each model
  performs the best.

But I still cannot get the point, why not train all training data can avoid the problem?

2.

From this article, in the prediction section, it mentions

  Simply, for a given input data point, all we need to do is to pass it
  through the M base-learners and get M number of predictions, and send
  those M predictions through the meta-learner as inputs

But in the training process, we use k -fold train data to train M base-learner, so should I also train M base-learner based on all train data for the input to predict?


texts: Precision or Recall when dealing with critical cases?

texts:
I have to create an AI that classifies mutliple objects in order to accept them or not inside a machine.
The problem is that some objects could be really harmful to the machine if they get accepted.
Should I focus on an high Precision or an high Recall, in order to not accept this kind of objects?


texts: Deep model ensemble giving different results

texts: I am making an ensemble of deep models for solving a classification problem. The initial weights follow the default distribution of keras layers. Each time I run the model(train the 'n' models, make predictions and then get their mode), I get a different result. What could be the possible reasons(rather errors) for this? And how shall I handle them?


texts: Using Keras masking layer with 2D convolutions (Conv2D)

texts:
I'm trying to design a neural network including time dependent input with different lengths and I'm currently using a Masking layer. This network worked well with TensorFlow version 1.9.0 but after updating to version 1.11.0, I get the following error:

  Layer conv2d_1 does not support masking, but was passed an input_mask: Tensor("cnn1/Reshape_2:0", shape=(?, 81, 81), dtype=bool)

Any idea on how to solve this problem?

I'm using the following code:

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, \
    TimeDistributed, Dense, Masking, Activation, BatchNormalization

model= Sequential()
# first layer
model.add(TimeDistributed(Masking(0., input_shape=(81,81,3)),
                                  input_shape=(None,81,81,3), name='mask'))
# CNN layers
model.add(TimeDistributed(Conv2D(filters=10,
                                 kernel_size=5,
                                 strides=1,
                                 padding='same'),
                         name='cnn1'))

model.add(Activation('relu', name='relu1'))
model.add(BatchNormalization())
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))

# output layer
model.add(TimeDistributed(Dense(3, name='output')))
model.add(Activation('softmax'))

# compilation
model.compile(loss='categorical_crossentropy')


texts: Forecasting energy consumption with no historical data when there is a trend

texts:
I want to forecast new customers' energy consumption. Let's say I can construct a set of attributes to describe new and existing customers (e.g. size of business, type of business etc.) and I have the time series data on energy consumption of existing customers. However, I have no historical data on new customers' consumption.

If existing customers' consumption has seasonality but no trend, I could use the set of attributes to create a regression model to predict new customers' consumption based on existing customers' consumption. But what if existing customers' consumption has a trend? How do I know at which point of the timeseries I should start the forecasting?

Let's take the following example time series of an existing customer:

If I want to forecast new customers' consumption for one year, how do I know at which point of this time series to start the forecasting?

I do not have a dataset yet, so I cannot provide data examples. The answers could include an idea on what types of additional information I would need (if any) to solve this problem. Enrgy consumption is just an example, my question is theoretical and could be generalized to any other field. Also, names of python libraries to deal with this problem would be welcomed!

Thank you for the help.


texts: what could this mean if your "elbow curve" looks like this?

texts:
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.

Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?

I'm computing the silhouette distance using a 80-20 train test split.


texts: Designing a pretrained DNN for image similarity

texts:
I am pretty new to deep learning and really hope that you can help me.

I want to write a python program that lets me choose an area in a reference image. This subimage of variable size should then be used to search in a database of images. Then the parts of the images with the highest similarity to the reference sub image should be given.
However I have big problems with the sizing of the reference and database images. I tried to read into pretrained DNNs (like VGG19) and use the features of a last layer for similarity computation. But these DNNs seem to accept input arrays only in certain resolutions.
Should I then rescale the reference image? The database images will most likely be much larger than the reference. Should I then partition all the database images into smaller subsets? Or use a single shot algorithm like YOLO?

Since there are so many different algorithms I would be very thankful for every comment or idea.

Lasse


texts: sklearn cross_validate without test/train split

texts:
I'm running cross validation on my training data:

scoring = {'acc': 'accuracy',
           'prec_macro': 'precision_macro',
           'rec_macro': 'recall_macro',
           'f1_macro' : 'f1_macro'}
all_scores = cross_validate(clf, X_train, y_train, scoring=scoring, return_train_score=True)

This works fine but the output contains scores with keys "test_...". This suggests that the function is splitting the training data into train and test subsamples. This isn't what I want. I already have an independent test sample so I don't need to make a further split in this function.

Perhaps I'm just confused about the terms used by sklearn:

Case 1) The "test" scores are the result of running the metrics on the test set for only one cross-validation split(this is fine).

Case 2) or the "test" scores are the result of running the metrics on a completely independent test set(not what I want).

The "test" scores are returned as an array of three values in my case. This leads me to believe that the "test" scores are simply referring to the set of data that is independent from the training data for each cross validation split. And thus that test set is different for each cross validation split (this is fine and not a problem).

If the "test" scores are from a completely independent sample that is never used for training in this function, then it should only contain one value, right? But it could return three values if the metrics were run on this data for each independently trained cross validation model.

My gut says its the first case and that everything is fine. Can anyone clarify this for me?


texts: Mixed geospatial and categorical clustering

texts:
I'm working on a project that seeks to identify clusters in urban development based on location (in lat/lon) and a categorical variable (what the particular site is zoned for). Ideally, the analysis would identify clusters of sites that are 1) near each other and 2) zoned the same. Below is a sample of what my data looks like:

 lat       lon      zone
 33.22320 -112.6741 R-43
 33.45324 -113.0888 R-43
 33.71800 -112.3885 R-43
 33.45626 -111.9408 AG
 33.45746 -111.9313 R-6
 33.45747 -111.9309 R-6

I've seen methods that define distance on just lat/lons using great circle distances, but I haven't seen any mixed clustering of the sort I'm trying to implement. I'm fairly new to cluster analysis so any guidance on implementing something like this in R or Python would be greatly appreciated!


texts: How do I plot linear regression results if input and target have different sizes?

texts:
For a linear regression model that I conducted, I'd like to review the regression plot of results. But since I have an input of size 6 parameters and target (output therefore) of 4, I get error when I use this code : source code

pyplot.scatter(x_train, y_train)
pyplot.plot(np.sort(x_values, axis=0,prediction)
pyplot.show()


texts:
I am attempting to create a demand forecasting model in python to predict future sales of a particular category of product, using historical sales data.

We are a B2B company, which means that we often get large orders at random times in a year, and there are other periods of no orders.

When using data for the past 5 years, the pattern looks a bit odd. from 2014 to 2016 there is an upward trend, followed by a downward trend.

When I attempt a linear forecasting method, the training set seems to predict the test set fairly well based on the graph. However, the metrics show that the projected sales count is off significantly.

What is the reason for such a huge error rate?

I realize that perhaps the best model for time series such as this would be something like SARIMA. I tried attempting ARIMA models, but because of the nature of the business (B2B) and very few data points available, SARIMA spits out too many errors. (for example, while trying to smooth out data for trend and seasonality, there are lots of logarithmic transformations. This removes resulting Nans and -infs, which leaves the data with almost zero data points.)

For a B2B company with very limited data points on purchase of a particular product (out of 5 years/1825days, only about 400 days when sales occur, for example), is it possible to build a reliable demand forecast model?

Is it possible to use linear regression to forecast future sales without the model projections being significantly off?

In summary, I would like to know if:

it is realistic to attempt reliable demand forecasting for a B2B sales with limited sales info.
What would be the best approach to forecasting B2B sales of particular products?


texts: Association Rule Learning for Home Electricity or Water Data?

texts:
My university has a building on campus where electrical usage and water usage is monitored down to the second.  Many electrical loads and water usageis tracked and stored in a database.  In the water dataset, the column headers are the location, and the rows contain the gallons used (per second).  In the electrical dataset, the power consumption (in Watts) is tracked in the rows and the load type is the column header.

My general problem is as follows: - how could I use the data from this highly monitored building, to benefit the university and give them insights into buildings that aren't monitored like this one?

My data is of the time-series, so I am trying to think of certain clustering or classification algorithms that could help me extract actionable data from it, and perhaps even association rule learning to find relations between the water and electricity data?

Thanks for any insights!


texts: How to estimate machine learning training speed on different hardware

texts: I need to put together a proposal for buying a better computer for machine learning. Is there any good way to estimate the general training speeds of computer hardware? Basicly, I want to be able to say that if we purchase a certain computer it in trains x times faster than our current computers. Currently I have a i5 63000U 2.4ghz 2 core cpu, with no gpu. I know switching to a computer with a gpu will speed things up greatly. I just need a way to quantify the improvement to justify the investment. Any ideas will be greatly appreciated. Thanks in advance.


texts: How to visualize (make plot) of regression output against categorical input variable?

texts:
I am doing linear regression with multiple variables. In my data I have n = 143 features and m = 13000 training examples. Some of my features are continuous (ordinal) variables (area, year, number of rooms). But I also have categorical variables (district, color, type). For now I visualized some of my feautures against predicted price. For example here is the plot of area against predicted price:

Since area is continuous ordinal variable I had no troubles visualizing the data. But now I wanted to somehow visualize dependency of my categorical variables (such as district) on predicted price.
For categorical variables I used one-hot (dummy) encoding.
For example that kind of data:

turned to this format:

If I were using ordinal encoding for districts this way:

DistrictA - 1
DistrictB - 2
DistrictC - 3
DistrictD - 4
DistrictE - 5

I would plot this values against predicted price pretty easy by putting 1-5 to X axis and price to Y axis.

But I used dummy coding and now I do not know how can I show (visualize) dependency between price and categorical variable 'District' represented as series of zeros and ones.

How can I make a plot showing a regression line of districts against predicted price in case of using dummy coding?


texts: How to compute the maximum likelihood hypothesis?

texts:
The Bayes theorem states that:

\begin{equation}
P(h|D) = \frac{P(D|h)P(h)}{P(D)}
\end{equation}
where $D$ is the dataset and $h$ is an hypothesis from the hypothesis space $H$.   Now (I'm not sure so if I'm wrong please correct me) I can consider:

$P(h|D)$ = the probability $h$ has generated the dataset $D$. More specifically, for each $h$ we have a probability that it has generated the dataset $D$.
$P(D|h)$ = the probability that $D$ has been generated by $h$. More specifically, for each possible dataset $D$, a certain hypothesis $h$ (that we have) can have generated it.

And I can represent them visually, for example:

Now, if we know the prior probability $P(h)$ then we can compute the maximum a posteriori hypothesis with the following formula:

\begin{equation}
h_{MAP} = argmax_{h \in H} P(h|D) = argmax_{h \in H} \frac{P(D|h)P(h)}{P(D)}
\end{equation}

Otherwise, we can consider the maximum likelihood hypothesis:

\begin{equation}
h_{ML} = argmax_{h \in H} P(D|h)
\end{equation}

At this step I don't understand how I compute $h_{ML}$ because if I consider $P(D|h)$ represented as in the previous example in the cartesian space we have $D$ in the x-axis, so if I consider the $argmax P(D|h)$ I will find the best $D$ and not the best hypothesis $h$.

What am I doing wrong? Are probabilities $P(h|D)$ and $P(D|h)$ not well interpreted in the cartesian space?


texts:
I would like to know what would be the best approach to compare two texts and see the differences between them.

For example:

Sent_1=“This toolset is a set of macros for performing a number of modelling tasks.”

Sent_2=“This tool is a set of macros which help performing a certain number of tasks.”

I do not mind the context/meaning at the moment, but I would like to know what it would be the best approach to spot differences (looking at each word, antecedent and subsequent to it) and see how accurate it is.


texts: Customize loss function for Music Generation LSTM (?)

texts:
I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract the data out of every song. The data in every midi song are organized as a series of discrete event messages.

I plan to have my input of the LSTM for each time-step as a vectorized event of three variables:

x_t = [note, velocity, timestamp]

note: categorical variable, since it can take only one integer value in the range [0:127] which corresponds to a pitch.

velocity: I think it is categorical too, as it also takes one integer value in the range [0:127] which corresponds to the force with which the note is played.

timestamp: the continuous variable that corresponds to the number of seconds (mostly milliseconds) or ticks that have passed between the current and previous events.

My question is since I have a combination of categorical and continuous variables to predict:

Is it ok to one-hot-encode note and velocity and keep timestamp as is?
Should I further standardize timestamp?
Can I use a customized loss function (sum of cross-entropy loss for note/velocity plus mean squared error for timestamp), or it would mess up my model?

Please, I feel a bit lost and need your help/suggestions.

Thank you!!!


texts: How to come up with the splitting point in a decision tree?

texts:
I read https://www.researchgate.net/post/How_to_compute_impurity_using_Gini_Index

I understand why choosing smallest gini index, but how do I come up with different candidate splits in the first place? How does R come up with the splits? Take the iris data as an instance:

plot(iris$Sepal.Length, col=iris$Species, pch=20)

How do I determine where I want to draw the horizontal line to separate each class? I can draw infinitely amount of lines and then compare their gini index, but practically, it will not work.


texts:
Hi everyone I want to learn about training many-to-one Long Short Term Memory (LSTM) for classification. My datasets configurations are consists of 100 rows, 44 frames, 13 features. I want to feed that data into say 32 LSTM cell followed by dense with softmax function (4 class). I am confused about how each rows being feed into 32 LSTM cell .
Which statement below is the correct one?

If the time step is 44, does the first frame go through all 32 LSTM cell, followed by the second frame go through all 32 LSTM cell, until the 44th frame?
The first frame will be feed into the first LSTM cell, the second frame will be feed into the second LSTM cell, until the 32th frame feed into the 32th LSTM cell. If this is the correct statement, how about 12 frames left? Or the number of LSTM cell must be larger than the number of frames ?

Please explain which statement is correct and why. Thank you.


texts: Adjusting imbalance in classification problem reduce precision, accuracy but increase recall

texts:
I've learned that adjusting imbalanced data when training a CNN affects model performance which got me thinking "what about in ML?" so I've done some testing on my own, you can check it out here -> https://haneulkim.medium.com/handling-imbalanced-class-in-machine-learning-classifiers-1b5c528f427f
I've ended up concluding that as we balance our data using random oversampling our precision and accuracy decrease. I want to know why this is the case...
If it requires me to do more research, link to resource would be greatly appreciated. Thanks!


texts: Combining Neural Network with Reinforcement Learning in a Continuous Space

texts:
I'm trying to learn how to do reinforcement learning on my own and I am not sure how to implement a neural network for a specific problem.

The game goes on for roughly 1 million steps. At each step, I have 36 continuous features available (unknown rules), and 4 actions to choose from. Afterwards, the environment will tell me what the 36 features are for the next step, and what my score is. I've been creating and looking at the data, and it sometimes only becomes clear a move was good/bad a lot later (10 moves or so).

So there are 2 things involved.

1) I have to "learn" how, given an action, the world would change from pre_state to post_state, and

2) I have to learn to optimize the reward given the state and choosing from 4 actions.

So I was thinking to just record state_before, state_after, reward of a lot of random moves. I could perhaps just use 36 * 10 of the last moves as predictors. But then again, perhaps I should only be interested in the difference between states?

The problem lies in the fact that I've looked at Markov Decision Processes, but they assume a discrete search space (whereas here it is continuous).

Any help would be useful in trying to understand what layers should be involved to solve this, and perhaps what the most logical way would be to sample data.

I really hope people can point me in the right direction. I'm willing to use any neural networks framework in Python basically.


texts: How to determine the function is linear in linear regression problem?

texts:
I know that the first degree of the polynomial equation is considered as a linear function.

But, I found some things confusing in linear regression.

1. f(x)= w1 x1+ w2 x2 + W3 x3 --> linear function
2. f(x)= w1 x1+ w2 x2 + W3 x1 x3 --> is it linear? if not, then why?
3. f(x)= w1 x1+ w2 x2 + W3 W4 x3 --> is it linear? if not, then why?
4. f(x)= w1 x1+ w2 x2 + W3 x3 x3 --> it is not linear.

Please help.


texts:
I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input text that clearly does not fall into a predefined category, it will randomly assign it to a certain classification with a high probability score
For example:
X_train = df_train.text
X_test = df_test.text
y_train = df_train.label
y_test = df_test.label

boosting = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('boosting', GradientBoostingClassifier()),])
boosting = boosting.fit(df_train.text, df_train.label)

list of categories -> {'A': fruit, 'B': animal, 'C': car, 'D': person, 'E': dessert, 'F': place}
docs = ['this is not category']

boosting.predict_proba(docs).tolist()

Output:
[[0.0016872033185414193,
  0.9915417761339475,
  0.0016865302624752719,
  0.0016921961567993337,
  0.0016974399174602914,
  0.001694854210776211]]

You can see that the second category is receiving a .99 probability when it is clearly not fitting into any of the options. Regardless of what I put through it, could be "fdahsjfkasl" it will return the same probability score for that second category
The model works so well for text that could logically fit into a category (not only performing well on test data, but also on new/random text too), but i need a way to handle text that does not, so that it can be labeled "Not a category" or something like it.
Does anyone have any suggestions?


texts: What are "belief maps" and "affinity maps"?

texts: When evaluating Nvidia-Deep Object Pose Estimation, I came across these terms, belief maps and affinity maps. I haven't been able to find a satisfying answer online on what these terms mean.


texts: Trouble with accuracy of multiclass perceptron

texts:
I have built a multiclass perceptron, but it has low accuracy (around 80%). I think I'm missing something. One possibility is that I should add a bias, but I'm not sure how to incorporate that.

The task is, given 2 dimensions, predict the class, which is between 0 and 8.

I could use some pointers as to where this code is going wrong. We are not given classes for test data, but rather need to populate it and it is checked elsewhere.

def dot_product(weights_list, training_data):
    return [sum([a * b for a, b in zip(training_data, i)]) for i in weights_list]

def predict(row, weights_list, n_classes):
    # get dimensions * weight
    activations = dot_product(weights_list, row[:-1])
    # get index of argmax
    predicted_label = activations.index(max(activations))
    return predicted_label

def train_weights(train, n_epoch, n_classes):
    # prepopulate just weights
    weights_list = [[0.0 for x in range(len(train[0])-1)] for x in range(n_classes)]
    for epoch in range(n_epoch):
        for row in train:
            actual_class = int(row[-1])
            # argmax from predicting each class
            prediction = int(predict(row, weights_list, n_classes))
            # if incorrect:
            #   lower score of wrong answer by this row's values
            #   raise score of correct answer by this row's values
            if actual_class != prediction:
                weights_list[prediction] = [a - b for a, b in zip(weights_list[prediction], row[:-1])]
                weights_list[actual_class] = [a + b for a, b in zip(weights_list[actual_class], row[:-1])]

    return weights_list

train = [[83.0, -14.0, 6.0],
        [77.0, 15.0, 6.0],
        [93.0, 35.0, 3.0],
        [86.0, -8.0, 6.0],
        [-51.0, -79.0, 1.0],
        [62.0, -73.0, 1.0]]

test = [[36.0, 27.0, -1],
        [6.0, 99.0, -1],
        [-3.0, 16.0, -1],
        [-40.0, -61.0, -1],
        [70.0, 67.0, -1],
        [86.0, -14.0, -1],
        [-92.0, 67.0, -1]]

n_classes = 9
n_epoch = 10000
weights_list = train_weights(train, n_epoch, n_classes)

for row in test:
    prediction = predict(row, weights_list, n_classes)
    row[2] = prediction


texts: Is it possible to make tensorflow print out everything it see in a given image and not just the top five results?

texts:
I'm working through the python API tutorials for Tensorflow and I'm seeing the results that are normally displayed, but it's always giving me the top five results.

I'm trying to discern all possibilities within a certain list of basic attributes, like if I'm given a picture of a forest, I want to ask tensorflow if the picture contains oak trees, pine trees, bushes, rivers, etc. I don't need to know if the image is a picture of a forest. Is this possible?

I'm not saying give me results it hasn't been trained to see, I'm saying I'm going to train the model with different types of trees/bushes/etc and I want to know if the given image contains any of those attributes (or the probability it thinks of for a given attribute).


texts: Why does my char level Keras tokenizer add spaces when converting sequences to texts?

texts:
I create a tokenizer with
import tf
tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...)
tokenizer.fit_to_texts(...)

But when I convert sequences of tokens to texts, the result contains a space after each character (except for the last one):
test_text = 'this is a test'
seq = tokenizer.texts_to_sequences([test_text])
r = tokenizer.sequences_to_texts(seq)[0]
assert(r == ''.join([ c+' ' for c in test_text ])[:-1])

Is there a way to avoid this added spaces? Am I missing some configuration parameter?


texts:
Recently, I was able to train a simple classification algorithm (my first ML-Project) and I even got a pretty satisfying precision score.

Now I am looking for a way to inspect, which datapoints in my train_data have been falsely classified.
My basic idea was something like:

If y_train != y_pred Then:
     (get indices of y_train)
     (look up the data in my csv and try to find a pattern)

My main problem is, that the train_test_split function provides me with a y_test subset like this:

print(y_test):

    28886    0
    23319    0
    8913     1
    25770    0

and y_pred is a list like this:

print(y_pred):

    [0 0 1 ... 0 1 0]

Since there already is an existing index in y_test, I can't just compare y_test[2] with y_pred[2]. It seems to me that, y_test[2] does not provide the third element of y_test. Rather it provides the third element of my original dataset.

I am looking for a way to compare position n of my y_test subset with position n of y_pred, so I can get the index of all false classified.

The Python code I used to get this result:

x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(X,Y, test_size=0.2)

clf = KNeighborsClassifier(n_neighbors=13)
clf.fit(x_train,y_train)

y_pred = clf.predict(x_test)
acc = metrics.accuracy_score(y_test,y_pred)
print(acc)


texts: Plotting multiple precision-recall curves in one plot

texts:
I have an imbalanced dataset and I was reading this article which looks into SMOTE and RUS to address the imbalance. So I have defined the following 3 models:
    # AdaBoost
    ada = AdaBoostClassifier(n_estimators=100, random_state=42)
    ada.fit(X_train,y_train)
    y_pred_baseline = ada.predict(X_test)

    # SMOTE
    sm = SMOTE(random_state=42)
    X_train_sm, y_train_sm = sm.fit_sample(X_train, y_train)
    ada_sm = AdaBoostClassifier(n_estimators=100, random_state=42)
    ada_sm.fit(X_train_sm,y_train_sm)
    y_pred_sm = ada_sm.predict(X_test)

    #RUS
    rus = RandomUnderSampler(random_state=42)
    X_train_rus, y_train_rus = rus.fit_resample(X, y)
    ada_rus = AdaBoostClassifier(n_estimators=100, random_state=42)
    ada_rus.fit(X_train_rus,y_train_rus)
    y_pred_rus = ada_rus.predict(X_test)

I then plotted the precision-recall curve for these 3 models. I chose this curve as I want to visualise how the models are performing, and I am not very interested in true negatives (the negative class is the majority class).
To plot the curve, I used ScikitLearn's plot_precision_recall_curve method, like so:

    from sklearn.metrics import precision_recall_curve
    from sklearn.metrics import plot_precision_recall_curve
    import matplotlib.pyplot as plt

    disp = plot_precision_recall_curve(ada, X_test, y_test)
    disp.ax_.set_title('Precision-Recall curve')

    disp = plot_precision_recall_curve(ada_sm, X_test, y_test)
    disp.ax_.set_title('Precision-Recall curve')

    disp = plot_precision_recall_curve(ada_rus, X_test, y_test)
    disp.ax_.set_title('Precision-Recall curve')

This resulted in 3 separate plots.

However, I want to have these 3 curves in one plot such that they can be easily compared. So I want a plot like the one in the article:

But I am not sure how to do this as the plot_precision_recall_curve method only takes one classifier as input.
Some help would be appreciated.


texts: What if we input sequence data to feedforward network?

texts:
One main advantage of RNN is the ability to take input of variable length like the case of sequences.
However, what if we neglected this case and assumed some applications that may accept some fixed window of input. For example, the weather prediction. We may input the temperature of the last 300 days for example to allow the network to predict the new temperature. We may fix this window of 300 all the time so that we have a fixed-length input.
My question now is what if we neglect the RNN in this case and input this window of 300 days into a feedforward network with 300 input dimensions and one output that is doing regression for it?
What will be the output as compared to RNN? In other words, why I should use RNN in this case?


texts: Is anyone here aware of any research study or industrial application where ML algorithms have been employed to reduce Adverse Selection and Moral Hazard risk in Insurance market?


texts: How to change the threshold value in orange?

texts: How can I change the threshold value for each classifier? In the ROC analysis curve we are allowed to remove the tick from "default threshold"


texts: Difference between AWS amazonei_tensorflow and tensorflow conda environments

texts:
In AWS SageMaker there are several Conda environments available based on different ML frameworks:

conda info -e

I would like to know the difference between a tensorflow and an amazonei_tensorflow environment.


texts: I am dealing with data which has only categorical features, whose values are just 0 or 1 and it is imbalance data. So, is it good to try smote because it will try to generate new data points. but, how can it generate new points if I have only 0 or 1? won't there be overlapping of points with majority class? because I tried it and used random forest but the f1score is below 0.5. So, I want to know is it good to try with smote and apply other models or not for this dataset ??


texts: Initial value space for Random Forest hyperparameter tuning

texts:
I'm building a Random Forest Classifier using Scikit Learn.
My problem consists in a 4 class classification task, the values are distributed as follows (after splitting my data in training set and test set with a proportion of 80%-20%):
y_train values
cautious_turn       386  # label and number of elements
aggressive_brake    356
cautious_brake      245
aggressive_turn     204

y_test values
cautious_turn       104
aggressive_brake     90
aggressive_turn      53
cautious_brake       51

The full dataset consists in 1489 samples. The training set is composed by 1191 samples.
I'm trying to optimize my random forest hyperparameters, using RandomizedSearchCV from sklearn.
My code is the following (just an example):
from sklearn.model_selection import RandomizedSearchCV
import numpy as np
from pprint import pprint

# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 1, stop = 150, num = 15)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}

So far, my code works perfectly and I have no problem.
My question is:
is there any way/empirical approach to decide which could be a possible good initial space for my hyperparameters values?
Right now I just copied those values from a tutorial. Is there any way to decide which could be (for example) a good range of values for min_samples_split looking at my data? Is there any methodology that allows me to reduce the "exploratory" space?
For example : I decided to search min_samples_leaf = [1, 2, 4] instead of min_samples_leaf = [10, 15, 20] because.... (possible motivation here)


texts: Classification report returns same accuracy precision recall averages at binary classification problem

texts:
My results are:

This is the Code:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report

df = pd.read_csv(dataset)
X = df.drop(columns=['backers_count','converted_pledged_amount','pledged'
      ,'id','usd_pledged','state','static_usd_rate','funded_percentage'])
y = df['state'].values
# Split dataset into train and test data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 42,stratify=y)

# Create Logistic Regression Classifier
logreg = LogisticRegression(solver='lbfgs',max_iter=400)
logreg.fit(X_train, y_train)
log_predict = logreg.predict(X_test)

#train model with cv of 10
cv_scores = cross_val_score(logreg, X, y, cv=10)

# Print the results
print("=== Confusion Matrix ===")
print(confusion_matrix(y_test, log_predict))
print('\n')
print("=== Classification Report ===")
print(classification_report(y_test, log_predict))
print('\n')
print("=== All Accuracy Scores ===")
print(cv_scores)
print('\n')
print("=== Mean Accuracy Score ===")
print("Mean Accuracy Score - Logistic Regression: ", cv_scores.mean())

Is it normal to have identical values to micro avgs macro avgs and weighted for precision recall f1-score and to accuracy as well or there's something wrong with my model? This also happens(getting identical results to precision recall f1-score accuracy) at other classification models with the same dataset that I've tried like random forest and SVC. If its normal how do I interpret this, or is it just wrong code and try to fix this?

Which avg matters and should record for binary classification(micro macro or weighted) or all?


texts: Let's say we have a categorical variables with 5 different categories (levels). I train and get a good model based on this dataset using embedding layer with, say, 3 embedding size and with some numerical features. However, how embedding layer reacts when we have an unseen data with an additional category (level) of this particular categorical variable?


texts: Kohen Kappa Coefficient of Naive Bayes with 62% overall accuracy is better than Logistic Regression with 98% accuracy?

texts:
I have been trying to evaluate my models used on fire systems dataset with a huge imbalance in the dataset. Most models failed to predict any true positives correctly however naive Bayes managed to do that but with a very high rate of False Positive. I had run the experiments on both the confusion matrix and classification report for both can be seen below. The same dataset and train/test split was used with both of the datasets

 Naive Bayes Confusion Matrix and Classification Report

     [[TN=732 FP=448]
     [FN=2   TP=15]]

          precision    recall  f1-score   support

       0       1.00      0.62      0.76      1180
       1       0.03      0.88      0.06        17

accuracy                               0.62      1197
macro avg          0.51      0.75      0.41      1197
weighted avg       0.98      0.62      0.75      1197

Logistic Regression Confusion Matrix and Classification Report

     [[TN=1180 FP=0]
     [FN=17   TP=0]]

          precision    recall  f1-score   support

       0       0.99      1.00      0.99      1180
       1       0.00      0.00      0.00        17

accuracy                              0.98      1197
macro avg          0.49      0.50     0.50      1197
weighted avg       0.97      0.99     0.98      1197

However I got the Kohen Kappa Coefficient for these models and I am quite confused on how to interpret the values. Please find values below

Logistic Regression=0.0
Naive Bayes=0.03

These values indicate very slight agreement. But why is the value of Naive Bayes slightly better than Logistic regression ?


texts: Predicting what time will a John go to the gym tomorrow?

texts:
I have a data set which contains the below columns:

Name    Day      Start_Gym_Time   End_Gym_Time
John    Monday   16:00            17:30
John    Tuesday  14:00            14:42
John    Tuesday  15:55            17:40

I need some advice on how to predict when will John go to the gym next? I am planning to use scikit-learn for trying out various algorithms? Any help would be highly appreciated.


texts: How to perform weighted addition of multiple layers in Keras using subclassing?

texts:
I am trying subclassing to perform weighted addition of three layers in Keras. The code for the class is shown below:
class Wt_Add(keras.layers.Layer):
  def __init__(self, units=1, input_dim=1):
      super(Wt_Add, self).__init__()
      w_init = tf.initializers.GlorotUniform()
      self.w1 = tf.Variable(
        initial_value=w_init(shape=(input_dim,units),dtype="float32"),
        trainable=True,
    )
      self.w2 = tf.Variable(
        initial_value=w_init(shape=(input_dim, units), dtype="float32"),
        trainable=True,
    )

  def call(self, input1, input2, input3):
      sum1 = tf.multiply(input1,self.w1) + tf.multiply(input2, self.w2) + tf.multiply(input3,(1-self.w1-self.w2))
      return sum1

However, when I am using it in the code (shown below)
A = Input(shape=(128,128,3), name='inputA')
B = Input(shape=(128,128,3), name='inputB')
C = Input(shape=(128,128,1), name='inputL')

wt_add = Wt_Add()
ad1 = wt_add(A,B,C)

model = Model([A,B,C], ad1, name = 'model')
keras.utils.plot_model(model)

I get the following output:

However, the expected output should be something like this:

Can somebody help with this code? I am using Keras with Tensorflow 2.0.


texts: How to determine if a neural-network has a static computation graph?

texts:
I'm looking at implementing some neural networks from research papers. However, I'm concerned about which framework I should use, because I'm uncertain if the networks form static computation graphs (making them suitable for TensorFlow/Keras) or if they are dynamic (making them more suitable for PyTorch).

What are some rules of thumb to determine if a network requires a dynamic computation graph?


texts: Which are the features selection techniques depending on the combination on cat num columns in independent and dependent features?

texts:
I am very confused:
For what I understood I should:

Multicollinearity check with Pearson corr and possibly consider to drop multicolliner features
Then?
I am very confused feature selection should be done with respect to the target variable or not and secondly in all the combinations of stat tests possible based on independent and dependent features: num cols and num target,  num cols and cat target,  cat cols and num target,  cat cols and cat target.
Sorry if the question is quite confused, but it reflects my confusion on all the possible statistical tests.


texts: How to preprocess data in a data table? (e.g. change a value in each row for some feature)

texts:
I Have a general question - how to preprocess data in Orange. E.g. I want to do something with each row value of some feature. How can I do this visually? And how can I do this via Python Script?

So far, I have only some ideas how to do this via python script:

Here is my code:

minimum = min(list(map(lambda row: row[0], in_data))).value

from copy import deepcopy
out_data = deepcopy(in_data)

for row in out_data:
    row[0].value = row[0].value - minimum

But it fails, because seems like the Value object is immutable...
Is there any way to create my own Value object with new data, and then do something like:

out_data = list(map(lambda row: Value(row[0].variable, [row[0].value - minimum]) + row[1:], in_data))

I am also interested in a "Visual widget-based" alternative, if it exists...


texts: XGBoost outputs tend towards the extremes

texts:
I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature in an observation by a very small amount can make the probability output jump from 0.5 to 0.99.

I barely see outputs in the 0.6-0.8 range. In all cases, the probability is less than 0.99 or 1.

I am aware of post training calibration methods such as Platt Scaling and Logistic Correction, but I was wondering if there is anything I can tweak in the XGBoost training process.

I call XGBoost from different languages using FFI, so it would be nice if I can fix this issue without introducing other calibration libraries, e.g., changing eval metric from AUC to log loss.


texts: Training a regression algorithm with a variable number of features

texts:
I need to train a regression algorithm with multiple features and a single label (predicted value). The problem is that this algorithm has to be able to do on-line learning and the number of features it will receive will vary. Let me give a clear example:

The algorithm is trained on a dataset of shape:

[--------Features-----------------] [Label]
[-- context11 -- | -- context12 --] [label1]

Then, for the next training example, one of the contexts might be missing, so the training example might either be:

[--------Features-----------------] [Label]
[-- context21 -- ]                  [label2]

or

[--------Features-----------------] [Label]
                  [-- context22 --] [label2]

How can I deal with this situation? So far, I thought about two possibilities:

Replace the missing part of the features with zeros. I am not sure how this will affect the algorithm though.
Use a decision tree or random forest? Do these have a more natural way of dealing with a variable number of features?

Any other ideas?


texts: Non Deterministc Dimensionality reduction

texts: could you please suggest me a nondeterministic algorithm for dimensionality reduction except t-SNE.


texts: Industry Standard Process for Fraud/Outlier/Anomaly Detection

texts:
I want to write my thesis about Fraud Detection in ERP Databases. I'm looking for a Industry Standard Processs such as CRISP-DM for Data Mining Projects, in order to justify my approach in solving the issue of finding outliers/anomalies in the data set.
Is there any established Standard Process which would be more suitable for such an project?

Thanks.


texts: Model selection metric for validation phase in deep learning

texts:
I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This selection of parameters is done using the same loss function that we use for training, but applied to validation data instead of training data.

My question is, does the metric used in the validation phase really have to be the same loss function, or can it be some other metric, such as accuracy or average precision?


texts: Adjusting predicted values based on average offsets from past predictions

texts:
This question is more about inference and decision making based on statistical data, so I hope I'm in the right place. If not, let me know if I should migrate this question elsewhere.

I have a data set specifying the predicted shoe size of different people for various shoe models, and also the manually measured correct shoe size.

For instance person #1 was predicted to have a shoe size of 8 for shoe model A, and the actual size she wears is 8.5. Each record includes the person, shoe model, predicted size and actual size. I already calculated the delta between these two values for each record, then summarized the average offset and standard deviation for all shoe models. Here's an example of the offset data (n is the sample size from the average offset and standard deviation was calculated for each model):

+-------+----------------+-----+----+
| model | Average Offset | SD  | n  |
+-------+----------------+-----+----+
|     1 | 0.4            | 1.0 | 16 |
|     2 | -0.8           | 0.8 |  5 |
|     3 | 0.8            | 0.7 | 10 |
|     4 | 0.5            | 0.9 | 12 |
|     5 | 0.7            | 0.8 |  6 |
|     6 | 0.1            | 0.9 | 28 |
|     7 | 0.5            | 0.8 | 16 |
|     8 | 0.1            | 0.7 | 18 |
|     9 | 0.3            | 0.5 |  6 |
|    10 | 2.7            | 0.3 |  5 |
|    11 | -0.2           | 0.6 | 33 |
|    12 | 1.0            | 0.5 |  6 |
|    13 | 0.0            | 0.0 |  5 |
|    14 | -0.1           | 0.6 | 13 |
|    15 | 0.0            | 0.4 |  4 |
|    16 | -0.9           | 0.5 |  7 |
|    17 | 0.2            | 0.8 |  9 |
|    18 | -0.2           | 0.8 | 20 |
|    19 | -1.1           | 0.7 |  9 |
|    20 | -0.1           | 0.6 | 14 |
|    21 | -1.1           | 0.8 | 55 |
|    22 | -1.2           | 0.8 | 12 |
|    23 | -0.3           | 0.5 | 12 |
|    24 | -0.1           | 0.4 | 10 |
|    25 | 0.6            | 0.9 | 29 |
+-------+----------------+-----+----+

And as a chart:

If you're wondering why analyze this per shoe model at all, then the answer is that we know from shop experience that different shoe models have particular structures and materials that have of consistent effect on shoe size preference.

Now to my actual question:
My goal is to plug average offsets back into the prediction calculation to adjust it based on previously recorded offsets.

The question is - when does it make sense to use an offset value, and when that value is unusable.

I tend to disregard average offsets where the SD is close to the size of the average, but mathematically, plugging in such an average to adjust all predictions for this model is likely to do more good than harm. Or does it?

What would be a good approach to decide which of these averages is useful?

Suggestion for additional insights or analysis techniques are welcome.


texts: How to use LSTMs for predicting the value for a specific hour within a day, given past daily data?

texts:
I think the heading tells everything. Let's say you have the stock price.

So you have for every day the price at every hour.

No I want to predict the next hour given all prevoius hours at a new day. I don't know how to choose an architecture and even how to train such case when you have data for past days. My problem is also (if it is a problem?!) the different kind of times : hours vs days.

So, I think you have to train a lstm for a day with time step of 1 hour to predict the next hour. But how to incoporate all other days in training?


texts: what is the reason behind the bad outputs gained by RNN, LSTM when using GloVe pretrained model in text classification?

texts:
the problem is with the results gained for accuracy and f1  afer training our model via pretrained models such as GloVe.
when I apply CNN as a classifier, the result are good as follows:

acc: 0.9345 - val_loss: 0.1513

but when I apply RNN and LSTM as a classifier the results will be as follows:

24931/24931 [==============================] - 188s 8ms/step - loss: 7.9559 - acc: 6.0166e-04 - val_loss: 7.9904 - val_acc: 0.0000e+00
Epoch 2/4
24931/24931 [==============================] - 189s 8ms/step - loss: 7.9645 - acc: 0.0000e+00 - val_loss: 7.9904 - val_acc: 0.0000e+00

the above result is reached both via RNN and LSTM.

the problem is that I use the same data set and the same structure and the same GloVe but I reach acc: 0.9345 for CNN and gain acc: 0.0000e+00 for both LSTM and RNN.
It is worth noting that I have changed optimizer but still get the same result. the applied dataset contains 41,399 items, totaling 60.3 MB and also is binary.
any guidance will be appreciated as I am a beginner in working with GloVe.

I apply keras with tensorflow backend with python 3.5 in ubuntu.


texts: Measuring the bias of a machine learning model

texts:
How can we measure the bias of a machine learning model? Can we determine it by just calculating its performance estimates difference on the train data and test data? For example, if a model SVM predicts on train data is 0.53 and then on test data it is 0.60; can we say that the bias is just 7 points?

I have read an article which mentioned that "The bias of a model validation technique is often measured in terms of the difference between a performance estimate that is derived from a model validation technique and the model performance on unseen data."

Thanks


texts: Improve model performance on unseen data

texts:
NOTE: This question was first posted on a different SO forum but I received suggestions to move it here

This is a follow-up question to a question I had previously posted on this forum
We conducted an experiment on 100 subjects and obtained a dataset that was used to train a machine learning model that works almost perfectly (we get a 99.99% accuracy) using cross-validation but fails miserably when test to new unseen data (we get only 39.8% accuracy). In detail, here is how we are testing our model:

Conduct an experiment (100 subjects) and recording physiological data (skin conductance and heart rate)
For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
Reserve the whole data of one subject for model validation
Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
Use the reserved dataset (see step 3) to validate the model obtained in step 5 above
The model performs very well (99.9% accuracy) when we used cross-validation (step 5) but fails to generalize (only 40% accuracy) when tested on new data from the unseen subject (step 3).

After reading the comments I received in my previous question, I decided to revise how I validate the model. Thus, instead of removing all the sample of one subject (as I did in step 3), I only remove all but a few samples (just 0.1% of the sample, i.e around 100 samples in my case). Surprising the model performs significantly better and achieves a 90% accuracy. This is puzzling and I would like to ask a few questions:

(1)How can one explain that a few samples (just 100 samples) are responsible for such an increase in performance (from 40% to 90% accuracy)(I will need to publish my result in a reputable conference)?

(2) Is there any published work that discusses this type of phenomena?


texts: Replace this for loop with a pandas vector operation

texts:
I am running this for loop in python to create a list of differences of means of two dataframes, and then using that list to generate a random values under a normal distribution.
df_means = []

for _ in range(10000):
    df_sample = df_common.sample(1000, replace=True)
    d_mean = df_sample[df_sample.doggo == True].retweet_count.mean()
    nd_mean = df_sample[df_sample.doggo == False].retweet_count.mean()
    df_means.append(d_mean - nd_mean)

# under null hypothesis mean will be 0
nulls = np.random.normal(0,np.std(df_means), 10000)

However, given that it is a for loop, this is taking a long time to run. Is there a way I can replace the for loop with a vector operation to speed up code execution?


texts: Sentiment Analysis Naive Bayes vs Logistic Regression

texts:
I am doing some sentiment analysis on Twitter data, and I wanted to compare a Naive Bayes Classifier and a Logistic Regression classifier as to if their performance is affected by spell checking the data. Feature extractor is a simple bag of words, and the spelling correction algorithm is an edit distance python package.

Now for Logistic Regression, after spell checking, the accuracy drops by 1% and the False Positives and False negatives numbers are raised by around 20. I expected the same performance from Naive Bayes, thinking that it might have to do with the training step, where probably some words are mis-corrected and the testing phase is affected. But it actually got a better percentage by 2% and more True positives and True negatives (but also more False Negatives).

Is the nature of the algorithms(generative/ discriminative) the reason behind it? Or is it something I am missing, because I can't wrap my head around it at the moment. (they both use the same training and test data, the same pre-processing procedure and the same spelling correction algorithm).

Thanks for any help!!

a (total) noob.


texts: How to build Explanatory Graph for Convolutional Neural Network?

texts:
I m reading very interesting paper (https://arxiv.org/pdf/1812.07997.pdf) that aims to interpret convolutional neural network using graph. The general idea is when there are co-related parts in layers those are likely to be semantic parts

The idea is nicely shown on this picture:

Authors use top down learning to predict and create those graph nodes.
I looks like there are couple of things that I do not understand.

(1) Since to learn subgraph it is required to pass infered locations R_L+1 how is it possible to learn the top layer?

(2) What do those M patterns learnt with greedy algorithm represent, the are learnt in layer L but correspond to layer L+1?

(3) I would really like to see like step by step how are those nodes selected cause it looks like i do not understand how are parameters theta constructed, cause in difference section of the article it looks like they are predefined with some position value, however they are said to be learnt.

The algorithm outline is below (more details in paper link):


texts: What to use in setting up a Speech to Text engine in production?

texts:
So i have the task to study the feasability of setting up a Speech-To-Text engine in a production environnement, and i've been researching on this topic, so I tried Google's Speech-To-Text API and there is a quota on how many transcribtions you can use daily => That's a no-no. Second alternative, CMU Sphinx ( Open Source pretrained model for that purpose ), just tested it, and the output is messy for the LiveSpeech feature ( Perhaps i need to adjust some parameters first ). Third option, using the available deep learning tools to create my own SpeechRecognition engine ( I have tons of recorded conversations of people speaking French ).

Question : What should i use for my case? building an engine from scratch using neural networks seems complicated for a beginner in ML like me, using CMU Sphinx seems easy, but i have a feeling it's not. Any advice on what approach should i follow?


texts: Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

texts:
I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning.

Should I include the prediction matrix (ie. df containing columns of prediction results from the various base learners) or should I just include the original features?

I have tried both methods with just the 'n_estimators' tuned with F1 score as the metric for cross-validation. (learning rate =0.1)

Method 1: With pred matrix + original features:

n_estimators = 1 (this means only one tree is included in the model, is this abnormal? )
F1 Score (Train): 0.907975 (suggest overfitting)

Method 2: With original features only:

n_estimators = 1
F1 Score (Train): 0.39

I am getting rather different results for both methods, which makes sense as the feature importance plot for Method 1 shows that one of the first-level predictions is the most important.

I think that the first-level predictions by the base-learners should be included in the gridsearch. Any thoughts?


texts: Compare between similar and dissimilar couples of instances

texts:
I label couples of similar and dissimilar instances based on user behavior.
each instance has a lot of features.
I have few ways of labeling the couples.
I know want to evaluate which of the label methods produce the most homogeneous distribution in the groups or to tell if the two groups comes from the same distributions.

I am looking for a statistical measures mostly.
Any suggestions?


texts: Calculating the Number of Parameters of a 2D CNN Layer

texts:
How can I calculate the number of parameters for a 2D CNN layer?
I usually use the equation: $output \ width= ((W-F+2*P )/S)+1 = (x)$
The same answer will be valid for the output height considering that they have the same dimension. So the output dimension will be (x,x).
However, what if I want to calculate the number of parameters for a 2D CNN layer with filter size 5x5, input channel/depth = 3, and output channel or depth of 8?


texts: What is the impact of adding a layer in neural networks?

texts:
I was playing with hyper-parameters on https://playground.tensorflow.org/ using spiral dataset (classification). So , first I trained a network with 2 hidden layers and the final test and train loss were around 0.02. After that , I introduced an extra layer (keeping everything other than that same) and trained it . To my surprise , the test loss was around 0.43 and train loss was around 0.37. I expected the test loss to increase , as the model was overfitting , however the significant increase in train loss was kind of counter-intuitive to me.
I am a newbie in this field and from what I know , adding a layer increases the model complexity so increase in test loss was expected but I also expected the train loss to be approximately same (or even decrease) as the model would overfit on train data. But the results show that model not only failed to generalise but even failed to fit on the training data.
I know that there can be many factors behind this , but can anyone please explain what could be the possible reason ?


texts: Linear Regression in python with multiple outputs

texts:
I have a time series dataset which represented as following:

x=[
       [12.19047619,  18.28571429,   6.0952381 ] ,

       [ 80.98765432,  14.17283951,  11.13580247 ] ,

       [ 50.82644628,  16.26446281,   9.14876033 ] , .... ]

and to predicted -->
 Y  = [13.9,  18,   14.987]

How I can use LASSO and SVR linear regression models in python to predict Y (which represented as a vector as shown in the above example)


texts: sklearn CountVectorizer token_pattern -- skip token if pattern match

texts:
I apologize if this question is misplaced -- I'm not sure if this is more of a re question or a CountVectorizer question. I'm trying to exclude any would be token that has one or more numbers in it.

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> import pandas as pd
>>> docs = ['this is some text', '0000th', 'aaa more 0stuff0', 'blahblah923']
>>> vec = CountVectorizer()
>>> X = vec.fit_transform(docs)
>>> pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
   0000th  0stuff0  aaa  blahblah923  is  more  some  text  this
0       0        0    0            0   1     0     1     1     1
1       1        0    0            0   0     0     0     0     0
2       0        1    1            0   0     1     0     0     0
3       0        0    0            1   0     0     0     0     0

What I want instead is this:

   aaa  is  more  some  text  this
0    0   1     0     1     1     1
1    0   0     0     0     0     0
2    1   0     1     0     0     0
3    0   0     0     0     0     0

My thought was to use CountVectorizer's token_pattern argument to supply a regex string that will match anything except one or more numbers:

>>> vec = CountVectorizer(token_pattern=r'[^0-9]+')

but the result includes the surrounding text matched by the negated class:

   aaa more   blahblah  stuff  th  this is some text
0          0         0      0   0                  1
1          0         0      0   1                  0
2          1         0      1   0                  0
3          0         1      0   0                  0

Also, replacing the default pattern (?u)\b\w\w+\b obviously messes with the tokenizer's normal function which I want to preserver.

What I really want is to use the normal token_pattern, but apply a secondary screening of those tokens to only include those that have strictly letters in them. How can this be done?


texts:
I recently started to read articles about Docker.
To me, in data science, Docker is useful because:

1) You have a totally different environment, which protect you against libraries and dependencies problems.

2) If your application modify, for example, the database of your company, you want first to make sure that the code works fine, and won't have bad consequences on the database. Thus, you use Docker to test your code first.

my questions:

Am I right if I say that only the second reason is about sand boxing? The first reason has nothing to do with sand boxing, right?
Are there other reasons why Docker is useful in Data science??
I don't find a lot of interesting research papers about Docker for data science. Do you know some famous ones?


texts: bias and variance trade off related question

texts:
I am having difficulty to understand the expected squared errors formula in this website:

$y=f(x)+e$  true regression line

$\hat{y}=\hat{f}(x)$   your estimated regression line

$error(x)=\bigg(\mathbb{E}[\hat{f}(x)]-f(x)\bigg)^2+\mathbb{E}\bigg[(\hat{f}(x)-\mathbb{E}[\hat{f}(x)]\bigg]^2+var(e)
        =bias^2+variance+\text{irreducible error}$

my question is what is the best way to understand bias and variance from this mathematical equation?

my understanding of variance here is:

assume you collect a sample with 1000 records and you generate regression line based on this sample. You get a model: $\hat{f}(x)=1+2x$

so at a given x value, for example: $x=1$ then $\hat{f}(x)=1+2*1=3$

then you re-sample another 1000 records and generate regression line:

$\hat{f}(x)=2+3x$

when $x=1$, $\hat{f}(x)=5$

you repeat this process for N times (assume N is a large number and close to infinity)

then your $E[\hat{f}(x=1)]$ is $(3+5+\dots+N)/N$

assume your $E[\hat{f}(x=1)]$ is $5.5$

once you have calculated $\mathbb{E}[\hat{f}(x)]$ for a given value of x=1, you should be able to compute $\mathbb{E}\bigg[\hat{f}(x)-\mathbb{E}[\hat{f}(x)])^2\bigg]$

$((3-5.5)^2+(5-5.5)^2+...)/(n-1)$

my interpretation above is focusing on variance of $\hat{Y}(x)$ at a each given $x$-point(re sampling to get $\hat{y}(x)$)

but my instinct tells me I should calculate based on only one set of data (1000 records) and calculate $\mathbb{E}[\hat{Y}(x)]$ at different x point.

please let me know which one of interpretation is correct or if both are wrong, please explain using this formula how I should interpret the terms in this formula.


texts: Nominal categorical variable with two levels: Label Encoding or One Hot encoding?

texts: For a nominal categorical variable that has two levels, e.g. Gender (levels = Male,Female), is it feasible to use label encoding instead of One Hot encoding ? If it is, are there any implications of using one encoding method over the other, for such a categorical variable ?


texts: Time series data combined with multiple feature. Predicting difference from the mean

texts:
I'm trying to predict the % attendance of people to gym classes that have previously been booked. It is heavily dependent on the time of day and also a load of other features (is it raining, fraction of class that booked yesterday compared to a week ago, etc.). Random forest alone performs very poorly.
Instead, I tried to predict the difference from the mean for the hour of the day using random forest.  Then I just add that on to the mean to recover the desired estimate.
This again performs worse than just the mean itself.

Is this approach (predicting the difference from the mean) a bad idea?
I can't find people using similar methods which makes me think it isn't a good idea.

Is there a better algorithm suited to this task?


texts: Sentiment Analysis: Creating dictionary from dataset

texts:
I've collected around half a million of unmarked comments from a newspage. The newspage has anti-foreign background. Because of that, a relativley  high number of comments contain hate.

Do you have an idea how to start creating a dictionary out of the comments? The dictionary shall contain words, which are connected to hate. With the dictionary I want to perform a text analysis for hate, later.

Because there is no such dictionary for German language I tried some things. For instance, I checked how often words, which are connected to emotions, appear in the comments. But it turned up, that they seem to be contained as often as any other word. I also thought about a Bag-of-Words and some other things, but I don't know.

I'm doing the analysis with Python 3 and R.

Kind regards and thanks in advance!


texts: How to segregate resume layouts into different types?

texts:
I'm looking for any suggestions on how to segregate resume layout into different types.

How do one proceed with such a task? I mean resumes are usually available as pdf or docx format and when we parse text from documents we lose a lof of information regarding layout or metadata.

So how one could build a system to segregate resumes based on layouts.

It'll be really helpful if you have any suggestions.


texts: How to serialize/pickle a spacy ner model?

texts:
I have trained a custom SpaCy named entity recognition model. I saved the model to disk using:
nlp.to_disk()

which results in keeping the model in a folder. Is it possible to make the nlp object to a pickle file?


texts: Is there any machine learning algorithm that can solve this problem?

texts:
I have a data set of 100000 samples with binary output.

I would like to study the impact of Col_A (a continuous feature) on the output result.

Col_A has values from 0 to 7000000 and when I add this feature to my classifier it gives bad accuracy. I have tried z-score and scalar number but it doesn't change a thing.

I would like to try another method to solve this problem.

I made some plot visualization to my data and I found that there is some range in which the majority of the outputs are negatives. ie. when col_A is ranging between (0 - 200), between (2500 - 2800) and between (5200-5400) the majority of outputs are negatives.

I would like to create a new feature to specify which category my Col_A is in, instead of using Col_A.

PS: I can't use if--else rules because I don't know where to delimit these ranges, I have just analyzed the previous values from my plot. but I need to have a dynamic method for getting those categories.

Is there any type of clustering that can solve this problem?

Does neural network help for this case?


texts: Time series plot using netcdf data in python

texts: I am trying to plot a time series for a 3d plot. I know that the variable must be made 1d inorder to make it viable for the time series plot. Actually I want the data within a box of latitude and longitude. So if anyone can help me in this regard of how to plot it will be much appreciated.


texts: Original Meaning of "Intelligence" in "Business Intelligence"

texts:
What does the term "Intelligence" originally stand for in "Business Intelligence" ? Does it mean as used in "Artificial Intelligence" or as used in "Intelligence Agency" ?

In other words, does "Business Intelligence" mean: "Acting smart & intelligently in business" or "Gathering data and information about the business" ?

This question was the topic of a debate among some fellows in our data-science team, so I thought to ask about it from other experts. One might say that both meanings are applicable, but I'm asking for the original intended meaning of the word as proposed in the 1980s.

An acceptable answer should definitely cite original references, and personal opinions are not what I'm seeking.


texts: Understanding the Shuffle and Split Process in a Neural Network Code

texts:
I need to understand how the splitting of labels in the following code happened :

import keras
import librosa
import librosa.feature
import librosa.display
import glob
import numpy as np
from keras.models import Sequential
from keras.layers import Dense , Activation
from keras.utils.np_utils import to_categorical

def extract_features_song(f):
    y, _ = librosa.load(f)

    # get mfcc
    mfcc = librosa.feature.mfcc(y)

    # make value between 1 -1
    mfcc /= np.amax(np.absolute(mfcc))
    return np.ndarray.flatten(mfcc)[:25000]

def extrac_features_and_labels():
    all_features = []
    all_labels = []

    genres = ['blues' , 'classical', 'country' , 'disco' , 'hiphop', 'jazz', 'metal' , 'pop', 'reggae', 'rock']
    for genre in genres:

        sound_files = glob.glob('genres/'+genre+'/*.au')

        print ('prcoessing %d songs in %s genre'% (len(sound_files), genre))

        for f in sound_files:
            features =extract_features_song(f)
            all_features.append(features)
            all_labels.append(genre)

    # one hot encoding
    label_uniq_ids , label_row_ids = np.unique(all_labels, return_inverse= True)
    label_row_ids = label_row_ids.astype(np.int32, copy= False)
    onehot_labels = to_categorical(label_row_ids, len(label_uniq_ids))

    return np.stack(all_features), onehot_labels

features , labels = extrac_features_and_labels()

print (np.shape(features))
print (np.shape(labels))

training_split = 0.8

alldata = np.column_stack(features , labels)

np.random.shuffle(alldata)
splitidx = int(len(alldata))*training_split
train , test = alldata[:splitidx,:], alldata[splitidx:, :]

print (np.shape(train))
print (np.shape(test))

# the concerned part: begin
train_input = train [:,:-10]
train_labels = train [:,-10:]

test_input = test [:,:-10]
test_labels = test [:,-10:]
#the concerned part: end

print (np.shape(train_input))
print (np.shape(train_labels))

the output is as follows:

(1000, 25000)
(1000, 10)
(800, 25010)
(200, 25010)
(800, 25000)
(800, 10)

Now, when he - code instructor - stacked both arrays together and shuffled them, he can't be sure that the last ten elements are the labels, am I right? If so how did he do it like this, this would cause an error.


texts: Read csv file and save images from the output

texts:
I have created some code that reads my CSV file and converts the dataset to a grayscale image. I want to know if there is any possible way to read through each row in the dataset and save each of the images created from the rows?
So far, I have got this code that reads the CSV files and creates an image using .imshow
import pandas as pd
import numpy as np
from sklearn.datasets import load_digits
from keras.preprocessing.image import array_to_img
import matplotlib.pyplot as plt

data_path = "dataset_malwares.csv"

data = pd.read_csv(data_path);

label = data.Malware.values

data = data.drop("Malware", axis=1)
data = data.drop("Name", axis=1)
data = data.values
data = data.reshape(data.shape[0], data.shape[1], 1)
data = np.tile(data, (1, data.shape[1]))

plt.imshow(data[1], cmap="gray")

plt.title("label: {0:}".format(label[1]))
plt.show()

print(data[0].shape)

I want to go through the dataset and save each image but not too sure where to start. Any suggestions would be great. Thanks :)
Rows/format of the data - I've provided a shared version of the dataset:
https://1drv.ms/x/s!AqFNg8FC48SSgtZSObDmmGHs3utWog


texts: How to label time series data correctly for training RNN/CNN models?

texts:
My Case

I want to tackle a deep learning classification task using various smartphone sensor data. I will use a self-built data acquisition app and basically walk around with the phone, manually adding class labels to the data when necessary.

Each data instance has a timestamp and sensor values, and these instances are recorded every 1/50 sec. The labels are assigned to the corresponding timestamp of the moment I click on a button in the app's interface while recording.

The Goal

My goal is to detect ~2 sec moments in another application along with their corresponding class.

The Problem

Now I never touched time series data before and I am wondering how I should preprocess the labels before using it for a model. For now, I was thinking about having a 2sec sequence of data instances for each label (n= 50 * 2). However, I am not sure if this is the correct way of labeling in time series, as for other data types labels are assigned to a single instance.


texts: Is there a disadvantage to letting a model train for a large number of epochs?

texts:
I created a model to solve a time series forecasting problem.  I had a limited amount of time series with which I could train the model therefore I decided to augment the data.  The data augmentation strategy I used is quite basic but has shown to increase the accuracy of my model.

I wrote my own data_generator which I use to train my model with, using the fit_generator function in keras.  Essentially it takes in the whole training data set that I have, shuffles all the time series and the augmentation process takes place specifically in each batch.  In each batch I randomly pick, per time series in the batch, start and end points, so that each batch contains varying length slices of each series within the batch.  This creates obviously an almost endless stream of data but it is entirely reliant on the number of epochs the model is run for as the dataset is not augmented upfront.  No noise or anything is applied to the data set, the augmentation is purely from varying the lengths of the time series and the start and end points of the series.

I observe that my loss continues to decrease over time I have tried 100, 500, 1000, 5000 and 10,000 epochs.  In general the accuracy of the model predictions does get better but at some point with diminishing returns.  It is hard to say when as I am still tuning the model architecture and hyperparameters.

Does such an augmentation strategy affect how I can interpret the loss of model?  As the longer I train the model for the more "new" data it sees instead of constantly seeing the same data and training on it.


texts: keras multiple text features input and single text label output classification

texts:
Am struggling to find the exact way to preprocess the text data with multiple text columns on the dataframe of features input and single output text label. Below is the sample dataset,

name    |   big_text_phrase     |   action      | action_type
--------------------------------------------------------------
test_name_1 | some bag of words |   action_1    | action_type_1
test_name_2 | some different bag of words   |   action_2    | action_type_2

When I was trying to do the text classification using just one feature big_text_phrase as input and output label as name it works fine and able to predict. Below is the model details with the single text feature input.

import numpy as np
import pandas as pd
import tensorflow as tf

from sklearn.preprocessing import LabelBinarizer, LabelEncoder
from sklearn.metrics import confusion_matrix

from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.preprocessing import text, sequence
from keras import utils
train_size = int(len(data) * .8)
train_posts = data['big_text_phrase'][:train_size]
train_tags = data['name'][:train_size]
test_posts = data['big_text_phrase'][train_size:]
test_tags = data['name'][train_size:]
max_words = 10000
tokenize = text.Tokenizer(num_words=max_words, char_level=False)
tokenize.fit_on_texts(train_posts)
x_train = tokenize.texts_to_matrix(train_posts)
x_test = tokenize.texts_to_matrix(test_posts)
encoder = LabelEncoder()
encoder.fit(train_tags)
y_train = encoder.transform(train_tags)
y_test = encoder.transform(test_tags)
num_classes = np.max(y_train) + 1
y_train = utils.to_categorical(y_train, num_classes)
y_test = utils.to_categorical(y_test, num_classes)
batch_size = 32
model = Sequential()
model.add(Dense(512, input_shape=(max_words,)))
model.add(Activation('sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(x_train, y_train, batch_size=batch_size, epochs=5, verbose=1, validation_split=0.1)
score = model.evaluate(x_test, y_test, batch_size=batch_size, verbose=1)

But am in full of confusion as how to implement the same with multiple input text features and single output text label, Googling doesn't get much details and heared that one hot encoding may suitable. So I tried with one hot encding where its kind of categorical method. Since I have huge bag of words in big_text_phrase column and the categorical columns are growing like anything.

Update 1:

I tried something different (might be stupid attempt) as below where the samples counts are going wrong,

train_posts = data[['big_text_phrase', 'action', 'action_type']][:train_size]
train_tags = data['name'][:train_size]
test_posts = data[['big_text_phrase', 'action', 'action_type']][train_size:]
test_tags = data['name'][train_size:]

print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)
print('y_train shape:', y_train.shape)
print('y_test shape:', y_test.shape)
x_train shape: (3, 10000)
x_test shape: (3, 10000)
y_train shape: (3587, 47)
y_test shape: (897, 47)

The model returned the error Input arrays should have the same number of samples as target arrays. Found 3 input samples and 3587 target samples..


texts: I have a 2D matrix, and I want to learn a threshold from this matrix. All items in the matrix bigger than the threshold will become 1s, and all smaller than the threshold will become 0s. Then, I have a differentiable loss function to evaluate the resulting 0-1 matrix. What methods are typically used to learn the threshold? Do we put the matrix across a network, or are there any non-neural-network methods?


texts: Attributes extraction from unstructured product descriptions

texts:
I am trying to match new product description with the existing ones. Product description looks like this: ￼Panasonic DMC-FX07EB digital camera silver. These are steps to be performed:

Tokenize description and recognize attributes: Panasonic => Brand, DMC-FX07EB => Model, etc.
Get few candidates with similar features
Get the best candidate.

I am having problem with the first step (1). In order to get 'Panasonic => Brand', DMC-FX07EB => Model, silver => color, I need to have index where each token of the product description correspond to certain attribute name (Brand, model, color, etc.) in the existing database. The problem is that in my database product descriptions are presented as one atomic attribute e.g. 'description' (no separated product attributes).

Basically I don't have training data, so I am trying to build index of all product attributes so I can build training data. So far I have attributes from bestbuy.com and semantics3.com APIs, but both sources lack most of attributes or contain irrelevant ones. Any suggestions for better APIs to get product attributes? Better approach to do this?

P.S. For every product there is a matched product description in the Database, which is as well in a form of one atomic attribute. I have checked this question on SO, it helped me and it seems we have same approach but I am still trying to get training data.


texts:
How to make a stem and leaf plot with the following data in "R":

1717 1719 1645 3739 3024 3664 3830 2991 2430 2730 3469 5086 2119 3021 3292 2844 3426 2067 3215 2767 3124 2573 2840 2449 2584 1505 1390 1645 2497 3466 3228 3192

The only way to get a reasonable amount of stems is to make the stem the first digit in each number and the leaves will all be 3 digit numbers. I am not able to split the data in this way, R represents the stem with 1 digit but round my numbers  so 2584 is shown as 2| 5 (3 decimals to the right) which would be 2500. How do I adjust my stem and leaf numbers?


texts: Identify similar features from a set of features for a given feature in an unsupervised learning way

texts: Suppose I have a set of 1000 features. Now for a given feature, how to find the similar features from this 1000 features in an unsupervised learning way? In other words, no target/label dataset is provided.


texts:
How would you interpret the following ROC and PRC curves?
For example, I find it weird to understand that the precision actually increases at some point when recall increases as well. Is that even possible?

Cheers


texts: Feature selection and Model structure for object detection in TensorFlow

texts:
So I'm making a TensorFlow model to detect an enemy player on screen (if present) and return me its position. I'm using transfer learning with some fine tuning because it's much faster than training my own network. However, I'm unsure how to structure my data and my output layer. Currently my output data is 3 units with a relu activation function.

The labels look like this: [enemyx, enemyy, enemypresent] e.g [500.0, 200.0, 1.0] currently. If the enemy player is not present, then it looks like [0.0, 0.0, 0.0].

The training loss is very high (~24000). I am using the MobileNetV2 with imagenet weights, with an input size of 224 x 224 x 3. The x and y of the enemy are between 0 and 900 because I have resized 900 x 900 images down to 224 x 224 in order to use MobileNetV2.

I need help structuring my model to do this detection effectively (activation functions, data structuring, etc)

Here is the model code:

base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE, include_top=False, weights="imagenet")
base_model.trainable = False
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()(base_model.output)
prediction_layer = tf.keras.layers.Dense(units=3, activation="relu")(global_average_layer)
model = tf.keras.models.Model(inputs=base_model.input, outputs=prediction_layer)
model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.003), loss="mse", metrics=["accuracy"])
model.fit(X, y, epochs=30, batch_size=128, validation_split=0.1)


texts: How to learn from time series with multiple values for each time points


texts: Netflow anomaly detection python packages

texts: Is anyone aware of any open source / python packages for Netflow Anomaly detection  ? I found some on github but anyone who has more experience with it. please advise.


texts: Visualizing Decision Tree of K-Nearest-Neighbours classifier

texts: I'm using Sklearn's KNN to build a classifier and was wondering if there is any way to visualize the decision tree that the algorithm builds. Maybe something of this fashion


texts: Convnet training error does not decrease

texts:
I'm training a convoluted neural net to drive a toy car, and no matter what I do the training accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?

Notes:

Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records.
Three equally proportioned classes: turn left, go straight, turn right
Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6
Simple NN uses the sigmoid activation function, the convnet is all relu activations
Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs
My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion
Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%
I'm using initial = tf.truncated_normal(shape, stddev=0.1) to initialize all weights
I'm using initial = tf.constant(0.1, shape=shape) to initialize all biases
It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy

Convnet Notes (all yield same poor results):

Batch normalization
Max pooling
50% drop-out probability
Various stride sizes
Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons.
3D convolution (took too much memory and caused my GPU to crash)
1x1 convolutions
The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine

Code:

Simple neural net
Deep Convnet


texts: Rearange input data - unknown size of input

texts:
I have some data of form showed below to squezze through some neural network.

Customer Step Var0  ... VarN
0        0    some  ... stuff
0        1    again ... stuff
0        2    foo   ... bar
0        3    bla   ... blub
1        0    other ... stuff
1        1    and   ... ongoing
2        0    and   ... so on
.        .    .     ... .
.        .    .     ... .
.        .    .     ... .

As the neural network has a fixed input layer size, but the customers step count can differ at each customer, I have following question:

What is the best practise (if exists) to rearange the given data for using as input of a NN?


texts: How to deal with attributes that can vary arbitrarily for each sample?

texts:
Let's say we are trying to classify cars into five different categories. For this, we have a lot of samples described by color, brand, model, year of manufacture and so on. For instance, imagine something like this:

cars

+-----+-------+---------+-------------+--+---+---------------------+
| id  | color |  brand  |    model    |     | year of manufacture |
+-----+-------+---------+-------------+--+---+---------------------+
| ... | ...   | ...     | ...         | ... |                 ... |
| 319 | Black | Ferrari | Dino 246 GT | ... |                1967 |
| 320 | Gray  | Ferari  | 250 GTE     | ... |                1960 |
| 321 | Red   | Ford    | Mustang     | ... |                1969 |
| 322 | Black | Jaguar  | E-Type      | ... |                1961 |
| ... | ...   | ...     | ...         | ... |                 ... |
+-----+-------+---------+-------------+-----+---------------------+

Also, in this context, we have more attributes which we know that, somehow, should be relevant while classifing these cars. These attributes are in another database (or table) and they describe historical changes (the relevant ones) applied to the car. For simplicity, let's assume that each modification is associated with a code (but we also have another attributes related that may be relevant, like date of modification and estimated cost, for example).

cars_changes

+-----+--------+-------------+-----------------+------------+-------+
| id  | car_id | change_code |   description   |    date    | cost  |
+-----+--------+-------------+-----------------+------------+-------+
| ... | ...    | ...         | ...             | ...        | ...   |
| 17  | 319    | CLR-93AA    | New painting    | 2009-11-18 | 800   |
| 18  | 319    | ENG-77TS    | Change engine   | 2011-06-04 | 3,000 |
| 19  | 319    | GAS-19BV    | Add gas as fuel | 2016-02-23 | 1,739 |
| 17  | 319    | CLR-93AA    | New painting    | 2017-09-18 | 1,100 |
| 20  | 321    | CLR-92BD    | New painting    | 2012-03-17 | 930   |
| 21  | 321    | GAS-19BV    | Add gas fuel    | 2016-05-11 | 1,385 |
| ... | ...    | ...         | ...             | ...        | ...   |
+-----+--------+-------------+-----------------+------------+-------+

As you can perceive, observe that:
 - a car does not necessarily had changes;
 - there is not a maximum amount of changes that a car may have made;
 - a change (with same code) can occur many times in the same car;
 - consider that the possibility of different changes in this context are of high dimension categories (let's say about 2 thousands!)

Also, consider that there is a classifier based on the first table and our goal is to improve this task (of classifying), somehow, using the information from the second database. Considering that the most important information for this task are present on the second table, how we can use this information in a appropriate way for this problem?


texts: Keras - error when adding layers to loaded model

texts:
I want to use ResNet50 as a feature extractor. For this purpose, I have loaded the pre-trained model, deleted a few layers and added my layers to the model. For adding my layers, I have used the Sequential API. The code is the following:

resnet_model = ResNet50(weights='imagenet')

# Delete layers
for i in range(12):
  resnet_model.layers.pop()

# Fix weights
for layer in resnet_model.layers:
  layer.trainable = False

base_model = Model(inputs=resnet_model.inputs, outputs=resnet_model.layers[-1].output)

model = Sequential()
model.add(base_model)

model.add(Conv2D(2048, kernel_size=(3, 3), input_shape=(7, 7, 2048)))
model.add(BatchNormalization(momentum=bn_momentum))
model.add(Activation('relu'))

model.add(Conv2D(2048, kernel_size=(3, 3)))
model.add(BatchNormalization(momentum=bn_momentum))
model.add(Activation('relu'))

model.add(Flatten())
model.add(Dense(1000, activation='relu'))
model.add(Dense(200, activation='softmax'))

This code works. However, when I run print(model.summary()) I obtain the following:

Model: "sequential_26"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
model_16 (Model)             (None, 7, 7, 2048)        19115904
_________________________________________________________________
conv2d_31 (Conv2D)           (None, 5, 5, 2048)        37750784
_________________________________________________________________
batch_normalization_23 (Batc (None, 5, 5, 2048)        8192
_________________________________________________________________
activation_917 (Activation)  (None, 5, 5, 2048)        0
_________________________________________________________________
conv2d_32 (Conv2D)           (None, 3, 3, 2048)        37750784
_________________________________________________________________
batch_normalization_24 (Batc (None, 3, 3, 2048)        8192
_________________________________________________________________
activation_918 (Activation)  (None, 3, 3, 2048)        0
_________________________________________________________________
flatten_10 (Flatten)         (None, 18432)             0
_________________________________________________________________
dense_17 (Dense)             (None, 1000)              18433000
_________________________________________________________________
dense_18 (Dense)             (None, 200)               200200
=================================================================
Total params: 113,267,056
Trainable params: 94,142,960
Non-trainable params: 19,124,096
_________________________________________________________________
None

As you can see, the conv layer adds 37750784 new parameters to the network!!! Obviously, this isn't right, but I can't see why this happens...


texts: Statistically showing that a certain phenomenon present in one sample is a general phenomenon for the entire population

texts: I am working on an EEG signal classification problem. After some certain processes, I noticed that the Band power of eeg signals are decreasing in a few samples that I checked manually. Now I have the band powers of eeg signals before and after the above mentioned processes for the entire sample space. What is the best way of plotting these before and after values such that people understand that there is a general decrease in the band power?