texts: GridSearchCV using Random Forest Reg Pipeline

texts:
rfr = RandomForestRegressor(n_estimators=1, min_samples_leaf=7)
preprocess = make_column_transformer(
                 (StandardScaler(), ~categorical),
                 (OneHotEncoder(handle_unknown='ignore'), categorical))

param_grid = {"RandomForestRegressor__n_estimators": [1,10,20,50,75,200],
              "RandomForestRegressor__min_samples_leaf": range(1,10)}
pipe = make_pipeline(preprocess, RandomForestRegressor())

grid = GridSearchCV(pipe, param_grid, cv=5)

When I do grid.fit(X,y) I am getting following error. Why?

ValueError: Invalid parameter RandomForestRegressor for estimator Pipeline(memory=None,
         steps=[('columntransformer',
                 ColumnTransformer(n_jobs=None, remainder='drop',
                                   sparse_threshold=0.3,
                                   transformer_weights=None,
                                   transformers=[('standardscaler',
                                                  StandardScaler(copy=True,
                                                                 with_mean=True,
                                                                 with_std=True),
                                                  VendorID                 False
store_and_fwd_flag       False
RatecodeID               False
PULocationID             False
DOLocationID             False
passenger_count           True
trip_distance             True...
                 RandomForestRegressor(bootstrap=True, ccp_alpha=0.0,
                                       criterion='mse', max_depth=None,
                                       max_features='auto', max_leaf_nodes=None,
                                       max_samples=None,
                                       min_impurity_decrease=0.0,
                                       min_impurity_split=None,
                                       min_samples_leaf=1, min_samples_split=2,
                                       min_weight_fraction_leaf=0.0,
                                       n_estimators=100, n_jobs=None,
                                       oob_score=False, random_state=None,
                                       verbose=0, warm_start=False))],
         verbose=False). Check the list of available parameters with `estimator.get_params().keys()`.


texts: Does an RL agent learn during exploitation?

texts:
I have started with RL and have some doubts regarding it.

Does an RL agent learn during exploitation, or does it only learn during exploration?
Is it possible to train a model only using exploitation (i.e. where exploration is not allowed)?


texts: Using NLP to automate the categorization of user description

texts:
I have a huge file of customer complaints about the products my company owns and I would like to do a data analysis on those descriptions and tag a category to each of them.

For example: I need to figure out the number of complaints on Software and Hardware side of my product from the customer complaints. Currently, I am using excel to do the data analysis which do seek a significant amount of manual work to get a tag name to the complaints.

Is there a way in NLP to build and train a model to automate this process? I have been reading stuffs about NLP for the past couple of days and it looks like NLP has a lot of good features to get a head start in addressing this issue. Could someone please guide me with the way I should use NLP to address this issue?


texts: What is the use of applying img_to_array() after cv2.imread()

texts:
In a book, I saw the following code to load images from a directory:
1.image = cv2.imread(imagePath)
2.image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
3.image = cv2.resize(image, (28,28))
4.image = img_to_array(image)
5.data.append(image)

cv2.imread()

It converts image to a numpys array in BGR format

img_to_array()

It converts PIL instance to numpy array, but in this case image will stay in BGR format as the image is loaded by cv2.

What is the use of img_to_array() (from keras.preprocessing.image module) after cv2.imread(), as it is already in numpy array.
Also:
To double check, I displayed images before and after applying img_to_array() using cv2.imshow().
BEFORE applying img_to_array()

AFTER applying img_to_array()###

Also Also:
But, if I try to save the images using cv2.imwrite into a file, they both are saved as normal (ie. like the first image). Why is this happening?


texts: StandardScaler before and after splitting data

texts:
When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before splitting the data into train/test, but when i was checking some of the codes posted online (using sklearn) there were two major uses.

1- Using StandardScaler on all the data. E.g.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_fit = sc.fit(X)
X_std = X_fit.transform(X)

Or

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit(X)
X = sc.transform(X)

Or simply

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_std = sc.fit_transform(X)

2- Using StandardScaler on split data.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

I would like to standardize my data, but I am confused which method is best!


texts: How to deal with similar feature values but each indicates to a different information?

texts:
If I have a feature with replicated values but each of these values indicates a different piece of information.

example: feature 'street name' with value 'A' which some of these 'A's are for Boston city and the others for New York City.

In this case, should I change the street name values which indicate NYC to something like 'A_NYC'? I assume that if I kept the 'A' values with no change then there will be a relation between Boston's 'A' and NYC and built a correlation between unrelated features.

I hope I made my point clear and sorry if this question was answered previously and I didn't notice.


texts:
In most of my projects, I come up with models and want to visualize how some property $x$ varies as a function of a subset of parameters $p_1$,$p_2$, .. etc.

So I'll often end up with figures of the "parameter scan" which look like this

Those are very helpful for explaining a model / process / datasets.

The problem is: I put an inordinate amount of work into producing the data necessary to generate these figures. Most of it wasted on the pipeline itself. I often have to couple anything from 2 to 10 mismatched programs, deal with thousands of intermediary files, track the parameters in the file name along the way, etc.
It quickly lead to overwhelming bash scripts.

Is there any principled way to go about managing a pipeline of mismatched programs, with parameter scans in mind?
Say a python module where the user can declare wrappers and how they interact together. Then the user just "launch the pipeline" on every point of the parameter space and recover the data in easily potable format array?


texts: What to do when we can't trust our human classifiers?

texts: Suppose we want to design a neural network that can diagnose skin cancer. We want this neural network to consider the possibility that the doctor we hired misclassified some of our images while labeling. How can we design our neural network?


texts:
I am working on a project using GP-regression models to model transition and measurements models in a Kalman Filter. This means I need to be able to sample from the derivative of the original GP model.

I am aware of how to combine the various kernels offered in the GpyTorch library, but is there any way I can implement my own mean and covariance functions?

In the case of an RBF-Kernel the posterior mean and covariance would be.

\begin{equation}
 \begin{aligned}
  \bar{f}_* &= \mathbf{k}(\mathbf{x}_* \, \mathbf{X}) K(\mathbf{X}, \mathbf{X}) ^{-1} \mathbf{y}\\
  &\stackrel{\triangle}{=} \mathbf{k}(\mathbf{x}_* \, \mathbf{X}) \mathbf{\alpha}
 \end{aligned}
\end{equation}

Posterior Covariance
\begin{equation}
 \begin{aligned}
  \text{covariance} &= K(\mathbf{x}_*, \mathbf{x}_*) - K(\mathbf{x}_*, \mathbf{X})K(\mathbf{X}, \mathbf{X})^{-1}K(\mathbf{x}_*, \mathbf{X})^T \\
 \end{aligned}
\end{equation}

The Jacobian of the mean function would then be which I would like to use to linearize the model would then be:
\begin{equation}
 \begin{aligned}
  \frac{\partial\bar{f}_*}{\partial \mathbf{x}_*} &= \frac{\partial k(\mathbf{x}_* \, \mathbf{X})}{\partial \mathbf{x}_*} \mathbf{\alpha}\\
  &= \left[ \Lambda^{-1} \tilde{\mathbf{X}}^T_* (\mathbf{k}(\mathbf{x}_* \, \mathbf{X})^T \odot \mathbf{\alpha}) \right] \in \mathcal{R}_{D \times 1}
 \end{aligned}
\end{equation}

Is there any way I can implement this as a mean function for my GP-model?
In the case of the filter the state-vector $\mathbf{x}_*$ would contain the current state and the control actions.


texts: Approipriate use of rfe with xgboost classifier

texts:
I have an imbalanced data set with about 6000 features. I am using the XGBoost classifier to classify the labels. However, due to the large number of features, I am opting for rfe function to select an appropriate number of features (to prevent overfitting).

Is there a less expensive feature selection alternative to rfe? I would like a demo maybe...

I am pretty stuck right now and I am not sure how to do this properly.

Here is my code:

y=training['prediction']
x=training.drop(['prediction'],axis=1)

imp_mean =SimpleImputer(missing_values=np.nan,strategy ='most_frequent')
impx=pd.DataFrame(imp_mean.fit_transform(x))
imptest=pd.DataFrame(imp_mean.fit_transform(test))

x=impx
test=imptest

model =XGBClassifier()

feat_selector = RFE(model,10)

feat_selector = feat_selector.fit(x, y)

f =feat_selector.get_support
x = df[df.columns[f]]

x_train,x_val,y_train,y_val = train_test_split(x,y,test_size =0.30,random_state=1)

scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_val=scaler.fit_transform(x_val)

abc =XGBClassifier(max_depth=2,n_estimators=50)

abc.fit(x_train,y_train)

y_pred =abc.predict(x_val)
print(accuracy_score(y_val,y_pred))
print(confusion_matrix(y_val,y_pred))

Thanks.


texts:
I am trying to find some kind of activation regularization that will have the following effect: assuming that the layer has output shape (batch_size,4), the regularization will force it instead of outputting (for one sample)
[0.9, 0.1, 0.2, 0.5]
will output:
[1.0, 0.0, 0.0, 1.0].
At first, I was trying to use top_k tensorflow function instead of regularization, but then I realized that is obviously not differentiable. Then I moved on to modifying softmax function with no success. Finally I realized that sparse regularization of a layer might be a solution. I know that sparsity is usually implemented with KL divergence, similarly to sparse autoencoder described here (https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf) but the sparsity here is calculated in a per-unit basis (mean over batch), while I would like to use per-batch basis (mean over units) if that is possible. Maybe I should try L1 regularization?
I am able to implement it fine but so far it hasn't given me the needed results. Let me know what you think. My current implementation for Keras custom regularizer (for a dense layer) looks like:
def __call__(self,x):
      p_hat = K.mean(x,axis=1)
      p_vec = self.p * K.ones_like(p_hat)
      loss = K.sum(self.p *KLD(p_vec, p_hat) + (1-self.p)*KLD(1-p_vec,1-p_hat))
      print("Loss: "+str(loss))
      print("Mean act: " +str(p_hat))
      return self.strength * loss

Cheers,
Maks


texts: How to find similarity/distance matrix with mixed Continuous and Categorical data?

texts:
Say I have a dataset like this:

Hotel  HasPool AvgPrice
 1        1      $123
 2        0      $234
 3        1      $200

Currently I have broken down the dataset into 2 (one containing all continuous, other all categorical). The continuous one, I am calculating euclidean distance b/w all observation whereas for the categorical one I calculate Cosine distance . Is there a way to combine both scores effectively. Or, is there a distance function which works for both data types ?
My final output should look like this:

Hotel  1   2   3
  1    1  0.3  0.7
  2    0.3  1  0.5
  3    0.7  0.5 1

Any help will be appreciated! Thanks


texts: If I have negative and positive numbers for a feature, should MinMaxScaler be -1 to 1?

texts:
I have a variable X with values ranging from -150 to 400. All the other variables in my training set are positive so I normalized them to be from 0 to 1, or they’re already binary, or they had a Gaussian distribution so I used StandardScaler.

For this variable X with some negative values, is it important, generally, that I normalize them to -1 to 1 (due to the negative values) rather than 0 to 1?


texts:
I have a problem, im designing a multiclass classifier to classify medic images, I have to classify in which grade of desease is it, this are 6 grades , each time the joint deforms a little, so, mi original dataset was imabalanced, each class have like 16 to 200  images that are very very similar with some little deatils of change, ,so i did some contrasts and flips to the images to have like 500 images per class aprox, i used a vgg16 architechture to train the model but, it soesnt work 30% acc, i ddid a simple model with 2 layers of convolution, I have like 70% of accurracy but my recall is very very low like 0.001 to 0.3, and when I predict with an image, all the images goes to the incorrect class, i dont know how to correct my model , use other metrics or some architechture for this type of images?.Here's a sample of the code https://github.com/Franciscogtu/OARSI-IMAGE-CLASSIFIER.git
Thank you for the help


texts: LightGBM - Why Exclusive Feature Bundling (EFB)?

texts:
I'm currently studying GBDT and started reading LightGBM's research paper.

In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by regrouping mutually exclusive features into bundles, treating them as a single feature. The researchers emphasize the fact that one must be able to retrieve the original values of the features from the bundle.

Question: If we have a categorical feature that has been one-hot encoded, won't this algorithm simply reverse the one-hot encoding to a numeric encoding, thereby cancelling all the benefits of our previous encoding? (suppression of hierarchy between categories etc.)


texts: Time series classification, without the time dimension

texts:
Edit
Thanks to the answer of @pcko1, I understand that I should use data augmentation to make my model resilient to order of data points.
Clarification after the answer of @Icrmorin :
My problem is actually more complex than just finding bullets. I also need to find title, order text appropriately (think 2-columns PDF), find header/footer, etc...
I am currently trying a rule-based approach, and it was working well so far. But as the number of PDF format I need to handle grow, the complexity of the code grow as well, and I'm reaching a point where PDF formats have opposite features and can't be handled by the same code.
I was hoping the machine learning approach might solve this issue and works well for any format.
According to the answer of @mariq vlahova, it seems there is no name for this kind of task ? Just use LSTM ?
end of edit

I'm looking for the name of a task, in order to search more literature about the subject.
The best I can describe this task is given in the title itself...
Basically I have data points with several features each, and I need to classify each of these data points, but not independently. However the order does not (or should not) matter.

Example
For example I have 3 data points [D1, D2, D3], and I want to classify is as [True, False, False].
These data points are dependent, ie changing 1 data point might alter the result of other as well :
[D1, D2', D3] might be classified as [True, True, True].
Also, the order does not matter (that's why I wrote "without the time dimension"):
If [D1, D2, D3] is classified as [True, False, False],
then [D2, D3, D1] should be classified as [False, False, True].

Context
A bit more context... Basically I want to classify PDF content, as being a bullet point or not.
So I want to parse my PDF file, extract chunks of text along with additional information (font size, position, etc...), and classify these chunks as bullet or not bullet.
But we can't classify each chunk independently. Consider the following example :
...End of previous paragraph

1. This is a title

Beginning of next paragraph...

In this case, 1. This is a title should not be classified as bullet. But :
1. This is a title

2. This is a second title

3. This is a third title

In that case, 1. This is a title should be classified as bullet.
I need to find literature about this problem but I don't even know the name of the task...


texts: How to store and query biometric data for an authentication system?

texts:
I am trying to design, and hopefully implement, an authentication system which centers around the use of biometric images. I plan to use different machine learning and deep learning techniques to help with this through Tensorflow. Specifically, the project will focus on the use of image processing for authentication.
I know that in a traditional authentication system, for example using a username and password combination, both data are stored in a database with a specific data type (usually strings). When verifying the user, we could just query the username and match the hashed password. This is pretty simple because we are storing structured data. However, with biometrics, data are usually unstructured usually in the form of images. It is difficult to efficiently store unstructured data and it is even more difficult to match a stored unstructured data with a new one for testing. It is just too computationally extensive to test the set of stored data, one by one, to the new input data.
One idea I had is to generate a function taking these images as input and output a number which would act as an identifier for that specific image. It is like a function which places images in a vector space and the images can be matched when they have the similar (or close to similar) value in the vector space. Is this appropriate approach with my use case? However, I am still having problem how and where will I store my data.
I am still a beginner in terms of data science so it will gladly help if you could recommend me further readings or similar projects.


texts:
In this paper (page 1 abstract) which considers regularization technique, the author used the word "norm" - what does it stand for?
Is it related to Batch Normalization / L1 or L2 Normalization?

  "We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing
  the squared distance between successive hidden states’ norms"

Googling "what is a norm Neural Net" doesn't provide any links


texts: How do you make an NN for image classification invariant to translation **and** rotation?

texts: CNN are (I think) invariant to small translations of the input image, i.e., they will classify to the same class an image $X$ and an image $X'$ such that all pixels have been translated along a vector $\mathbf{v}$ by some "small" distance $d$. They're not, however, invariant to rotation. I.e., if $X'$ is obtained by $X$ applying a rotation of arbitrary degree $\theta$ around an axis $z$ orthogonal to the plane of the image, they won't necessarily classify it to the same class as $X$. Practitioners usually solve this by data augmentation, but this is unnecessarily wasteful and anyway not an option in my case. Is there a way to make the NN architecture invariant to both isometries? I don't have to use a CNN, but since CNN already enjoy some sort of translation invariance, I figured it would be easier to modify them to get rotational invariance , than to use a completely different architecture.


texts: What does localist one-hot vector mean in cs224n NLP course?

texts:
Chris said one-hot is a "localist" representation.

what does "localist" mean here?

I've searched on recommended text, didn't find explanation.

any clue?


texts: Keras: very low accuracy, very high loss and predictions are the same for every input

texts:
I am developing a system for general-purpose audio tagging using keras.

I have the following data input:

x_train has 10 different data for each input (data length, max, min, etc.) y_train represents 41 possible labels (guitar, bass, etc.)

Here is my code below:

x_train shape = (7104, 10)
y_train shape = (41,)

print(x_train[0])

[ 3.75732000e+05 -2.23437546e-05 -1.17187500e-02  1.30615234e-02
  2.65964586e-03  2.65973969e-03  9.80024859e-02  1.13624850e+00
  1.00003528e+00 -1.11458333e+00]

print(y_train[0])

[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]

My model is:

from keras.models import Sequential
from keras.optimizers import SGD
from keras.layers import Dense, Dropout, Activation

model = Sequential()

model.add(Dense(units=128, activation='relu', input_dim=10))
model.add(Dropout(0.5))
model.add(Dense(units=64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(units=32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(41, activation='softmax'))

opt = SGD(lr=0.001, decay=1e-6, momentum=0.9, nesterov=True)

model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

model.fit(np.array(x_train), np.array(y_train), epochs=5, batch_size=8)

This is my result:

Epoch 1/5
7104/7104 [==============================] - 1s 179us/step - loss: 15.7392 - acc: 0.0235
Epoch 2/5
7104/7104 [==============================] - 1s 132us/step - loss: 15.7369 - acc: 0.0236
Epoch 3/5
7104/7104 [==============================] - 1s 133us/step - loss: 15.7415 - acc: 0.0234
Epoch 4/5
7104/7104 [==============================] - 1s 132us/step - loss: 15.7262 - acc: 0.0242
Epoch 5/5
7104/7104 [==============================] - 1s 132us/step - loss: 15.6484 - acc: 0.0291

As you can see, my results show very high data loss and very low accuracy. The main problem occurs when I try to predict the result. For each one input the output is the same. How can I fix this?

pre = model.predict(np.array(x_train), batch_size=8, verbose=0)

for i in pre:
    print(i)

[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
...

```


texts: How to Visualise a CNN model using Python

texts:
I am new to Deep Learning and have been trying to show a plot of the CNN architecture using Python in Google Colab. Besides importing the necessary libraries, I have noticed from other resource that normally, we would declare a model {model = sequential()}, and then model.add (Conv2d or MaxPool or Activation etc.) and for visualising, print(model.summary()).

However, I have developed the CNN architecture using the below code and struggling with generating a model print. Could anybody tell what I am missing here. Thanks

input = Input(shape=X.shape[1:])                                  # 154x154x3
x = Conv2D(12, (3, 3), padding='same', activation='relu')(input)  # 154x154x12
x = Conv2D(12, (2, 2), strides=(2, 2), activation='relu')(x)      # 77x77x12
x = Conv2D(16, (3, 3), padding='same', activation='relu')(x)      # 77x77x16
x = Conv2D(16, (2, 2), strides=(2, 2), activation='relu')(x)      # 38x38x16
x = Conv2D(24, (3, 3), padding='same', activation='relu')(x)      # 38x38x24
x = Conv2D(24, (2, 2), strides=(2, 2), activation='relu')(x)      # 19x19x24
x = Conv2D(32, (3, 3), padding='same', activation='relu')(x)      # 19x19x32
x = Conv2D(32, (2, 2), strides=(2, 2), activation='relu')(x)      # 9x9x32
x = Conv2D(48, (3, 3), padding='same', activation='relu')(x)      # 9x9x48
x = Conv2D(48, (2, 2), strides=(2, 2), activation='relu')(x)      # 4x4x48
x = Dropout(0.5)(x)                                               # 4x4x48
x = Conv2D(n_classes, (1, 1))(x)                                  # 4x4x62
x = GlobalAvgPool2D()(x)                                          # 62
output = Activation('softmax')(x)                                 # 62


texts: Check If Answer is Correct by Similarity

texts:
I am new to data science and machine learning.

Let's say that I have a question, and some correct answers for that question (for example, 10 correct answers). Is there a way to get a new answer as input, and "calculate" whether it is right? If you can recommend some readings, that would be great.


texts: How to detect multiple digits in one image using CNN

texts:
The network in CNN recognize one digit for one image,
 how can I recognize two or more digits in one image.
A proper way I've thought about is modifying the weights in the FC layer.
In other words, if there is two three digits in a image, the fully connected layer's weight can be expressed as (in tensorflow):

w_output = __weight_variable([1024, number_of_digits*number_classes], stddev=0.1)
If we want to recognize one digit in one image, just remove the
number_of_digits in the above statement.

Another question occurred when the number of digits is unknown(Maybe two or more). How to modify the network to recognize all the digits in the image?


texts: What does it mean to "share parameters between features and classes"

texts: When reading this paper there is a line which says "linear  classifiers  do not  share  parameters  among  features  and  classes." What is the meaning of this statement? Does it mean that linear classifiers such as logistic regression need features that are mutually independent?


texts: Applying mean encoding before or after splitting into train and test set

texts:
I have a dataset of 50000 observations with columns of high cardinality. The best way to encode them is with mean encoding, then to use regularization. I will use CV rather than smoothing. But when I see people use it, they use it on train and test set.

Should I first split my dataset into train and test set and then encode or can I encode directly from the beginning on my full dataset?

If I should split the data into train and test set first, can someone tell me why?


texts: Improving Validation Loss and Accuracy for CNN

texts:
I am new to CNNs and need some direction as I can't get any improvement in my validation results.

I am trying to do binary image classification on pictures of groups of small plastic pieces to detect defects. Unfortunately, I am unable to share pictures, but each picture is a group of round white pieces on a black background. One class includes pictures with all normal pieces, the other class includes pictures where two pieces in the picture are stuck together - and therefore defective.

I have a small data set: 250 pictures per class for training, 50 per class for validation, 30 per class for testing. The pictures are 256 x 256 pixels, although I can have a different resolution if needed.

Here is my CNN architecture:

classifier = Sequential()
classifier.add(Conv2D(32, (7, 7), padding="same", input_shape=(256, 256, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Dropout(0.5))

classifier.add(Conv2D(64, (5, 5), padding="same", input_shape=(256, 256, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Dropout(0.5))

classifier.add(Flatten())

classifier.add(Dense(units=128, activation='relu'))
classifier.add(Dense(units=1, activation='sigmoid'))

classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Treatment done to images:
train_datagen = ImageDataGenerator(rescale=1./255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)
validation_datagen = ImageDataGenerator(rescale=1./255)

train_batch_size = 10
val_batch_size = 10
num_epochs = 100
train_images = 250
val_images = 50

classifier.fit_generator(training_set,
                             steps_per_epoch=train_images // train_batch_size
                             epochs=num_epochs,
                             validation_data=validation_set,
                             validation_steps=val_images // val_batch_size)

Here are the results:

It's overfitting and the validation loss increases over time. The validation accuracy is not better than a coin toss, so clearly my model is not learning anything.

I have tried different values of dropout and L1/L2 for both the convolutional and FC layers, but validation accuracy is never better than a coin toss.

I understand that my data set is very small, but even getting a small increase in validation would be acceptable as long as my model seems correct, which it doesn't at this point.

Update:
Switching from binary to multiclass classification helped raise the validation accuracy and reduced the validation loss, but it still grows consistenly:

Any advice would be very appreciated. Thanks in advance!


texts: Is the Tree Parzen Estimators topological structure supposed to be updated during HPO?

texts: IN the original hyperopt paper, it is not clear whether the tree parzen search space is supposed to be updating its topological order with every update.


texts:
I have a dataset of 5K records and 60 features focussed on binary classification. Please find my code below for SVM paramter tuning. It's running for a longer time than Xgb.LR and Rf. The other algorithms mentioned returned results within minutes (10-15 mins) whereas SVM is running for more than 45 mins.

Questions

1) Is SVM usually slower and takes longer time?

2) Is there any issue with my code below?

3) How can I make the gridsearch faster?

from sklearn.svm import SVC
param_grid = {'C': [0.001,0.01,0.1,1,10,100,1000],
              'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
              'kernel': ['linear', 'rbf','poly'],
              'class_weight':['balanced']}
svm=SVC()
svm_cv=GridSearchCV(svm,param_grid,cv=5)
svm_cv.fit(X_train_std,y_train)


texts: How to have an insight of very similar processing methods by measuring the distance between observations produced by them?

texts: Suppose that we use very similar methods(suppose now there are two such similar methods) to process upon each sample of a dataset. For each sample of the dataset with one of the two methods, we obtain a set of observations. Every observation has the same dimension and the same range(e.g. 0.0~1.0) for every data point, even between the two methods, they are the same. From now on, we say these two methods are method 1 and method 2. For method 1, there are observations, M1_O_1, M1_O_2, ...... , M1_O_n; for method2, there are observations, M2_O_1, M2_O_2, ...... , M2_O_n. Say M1_O_i and M2_O_i are the two obsevations taken on the same sample of the considered dataset. For observation M1_O_i, there are p data points, d_1_M1_O_i, d_2_M1_O_i, ...... , d_p_M1_O_i, although their dimension and range are the same, their order is important, we cannot, for example, mix d_p_M1_O_4 with d_p_M1_O_5. The same for method2, so there are observations, M2_O_1, M2_O_2, ...... , M2_O_n, and for observation M2_O_i, there are d_1_M2_O_i, d_2_M2_O_i, ...... , d_p_M2_O_i. We want to measure the distance between d_j_M1_O_i and d_k_M2_O_i for each j and k by iterating over i. The final distance designated by each j and k is irrelevant with i, but we have to iterate over i to get it. We want to measuring them to have an insight of these two methods(The methods are given and cannot be changed). Besides average Euclidean distance, what other measuring method can we taken? Or if there is any other distance measuring method even neural network ones can be adopted?


texts: What is the difference between these 2 training scenarios?

texts:
I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The original scenario (assuming I have a very big RAM that can contain the whole ~30GB dataset to be processed) is that I'm going to train this for 10 epochs with the first 6 epochs to have a learning rate of 0.001. After that, every subsequent epoch, the learning rate will be reduced by half. At the moment, what I am doing is to train each part for 10 epochs following the scheduled learning rate stated previously. After some thinking, I then realized that there may be a better training scenario that resembles better with the original scenario. Which is to loop all the 20 parts using the first 6 epochs (learning rate= 0.001), and then going to the next loop will be 1 epoch with learning rate of 0.0005 (half of the previous epoch), and looping again by halving the learning rate until it reaches the original 10 epochs that I want to achieve. Here is the summary of these 2 scenarios.

Scenario 1: Part 1 data are trained with 10 epochs with scheduled learning rates stated above. After finish training part 1, the model is saved and then loaded again for further training using part 2 data. This part 2 data will be used for training the previously trained model for 10 epochs with the same scheduled learning rate. This loop is continued until all 20 parts are gone through.

Scenario 2: Part 1 trained with 6 epochs with a learning rate of 0.001. After that, the model is saved and further continue the training using part 2 data with another 6 epochs using the same learning rate (0.001). This continues until all 20 parts have been feed into the model. After that, we go back to part 1, but now we trained the model with only 1 epoch using a 0.0005 learning rate (0.5 * 0.001). Continue the training until all 20 parts are gone through. Re-loop again with 0.00025 learning rate (0.5*0.0005). We will stop re-looping until the learning rate becomes 0.0000625.

I think scenario 2 is the one that matches the original scenario but I'm not sure why scenario 1 can't achieve the same thing as scenario 2. Thank you in advance, and sorry if there is any wrong grammatical usage.


texts: xgboost and linear regression new feature analysis

texts:
For linear regression, seems like a new feature has to be a linear relation with the target variable.

But If you make the new feature for the Xgboost, what do you have to consider to make a new feature for a xgboost or lightgbm ?


texts: Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib

texts:
I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:

Storm alone and implement streaming naive bayes on my own in storm topology.
Storm + TridentML
Storm + SAMOA
Spark Streaming + MLlib

What is the best framework set to choose and start working on. Any suggestion will be of great help.


texts: What is the formula and log base for idf?

texts:
To calculate tf-idf, we do:

tf*idf

tf=number of times word occurs in document

What is formula for idf and log base:

Log(number of documents/number of documents containing the word)
Log((1+number of documents)/(1+number of documents containing the word))
1+Log(number of documents/number of documents containing the word)
1+Log((1+number of documents)/(1+number of documents containing the word))


texts:
I have 1 year transaction level data aggregated at a weekly level for 1000 different stores. I want to cluster similar stores based on 8 variables such as sales, customer count etc. The concern is that as the data is at a store week level I cannot directly run the clustering algorithm as it may compare and classify the data points from same store into one cluster, which is not what is required.

So the second option is to transpose the data i.e. 1 data point for each store having 8(variable) * 52(week) columns and then run clustering on it. I am not sure if k-means or the converntional clustering method will work well on such high dimensional data.

So I am looking for suggestions as to how can we handle this situation.


texts: Does data anonymization conflict with GDPR rules?

texts: There are GDPR articles which relate to a persons ownership about their data e.g. Art. 17 GDPR Right to erasure (‘right to be forgotten’) and Art. 20 GDPR Right to data portability. In case one would anonymize the data without a way to "restore" the relation between the person (name + e-mail adress) (which in turn would allow handling of the person specific data) I'd say this would conflict with these GDPR articles. Are there data anonymization techniques which allow to "restore" the relation between name + contact e-mail after the data has been anonymized? This would allow to satisfy these GDPR rules.


texts:
I have the following bar plot showing retrieval precision for specific documents:

First n documents have no specific precision because retrieval returned no data for them. This is ambiguous because no bar can mean either 0 value or lack of data.

How can I make it clear in a bar plot?

Some ideas:

render specific values close to the bar
fillna(-1) and render negative values in red

Any other?

Here's Panda series I am trying to render:

9881449     0.375
22860031    0.625
18222644    1.000
5708161       NaN
10021344      NaN
10021491      NaN
10021595      NaN
10021614      NaN
10021709      NaN
10022498      NaN
10023240      NaN
10023252      NaN
10023458      NaN
10023706      NaN
22239796      NaN
dtype: float64


texts: Is the PNY NVIDIA Quadro RTX 4000 a good GPU for Machine Learning on Linux?

texts:
As a web developer, I am growing increasingly interested in data science/machine learning, enough that I have decided to build a lab at home.

I have discovered the Quadro RTX 4000, and am wondering how well it would run ML frameworks on Ubuntu Linux. Are the correct drivers available on Linux so that this card can take advantage of ML frameworks?

LINUX X64 (AMD64/EM64T) DISPLAY DRIVER

This is the only driver that I could find, but it is a "Display Driver", so I am not sure if that enables ML frameworks to use this GPU for acceleration. Will it work for Intel based processors?

Any guidance would be greatly appreciated.


texts: How is a coincidence matrix constructed for computing Krippendorff's alpha?

texts:
I am looking at two documents to help me learn about constructing coincidence matrices in order to gain a better understanding of Krippendorff's alpha. I am using these two:

https://repository.upenn.edu/cgi/viewcontent.cgi?article=1043&context=asc_papers
https://en.wikipedia.org/wiki/Krippendorff%27s_alpha

There seems to me to be a discrepancy between the two. There probably isn't, but I'm looking for some help in figuring out whether my understanding is wrong, or if there is indeed a discrepancy.

In link 1, I am looking at section B ("Nominal data, 2 observers, no missing data"), where the coincidence matrix is presented. In link 2, I am looking at the section "Coincidence matrices".

Consider the reliability matrix presented in link 1:

In order to calculate the elements of the coincidence matrix, we have the following definition in link 2:

$$o_{vv'}=\sum_{u=1}^{N}\frac{\sum_{i\neq i'}^{m}I(v_{iu}=v)I(v_{i'u}=v')}{m_u-1}=o_{v'v},$$

where $u$ is the horizontal element of the reliability matrix (the columns), $m_u$ is the number of labels actually present in column $u$.

This seems simple enough. For element $o_{aa}$ (or $o_{11}$) we should have:

$$
o_{aa}=\frac{I(a=a)I(b=a)}{2-1}+\frac{I(a=a)I(a=a)}{2-1}+\frac{I(b=a)I(b=a)}{2-1}+\frac{I(b=a)I(b=a)}{2-1}
$$

$$
+\frac{I(d=a)I(b=a)}{2-1}+\frac{I(c=a)I(c=a)}{2-1}+...
$$

and so on. Clearly, only one of the summation elements is non-zero, namely the second element. Hence $$o_{aa}=1.$$

Using the same formula/logic, we arrive at $$o_{bb}=2.$$

But if we look at link 1, it is getting double the value for the coincidence matrix elements, and I don't understand why. I don't even understand the link's explanation as to why it is getting those values for the coincidence matrix.

Can somebody help?


texts: What does it mean that classes are mutually exlcusive but soft-labels are accepeted?

texts:
The Tensorflow's documentation of softmax_cross_entropy_with_logits:

  Measures the probability error in discrete classification tasks in
  which the classes are mutually exclusive (each entry is in exactly one
  class). For example, each CIFAR-10 image is labeled with one and only
  one label: an image can be a dog or a truck, but not both.

  NOTE: While the classes are mutually exclusive, their probabilities
  need not be. All that is required is that each row of labels is a
  valid probability distribution. If they are not, the computation of
  the gradient will be incorrect.

At first glance it may seen contradictory, but my guess is that, provided these conditions for the classes and for the labels, the probability distribution of the labels have always one maximum. For instance, labels = [0.5 0.5] is a valid probability distribution in a binary case but it does not comply with the exclusivity of the classes. If I'm wrong, I don't get the idea behind the documentation.

It is also pointed out that for mutually elusive probabilities, sparse_categorical_cross_entropy should be used, but I think that softmax_cross_entropy_with_logits can be used if the labels are one-hot encoded, which is a valid probability distribution (a deterministic one)

Edit:

When going around this question again, I'm wondering if in a multi-class exclusive case where the only constraint on the labels are that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...


texts: Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

texts:
I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity.

For example :

string one : 'Pair of women's shoes'

string two : 'women shoes' pair'

Logically I would want a high score between the two strings. Is there any way to do so ? I am comparing array of strings with another array in a single column in my dataframe. I want to find similar rows this way. Can this be achieved ?


texts: Keypoint detection from an image using a neural network

texts:
I am trying to design and train a neural network, which would be able to give me coordinates of certain key points in the image.

Dataset

I've got a dataset containing 1800 images similar to these:

This dataset is generated by me. Each image contains two circles, one smaller and one bigger, generated randomly in the image. My goal is to train the neural network to return 2 sets of coordinates, each of them pointing precisely at the center of the circle. Each image has the shape (320, 320, 1).

Current model

I've been successful to do so to some degree, but it's not good enough. Below, you can see the neural net architecture I've been most successful so far. I use Python, Tensorflow 2 including Keras. I use Adam optimizer, MeanSquaredError loss and RootMeanSquaredError as a metric.

The current result looks as follows. The coordinates that the neural net gave me are drawn in the image.

As you can see from the first image, the result is quite precise that I am almost satisfied with. But the average result looks like as in the second image, which is not good at all.

I have trained this model for 35 epochs in a total of 4 runs and it is not able to learn any further as you can see from the Tensorboard.

I have tried many different variations of the architecture and tuned hyperparameters. I am not satisfied with the result I got so far. I plan to continue on detecting keypoints from images on more complex datasets and that is the reason why I'm trying to make some progress on much simpler datasets first and add complexity gradually.

I would appreciate any advice you can give me on the model architecture that I would have better results with or maybe a different approach. Tell me if you need to know more implementation details.

Thanks

Edit: To complete the details, Conv2D layers use leaky relu activation function and both dense layers use the sigmoid activation function.


texts: Estimate the location of an object in a field using computer vision and math

texts:
I'm trying to see how to detect the location of a soccer ball in the field using the live camera. What are some ways to achieve this? At this point I'm more interested in ideas and thoughts and not really code.
1- Assuming we have a fixed camera with a wide shot. How to find the ball location on the actuall field? It's probably using a computer vision approach.
2- A camera is zooming into the ball. But we know the location of the camera and maybe it's turning angle. Can we estimate the ball location on the field using this info? Or maybe we need additional info? Can we do it with two cameras as reference points? Maybe some math or triangulation sort of approaches would help here.
Any thoughts would be helpful.


texts: visually explication of dense layer of keras?

texts:
I am learning keras and I would like to know how the dense layers of keras are visually interpreted, I do not understand very well the concepts and arguments of

model.add (Dense (512, activation = 'relu', input_shape = (784,)))

I know that input_shape = (784,) would be drawn as 784 input neurons ,but I don't understand what 512 is?


texts:
Let's say I've found some outliers in a column in my dataset and have decided to remove them.

Should I do this before or after I split the dataset into train/test sets?


texts: What is the best algorithm/solution for predicting the following?

texts:
I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country:

If I were to take a random sample of the 6 variables - an individual within one of the countries - how would I best go about predicting to which country that individual belongs?

I have a whole separate dataset, with thousands of data points, so I know which country the data belongs to and can thus know for certain whether the algorithm is predicting accurately.

Thus far I have been looking at decision trees and random forest options, but none of the example use cases I've seen translate very well to what I am trying to do. Perhaps I'm looking in the wrong place.

It's not predicting the behavior, as the behavior is already known...it's more about predicting to which country-classification the behavior belongs. Ideas? Comments?


texts: Extracting NER from a Spanish language text file

texts: I am trying to extract various Named Entities from a Spanish language text file. I tried using nltk but to no success. I am using Python 2 with nltk 3.x.


texts: Regularization with Batch Normalization on DNN

texts:
When using Batch Normalization I've seen that L2 Regularization shouldn't be used because the only impact will be on the learning rate (source: https://blog.janestreet.com/l2-regularization-and-batch-norm/). I've also seen it stated that dropout can have unintended consequences when paired with batch normalization (source: https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/)
Batch normalization is only providing a minimal amount of regularization. What options exist for adding regularization to my neural networks? Is it just L1?


texts: How to find which patch in orignal image does an activation correspond to in vgg net after the final pooling layer

texts:
So I am working on the NeurIPS 2019 reproducibility challenge, The link to the paper is https://arxiv.org/abs/1806.10574. So basically we have a vgg-16 net with the final fully-connected layers removed, so we get the 7x7x512  dimensional activation maps, Now the paper introduces a novel prototype layer in which we compare these 7x7 activation maps with 1x1x512 dimensional prototypes using the L2 norm, and wherever the prototype has the maximum similarity after considering all the input images we equate the prototype to that 1x1x512 patch in the activation maps

Now all is good and fine, but the paper further wants to find the patch in the original image corresponding to the 1x1 patch, so we can say that the prototype is effectively somewhat that patch in the original image.

Now what I found out is that the 224x224 to 7x7 transition can be effectively done by 212x212 kernel with 32 strides (using the effective local receptive field) image explaining this  (Not mine, copied the image from somewhere)

Okay so now if I get a 1x1 activation which I need to find the original image patch of ..its quite easy...but the problem is that in VGG net we apply zero-padding of 1 many time, so effectively we are getting a padding of 90(calculated using input size of 224, kernel size of 212 and stride of 32) in the original image , so suppose I get the top-left 1x1 activation as the one I want to find the patch of, then it will have a lot of portion = 0 due to the heavy effective padding in the original image, So is this okay ?

Sorry for the extremely long question, but it's a conceptual one and needed a deep explanation


texts: Can't train input variable with Keras+Tensorflow

texts:
I have a Keras model and I want to do some cool visiualizations with it. It's an object recognition network.

So I thought, It would be cool to input a blank image into the network and treat the image as the variable and not the weights, and then train the network to always output an icecream for example.

So I wrote the following code:

#loading the model
model = load_model('model.h5')

#create the input image as a variable
w = tf.Variable(tf.zeros([1,224,224,3]))

#create the flowgraph with the variable input
pred = model.call(inputs=w)

#create the desired output distribution
desired = np.zeros((1000))
desired[928] = 1.0

err = tf.reduce_mean(tf.subtract(pred,desired))
lr = tf.placeholder(dtype=tf.float32, shape=None)

#create an optimizer that can only affect the inital input variable I created
optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(err, var_list=[w])

#train the network
for i in range(0,100):
    _,cost = sess.run([optimizer,err])
    print(cost)

So I thought the code would work well, but the cost literally doesn't change. It stays in place as if it's entirely unaffected.


texts:
Given a very large population, is there an accepted recall estimation method to use when one is validating a trained model against the population?

In my case, a trained model was tested on a disjoint data set, so it has test precision and recall values, but the test data is an extremely small fraction the population.

When the model is run against 'live' data, and when I look at randomly sampled results, I observe an overfit. I want to estimate how bad the situation is, so it makes sense to estimate precision and recall estimates in the population.

I can think of a way to do this for precision calculations, as one can randomly select from the classifier's positive outcomes at various times to calculate a sample mean and standard deviation for precision. However, it does not appear to be easy to estimate recall in the case of drastic class imbalance, and samples drawn from the population keep coming up with zero positive-class data points.

Has anyone come across a situation like this? I would appreciate any pointers or reference materials on the subject.


texts: How to get accurate estimates on Neural Networks Hessian?

texts: I need to get not only accurate estimates on the neural network output itself but also on its second order derivatives in order to use the NN for optimization problems. With Adam optimizer I can't get these accurate estimates. It seems that the neural network can't get good estimates on its second order derivative beyond the domain it were trained getting erroneous erroneous in some particular data input. Is there any optimizer or regularization technique that can give me a better accuracy on the neural network derivative i.e get more information on it?


texts: How do you search a high dimensional for the global maxima using as few samples as possible?

texts:
Suppose the value at any point in the space is defined by Y = f(x1, x2 .. xk). For simplicity, we can assume that x takes only binary values. Which means that we have a total of 2^k possible values. I want to find the point in this k-dimensional space such that Y at this point is the highest.

The constraint lies in the cost of calculating Y. Because it is very 'expensive' to calculate Y, I want to do it as infrequently as possible. That being said, I have the ability to measure Y for any point in this k-dimensional space. I'm looking for a way to use only a small subset of values n, where n << 2^k and use it build a prediction model that extrapolates to the entire space

I've been able to do this by starting off with a set of randomly selected n points and using them to construct a regression tree. The prediction error is acceptable. But the size of n is still quite high.

Is there a more intelligent way to choose my n samples? What are some alternative approaches to extrapolate n measurements into the entire feature space?

If I'm willing to settle for the local maxima, how can I find it? Seems like a straight forward problem in linear algebra. But it's been ages and I could really use some help. TIA!


texts: How to run unmodified Python program on GPU servers with scheduled GPUs?

texts:
Say I have one server with 10 GPUs. I have a python program which detects available GPU and use all of them.
I have a couple of users who will run python (Machine learning or data mining) programs and use GPU.
I initially thought to use Hadoop, as I find Yarn is good at managing resources, including GPU, and YARN has certain scheduling strategies, like fair, FIFO, capacity.
I don't like hard-coded rules, eg. user1 can only use gpu1, user2 can only use gpu2.
I later find Hadoop seems to require the program written in map-reduce pattern, but my requirement is to run unmodified code as we run on Windows or local desktop, or modify as little as possible.
Which knowledge should I look at for running and scheduling python programs on a machine with multiple GPUs?


texts: Multi-arm bandit problem for bernoulli reward distribution

texts:
Suppose in the multi-arm bandit problem I know my rewards are distributed as $0$ or $1$ i.e according to a Bernoulli distribution rather than the condition that they lie in the range $[0,1]$. Does anyone know if we can do better with our confidence bounds with this restricted condition?

In particular how does the upper confidence bound algorithm change and what is the corresponding upper bound on the expected regret?

Can someone provide links to a paper or a set of notes on this particular problem?


texts:
Im fairly new to data science and trying to see if a type of classification exists for my needs.

I understand that a classification into 2 categories will look something like this:

You have 2 desired outcomes and you try to build a model that classifies as 0 or 1. If these models are not 100% accurate then you will:

a) Miss some true values (outside edges of circles)
b) Get some of the wrong values in each category (overlaps between circles)

However, I am looking for something more like this:

In this case, I want to predict only 1, and I dont mind if some 0s are included but want to make sure that as many as possible 1s are predicted.

In my mind, this is effectively just widening the orange circle (the classification of 1s) in the picture.
How can i achieve this?


texts: Are there neural networks packages that use complex numbers?

texts:
Can you build complex (in terms of complex numbers) neural networks in Keras or Tensorflow or something similar?

This would mean the inputs, weights, activation functions, and outputs would all potentially use complex numbers. I know this can be done in theory, but all the software packages

I have come across seem to assume only real numbers are used for the weights and inputs.


texts: How to Build Mobile Application for Image Recognition?

texts:
I want to write an application on (Android) phone for image recognition. The (Keras) model itself is written and trained on a desktop machine and works satisfactorily with standard images. However, I have no experience with app programing so I have no clue how to write an app itself which could utilize the model.

The purpose is to use the mobile camera for seeking the object; have a big button to shot an image; have the image classified and display the output label as text. No need for any sophisticated outlook. I can write programs in Python.

Is there any handy framework that could be used for this purpose?


texts: Machine learning graphics card - half precision performance

texts:
Are there any consumer level graphics cards (i.e. less than, say, $1500) that have decent half precision (fp16) support?

The latest Titan X was rumoured to have native fp16 support but that didn't turn out to be true. Are there any others that support fp16, maybe on older architectures (but are still semi-decent for deep learning)?


texts: The meaning of γt−t0 in Reinforcement learning with pytorch

texts:
When reading pytorch tutorial:

  Our aim will be to train a policy that tries to maximize the discounted, cumulative reward Rt0=∑∞t=t0γt−t0rt, where Rt0 is also known as the return

I know γ is the discount factor, but I am not sure that what t-t0 ofγt−t0 mean?

Thank you.


texts: Isn't graph embedding a step back from non-euclidean space?

texts: As I understand, we use graph embedding to make a euclidean representation of non-euclidean structure - graph. Does it mean that conceptually we just take a step back to, may be, more complex, but still grid processing?


texts:
I have a pet project to figure out the birth year of a significant person in history.

I'm collecting a lot of data on other people with similar status during that time period. I have data such as education length, year married, year of child bearing, information about siblings and age difference between each child, marriages, etc...

The age of this person is disputed between two years, one making the person very old, and the other making the person young. I want to regress the persons age. My first idea was to draw gaussians over each variable and see if one is more likely to be an outlier than the other.

What way would you tackle this problem?


texts: How to properly interpret the train and val loss?

texts:
I am currently doing some research in neural networks for regression problems. Following some plots of the train and validation loss of different models. The blue line is the train loss and the orange line the validation loss.

2x 1000 Dense layers:

It seems like the model is not complex enough to model the validation data. Is that right?

2x 2000 Dense layers:

The same as for the model before, but a little bit better. So a too complex dataset for the model. Is that true?

2x 1000 Recurrent layers:

Alot better then the models before, but maybe still not enough epochs for the data or not complex enough model. True?

2x 2000 Recurrent layers:

A little  bit better, however there are still validation data points that are not modelled properly. Do you have any suggestions how to get a better model?


texts: Loss function in GradientBoostingRegressor

texts:
I was looking at the Scikit-Learn documentation for GradientBoostingRegressor.
Here it says that we can use 'ls' as a loss function which is least squares regression. But i am confused since least squares regression is a method to minimize the SSE loss function.
So shouldn't they mention SSE here?


texts: Stacking when the the target variable is categorical?

texts:
I'm trying to use stacking when predicting for the infamous Iris dataset. Also, I'd like to build to stacked classifier by myself which means I don't want to use mlxtend because it's too "easy" and out of the box. I'm using python 3 and working on windows 10.

I'm not having any issue choosing algorithms or during cross-validation but when I try to stack the base learners ( gaussian and multinomial naive bayes, logistic regression) which gave me a string answer : the species of iris ; with the meta-learner (gradient boosting classifier), I can't seem to have a string as an answer, only a probability and it my accuracy score is much lower than any of the individual base learner.

Could anyone direct me at some code and/or explanations as to what I should be doing instead?

Thanks a lot!


texts: How many dimensions should matrix be reduced to with PCA?

texts:
I want to reduce the dimensions of my dataset to decrease the complexity without losing too much information.

But I am not sure how many dimensions I should reduce to, so that I don't lose too much information.

Do we have any direct attribute/function to use for such task?

Thanks in advance.


texts: Help deciding the best statistic tool for my data

texts:
I'm not from the data science area, and I need to perform some statistical analysis on my data, but I don't know which analytical tools I should use.
Number of variables - 6
Independent Variables 3 - (one is scalar (e.g., 2,4,6,8,10) and the other two are nominal (e.g., A,B,C and Lote 1, Lote 2, Lote 3, Lote 4, Lote 5)
Dependent Variables - 3 (all scalar variables and dependent on all three independent variables, they are basically performance metrics)
One question I would like to answer is how the dependent variables behave when the scalar independent variable increases, taking into account the nominal variables. The picture below shows a sample of my data.


texts:
Is it possible to apply a custom loss function in a regression model (or any other algorithm for predicting continuous variable) ? I'm working on a stock market prediction model and I need to maximize the following loss function: if [predicted] < [actual]  then [predicted] else [-actual]. Would that be possible ?
Thanks


texts: Preserving labels in latent space clustering

texts:
I generated a latent space from a VAE model for my dataset which consists of images depicting different font styles.  The objective is to reduce the dimensionality of this latent space in order to visualize the clustering of the font styles. However, I'm not sure which font type corresponds to which row in this latent space. Does anyone know how to preserve labels while performing clustering? Here is the method being called when generating the latent space, which is a module I borrowed from here.

def predict_embed_samples_into_latent(self, x, batch_size=None, verbose=0, steps=None):

    return self.encoder.predict(x, batch_size, verbose, steps)[2]


texts: Stacking - Appropriate base and meta models

texts:
When implementing stacking for model building and prediction (For example using sklearn's  StackingRegressor function) what is the appropriate choice of models for the base models and final meta model?
Should weak/linear models be used as the base models and an ensemble model as the final meta model (For example: Lasso, Ridge and ElasticNet as base models, and XGBoost as a meta model). Or should non-linear/ensemble models be used as base models and linear regression as the final meta model (For example, XGBoot, Random Forest, LightGBM as the base models, and Ridge as the final meta model)


texts:
I am writing a musical transcription system with a RNN (LSTM).

Input: 1 vector of features per timestep (about 40 timesteps in a second)
Output: 1 binary vector of notes per timestep (dimension=36) (1 is on, 0 if off).
Model : LSTM(512) + LSTM(256) + Dense(36, activation='sigmoid')

I am currently training on monophonic data (i.e. my output has at most one 1 per timestep).

The problem is that because of the sparsity the best strategy to use is to always return 0.
I tried the loss functions 'mean_squared_error', 'mean_absolute_error', 'binary_crossentropy', 'cosine_proximity' and one custom I wrote with keras :

K.sum(K.abs(y_pred - y_true), axis=-1) * K.mean(y_pred, axis=-1) / K.maximum(K.mean(y_pred * y_true, axis=-1), K.epsilon())

All those function led with sufficient training to the always zero output.

I can either change my loss function or my encoding, but the problem is that I need to support polyphonic data, i.e. when there is more than one class to select.

Or can I train as many LSTM as there are notes (88 for a piano keyboard), and each one detects if its note is played but I think this is pretty much equivalent, isn't it ?

I think my data is pretty much equivalent to what keras calls categorical data (binary matrix).

Currently I have only 36 classes, and 24 are always zero.


texts:
Can we calculate PSI value if there is no training data?
if we can please suggest me the steps how  can we do it.


texts: Scikit-learn SelectKBest is picking up obviously unwanted Features

texts:
Dataset
Dataset Summary: Bank Loan (classification) problem
Problem Summary:

I am exploring ways to simplify EDA Process (Exploratory Data Analysis) of finding the best fit variables
I came across SelectKBest from Scikit Package
The implementation went fine except some variables it returned me are obviously not going be a good factor (like primary keys in the dataset)
Is there a problem in the implementation? or is the package supposed to behave in that manner?

import numpy
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_classif
from sklearn.preprocessing import LabelEncoder

# My internal code to read the data file
from src.api.data import LoadDataStore

# Preping Data
raw = LoadDataStore.get_raw()
x_raw = raw.drop(["default_ind", "issue_d"], axis=1)
y_raw = raw[["default_ind"]].values.ravel()

# NA and Encoding
for num_var in x_raw.select_dtypes(include=[numpy.float64]).columns.values:
    x_raw[num_var] = x_raw[num_var].fillna(-1)
encoder = LabelEncoder()
for cat_var in x_raw.select_dtypes(include=[numpy.object]).columns.values:
    x_raw[cat_var] = x_raw[cat_var].fillna("NA")
    x_raw[cat_var] = encoder.fit_transform(x_raw[cat_var])

# Main Part of this problem
test = SelectKBest(score_func=f_classif, k=15)
fit = test.fit(x_raw, y_raw)
ok_var = []
not_var = []
for flag, var in zip(fit.get_support(), x_raw.columns.values):
    if flag:
        ok_var.append(var)
    else:
        not_var.append(var)

ok_var
['id', 'member_id', 'int_rate', 'grade', 'sub_grade', 'desc', 'title', 'initial_list_status', 'out_prncp', 'out_prncp_inv', 'total_rec_late_fee', 'recoveries', 'collection_recovery_fee', 'last_pymnt_d', 'next_pymnt_d']

not_var
['loan_amnt', 'funded_amnt', 'funded_amnt_inv', 'term', 'installment', 'emp_title', 'emp_length', 'home_ownership', 'annual_inc', 'verification_status', 'pymnt_plan', 'purpose', 'zip_code', 'addr_state', 'dti', 'delinq_2yrs', 'earliest_cr_line', 'inq_last_6mths', 'mths_since_last_delinq', 'mths_since_last_record', 'open_acc', 'pub_rec', 'revol_bal', 'revol_util', 'total_acc', 'total_pymnt', 'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int', 'last_pymnt_amnt', 'last_credit_pull_d', 'collections_12_mths_ex_med', 'mths_since_last_major_derog', 'policy_code', 'application_type', 'annual_inc_joint', 'dti_joint', 'verification_status_joint', 'acc_now_delinq', 'tot_coll_amt', 'tot_cur_bal', 'open_acc_6m', 'open_il_6m', 'open_il_12m', 'open_il_24m', 'mths_since_rcnt_il', 'total_bal_il', 'il_util', 'open_rv_12m', 'open_rv_24m', 'max_bal_bc', 'all_util', 'total_rev_hi_lim', 'inq_fi', 'total_cu_tl', 'inq_last_12m']

Its clear id, member_id should NOT belong to the best features list! any idea what I am doing wrong?

Edit: Did more digging and the reply by @Icrmorin is right. (its a kaggle dataset, so will not know why) but here is the box plot for id


texts: How to deploy a deep learning model in Flask (Python)?

texts: I am still an amateur with respect to deploying deep learning models as a flask app. I want to deploy a VQA model based on MMF (Pythia) in a flask app. Github repository for reference. Can someone explain in brief as to how to go about it? Thanks in advance.


texts: Entropy applied to policy gradient prevent our agent from being stuck in the local minimum?

texts:
In the information theory, the entropy is a measure of uncertainty in
  some system. Being applied to agent policy, entropy shows how much the
  agent is uncertain about which action to make. In math notation,
  entropy of the policy is defined  as : $$H(\pi) = -\sum \pi(a|s) \log
 \pi(a|s)$$ The value of entropy is always greater than zero and has a
  single maximum when the policy is uniform. In other words, all actions
  have the same probability. Entropy becomes minimal when our policy has
  1 for some action and 0 for all others, which means that the agent is
  absolutely sure what to do. To prevent our agent from being stuck in
  the local minimum, we are subtracting the entropy from the loss
  function, punishing the agent for being too certain about the action
  to take.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 254.

In code, it might look like :

 optimizer.zero_grad()
 logits= PG_network(batch_states_ts)
 log_prob = F.log_softmax(logits, dim=1)
 log_prob_actions = batch_scales_ts * log_prob[range(params["batch_size"]), batch_actions_ts]
 loss_policy = -log_prob_actions.mean()

 prob = F.softmax(logits, dim=1)
 entropy = -(prob * log_prob).sum(dim=1).mean()
 entropy_loss = params["entropy_beta"] * entropy
 loss = loss_policy - entropy_loss

I know that a disadvantage of using policy gradient is our agent can be stuck at a local minimum. Can you explain mathematically why subtracting the entropy from our policy will  prevent our agent from being stuck in the local minimum ?


texts: tf.keras.layers.Reshape() is significantly slowing down execution speed

texts:
I have the following toy example to illustrate the problem I am having. The encoder is a pretrained Xception model (its forward step takes negligible time). The strange thing is that I get a huge speed boost (more than twice as fast) when I simply use tf.random.uniform([32, 100, 2048]) when compared to getting the output from the encoder. This makes no sense since the size and type of the tensors are identical and therefore tensorflow should not be retracing. I would also like to mention that I do run each of the blocks of code below in different cells on jupyter notebook and not as one single block. I am really not sure how to proceed, so any help would be immensely appreciated!
@tf.function
def get_image_features(imgs):
    return tf.keras.layers.Reshape((100, 2048))(encoder(imgs))

@tf.function
def step(img_features, hidden_state, input_words):

    preds, hidden_state, _ = generator([img_features, input_words,
                                     hidden_state], training=False)
    return hidden_state, preds

imgs = tf.random.uniform([32, 299, 299, 3])
img_feats = get_image_features(imgs)
hidden_state = tf.random.uniform([32, 512])
input_words = tf.random.uniform([32, 1, 1], maxval=10, minval=0, dtype=tf.int32)

# timing part I am interested in
s = time.time()
hidden_state, preds = step(img_feats, hidden_state, input_words)
input_words = tf.random.categorical(tf.math.log(preds), 1)
print(time.time() - s)


texts: Anomaly detection in cooling process data without exact labels

texts:
I have a data set where I look at the cooling of a process. The starting temperature may vary between 580 and 180 degrees. I know that at some point the cooling system failed (see examples in the plot) and I tried to predict this failure.

As the data is completely unlabeled (the failure itself had to be detected manually, too), I ran unsupervised algorithms on it.

To give an idea about the data, here is a plot including the results of one example of unsupervised learning. The top plot show two parameters that I extracted for each process, which varies in duration. The colors which resemble the position in the bottom plot result from an unsupervised learning algorithm using only 30 minutes of data, append the last value as often as possible if less than 30 minutes were available.

The example results of three clusters would look like this (to give an idea about the shape of the data)

Up to know, I have changed the algorithms between AgglomerativeClustering, Birch and DBSCAN with various parameters.

Almost all clusters indicated that the failing of the cooling system occured by showing new or unusal clusters during the time the problems occured (see here clusters 4 and 7), but none of them showed a similar behaviour predicting the failure.

This led me to the following quesions:

Would the same procedure using (obviously unsupervised) neural networks be an option and if so, how (I am working in python)?
What other approaches could there be to handling this problem?
At what point can I savely say that predicting the failure (not merely detecting it) is not contained in the data? I assume this is true as the processes are very different and any minor change in cooling due to a system that is about to fail would remain undiscovered as other parameters have a much bigger influence, but I'd very much appreciate any opinion on this.

Edit: Sample Data

   [[565. , 565. , 564. , 555. , 542. , 527. , 511.5, 496. , 460. ,
    434. , 413. , 393. , 376. , 359. , 344. , 329. , 315. , 303. ,
    291. , 279. , 268. , 258. , 249. , 239. , 231. , 222. , 214. ,
    207. , 200. , 193. ], #would go on with 188,...
   [540. , 540. , 539. , 531. , 520. , 508. , 496. , 494. , 456. ,
    436. , 420. , 404. , 390. , 377. , 364. , 353. , 341. , 331. ,
    321. , 312. , 303. , 295. , 286. , 279. , 271. , 263. , 263. ,
    263. , 263. , 263. ], #the process was ended too early, 263 got repeated to match the format
   [530. , 530. , 529. , 520. , 509. , 495. , 455. , 427. , 405. ,
    384. , 365. , 348. , 332. , 317. , 302. , 288. , 275. , 263. ,
    252. , 242. , 232. , 222. , 213. , 204. , 196. , 188. , 181. ,
    174. , 168. , 161. ], #would go on with 154
   [181. , 174. , 165. , 158. , 152. , 147. , 146. , 141. , 137. ,
    132. , 128. , 125. , 121. , 118. , 114. , 111. , 109. , 106. ,
    103. , 101. ,  98. ,  96. ,  94. ,  92. ,  91. ,  89. ,  87. ,
     85. ,  84. ,  82. ]] #would go on with 81


texts: How would I apply anomaly detection to time series data in LSTM?

texts:
I am using a LSTM RNN in Python and have successfully completed the prediction phase. My ultimate goal is anomaly detection. I'm hoping to have something like what you could see on Facebook Prophet, with anomalies marked as black dots below:

I've read loads of articles about how to classify with text/sequence data but there's not much on univariate time series data- only timestamps and randomly generated values with a few anomalies. One answer linked to the following method:

  Anomaly detection is done by using the prediction errors as anomaly indicators.Prediction error is the difference between prediction made at time t−1 and the in-put value received at time t. The prediction errors from training data are modeled using a Gaussian distribution. The parameters of the Gaussian, mean and variance, are computed using maximum likelihood estimation (MLE). On new data, the log probability densities (PDs) of errors are calculated and used as anomaly scores: with lower values indicating a greater likelihood of the observation being an anomaly. A validation set containing both normal data and anomalies is used to set a threshold on log PD values that can separate anomalies from normal observations and incur as few false positives as possible. A separate test set is used to evaluate the model.
  Source

I understand overall why this method would be used but I have little idea of how to implement it with a Sequential Keras model. Does anyone have or know of some example code? Bonus if there is a way to visualize this because I don't know how I would go about that side of things either.

Please help and thank you!


texts: Semantic analysis score as input to LSTM model for improving stock price prediction accuracy

texts: I have created a univariate LSTM model that is predicting value of Open Price based on last 5 years opening price of a particular stock. I'm getting a decent accuracy. Now, I want to do sentiment analysis of the news data of the last 5 years of that stock and combine that score with my existing LSTM model in order to improve the predicted values of LSTM model. Does anyone has an idea whether this implementation is possible and if yes how can I proceed. Thanks in advance :)


texts: How do you predict a continuous variable when all your independent variables are categorical

texts:
I am new to data science and ML. Recently I have been given a sales dataset which contains weekly sales of a fashion brand. It has information about the product like category(t shirt, polo shirt, cotton shirts, briefs, jeans, etc.), gender (male, female, unisex), partner stores ( multiple retail outlets), the color of the apparel (some 150 color codes in hex format), MRP, Sold MRP and quantity sold. The data is for two years timeframe.
I added a new column called discount which is deriver from sold mrp/qty and mrp.
Can anyone suggest any kind of predictive modelling scenario for the above kind of data? I have already done time series forecasting and clustering based on high-performing partner stores. I tried Regression but how do you perform regression when most of the independent variables are categorical in nature ?
Any help would be much appreciated.


texts: records with perfect correlation to the answer. Drop or Keep?

texts: I have about 1000 records (5 numeric, 5 categorical vars) and about 25 of them have something in 5-level categorical variable that just gives away answer. It's just too obvious and I'm not worried about over-fitting. I'm using methods are xgboost or catboost and wondering if keeping these hurt or improve fitting of other variables. When I just keep it, once in a while it gets 1 out of 25 in test set wrong and I just override that. I'm wondering if it's better to just leave those out and label 1 manually.


texts: spikes in the cost vs iteration graph using ReLU activation function

texts:
The cost vs iteration graph while using the ReLU activation function has number of spikes

The accuracy on MNIST dataset (both train and test) is around 95%, also on using sigmoid as an activation function I get a smooth downward sloping curve so I think the implementation is correct. Are these spikes expected for ReLU? How would you explain this property?


texts: How to detect the match precision of OneVsRestClassifier

texts:
I've improved my text classification to topic module, from simple word2vec to piped tfidf and OneVsRestClassifier (using sklearn). It does improve the classification but with word2vec I was able to calculate the match percentage for each topic and with OneVsRestClassifier i get a match or no match to a specific topic. Is there a way to see with OneVsRestClassifier what was the percentage of the classification?

P.S.
I am not talking about evaluating the performance of the training but the actual real time matching percentage.


texts: Loss function for group aggregated prediction results

texts:
I have a set of individual level data x11, x12, x13... that is grouped into each own groups g1, g2, g3. I'm trying to create a prediction model which predicts y1, y2, y3...
The simplest method would be to implement a regular loss function such as MSE on the individual prediction results. However, in my real world application, I don't actually care too much about the individual accuracy, but rather the aggregated accuracy of g1, g2, g3 being ĝ1 = y1 + y2 + y3... and so forth.
This Medium article shows that using an individual level loss function does not equate to optimizing a group target accuracy. However, it only talks about how to treat the complete data set as a group, and not the data also being segmented into multiple groups.
One method I've thought about is to train a model for each group, and aggregate the results through some sort of ensembling to create my final model, however I feel that this doesn't seem quite correct, and my individual groups also don't have a very large amount of data associated with each, so the individual model accuracy is likely to be quite low / highly biased.
I am also unable to pivot the the individual data into a single group feature, and it is also difficult to generated aggregated group features from them as my groups have some quite big variations between them. One group may have 100 individual data, another may have 1000.
In short, the following diagram is what I'm trying to achieve.
┌─ ───────── Model ────────────┐      ┌ Actual ┐
a1, a2, a3, a4, a5... => ya  ─┐
b1, b2, b3, b4, b5... => yb   |
c1, c2, c3, c4, c5... => yc   ĝ1         g1
d1, d2, d3, d4, d5... => yd   |
e1, e2, e3, e4, e5... => ye  ─┘
f1, f2, f3, f4, f5... => yf  ─┐    ==>
g1, g2, g3, g4, g5... => yg   |
h1, h2, h3, h4, h5... => yh   ĝ2         g2
i1, i2, i3, i4, i5... => yi  ─┘
j1, j2, j3, j4, j5... => yj  ─┐
k1, k2, k3, k4, k5... => yk   ĝ3         g3
l1, l2, l3, l4, l5... => yl  ─┘

Accuracy Score =
  (ĝ1 - g1)² + (ĝ2 - g2)² + (ĝ3 - g3)²
  ─────────────────────────────────────
                    3

Don't care about the actual scoring function used at the group level,
just that it should be on the group level and not individual levels
What would be a good approach to solve this problem? Is there a type of ML model that I should be looking at, or some what to separate/manipulate the data? Or some way to design the scoring function to reflect my needs?


texts: Could the Input shape of the LSTM layer not be a constant?

texts:
I am trying a vanilla LSTM on my dataset.
According to this course; the LSTM layer should be built as follows:
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))

Key in the definition is the shape of the input; that is what the model expects as input for each sample in terms of the number of time steps and the number of features.

In my case, it's a univariate regression also, with "multi-step" though...
Instead of some constant n_steps, I have a different "step" for each Input.
eg:
...
X                                                                                y
[2006-10-11 10:00:00  0.295046,2006-10-21 10:00:00  0.325147,2006-11-11 10:00:00  0.361293] [0,666506515]
[2006-12-01 10:00:00  0.249307,2006-12-11 10:00:00  0.444077] [1,885262136]
[2009-05-11 14:00:00  0.570017,2009-05-21 14:00:00  0.525082,2009-06-01 14:00:00  0.475404,2009-06-11 14:00:00  0.390616] [1,998273253]

Let's assume that fsplit is the list containing the different steps:
What I've tried so far is doing this:
model.add(LSTM(50, activation='relu', input_shape=([i for i in fsplit], 1)))
but it doesn't work of couse
Thanks in advance :)


texts: sklearn.cross_validation.cross_val_score "cv" parameter question

texts:
I was working through a tutorial on the titanic disaster from Kaggle and I'm getting different results depending on the details of how I use cross_validation.cross_val_score.

If I call it like:

scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)

print(scores.mean())

0.801346801347

I get a different set of scores than if I call it like:

kf = KFold(titanic.shape[0], n_folds=3, random_state=1)

scores = cross_validation.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=kf)

print(scores.mean())

0.785634118967

These numbers are close, but different enough to be significant. As far as I understand, both code snippets are asking for a 3-fold cross validation strategy. Can anyone explain what is going on under the hood of the second example which is leading to the slightly lower score?


texts: How to decide the processing power required based on the dataset?

texts:
To train a machine learning model, the computer often needs more processing power. In this case, a powerful CPU is needed, since it is a large data set, it needs more memory, so rather than a CPU, GPU is the solution.

Do we need to decide which processor to use before we proceed? For example, will a 30000 sample data set need this much processing power?

Thanks in advance.

If any part of this question is not clear, please comment it.


texts: Create an RNN on text sources with different lengths

texts:
I want to create an RNN to generate a new text based on many examples of existing texts of a certain format in the training data.
The type of texts in the training data consists of 3 segments, like so:
Example text 1:
[Segment 0, ~20 characters]
[Segment 1, ~200 characters]
[Segment 2, ~400 characters]
It is worth mentioning that the segments are all still alphanumerical, but of varying structure. Segment 1 contains more numbers and Segment 2 has more letters with references to occurrences in Segment 1.
I have training data with many instances of such example texts, but with varying length in each segment.
In most RNN text generation examples I have seen before the training set consists of a single long text which is split into segments of a fixed length for training, and I guess that could be done in my case as well by simply concatenating all of the different examples into one long string. Then dividing that long string into sequence pairs of "input" and "target" strings of a fixed length, say 100 characters and use them for training.
My question is if I can not instead take advantage of the fact that I have the text already split up into segments which resemble what I want to generate, and use each example as a batch for training.
In my opinion feels like it would be easier for the RNN to learn the relations between the different text sequences if it got fed data (input + target string) that was belonging to one example at a time, as opposed to data (input + target) spanning over the entire training set.
Is there any aspect that I am overlooking which would favour simply concatenating all training data into one large string?
Many thanks in advance!


texts: How to mantain the nested structure of a tf.dataset after applying map?

texts:
I'm creating a tf.dataset object containing 2 images as inputs and a mask as target. All of them are 3D in grayscale. After applying a custom map, the shape of the object changes from  ((TensorSpec(shape=(), dtype=tf.string, name=None), TensorSpec(shape=(), dtype=tf.string, name=None)), TensorSpec(shape=(), dtype=tf.string, name=None)) to (TensorSpec(shape=<unknown>, dtype=tf.float32, name=None), TensorSpec(shape=<unknown>, dtype=tf.float32, name=None), TensorSpec(shape=<unknown>, dtype=tf.int32, name=None)), losing the nested structure. When I fit the data, my model throws an error because it only detects one input instead of 2.
Here is what I'm doing:
x, y = get_filenames(train_data_path, img_type='FLAIR')
z = get_filenames(train_data_path, img_type='mask')

path_dataset = tf.data.Dataset.from_tensor_slices((x, y))
mask_dataset = tf.data.Dataset.from_tensor_slices(z)
dataset = tf.data.Dataset.zip((path_dataset, mask_dataset)).shuffle(50).repeat(10)

ds = dataset. \
    map(lambda xx, zz: ((tf.py_function(load, [xx], [tf.float32, tf.float32])),
                        tf.py_function(load_mask, [zz], [tf.int32])),
        num_parallel_calls=tf.data.AUTOTUNE)

ds = ds.map(lambda xx, zz: (tf.py_function(random_crop_flip, [xx, zz],
                                           [tf.float32, tf.float32, tf.int32])),
            num_parallel_calls=tf.data.AUTOTUNE)
ds = ds.batch(2)
ds = ds.prefetch(tf.data.AUTOTUNE)

I can't map separately the images and the masks because they need the same seed for the random cropping and flipping. Is it possible to change the shape after the map so that I can feed it to my 2 input model?
My random_crop_flip function is as follows:
def random_crop_flip(images, mask, width=128, height=128, depth=128):
    img_bl, img_fu = images
    img_bl = img_bl.numpy()
    img_fu = img_fu.numpy()
    mask = mask.numpy()
    x_rand = random.randint(0, img_bl.shape[2] - width)
    y_rand = random.randint(0, img_bl.shape[1] - height)
    z_rand = random.randint(0, img_bl.shape[3] - depth)
    img_bl_f = img_bl[:, y_rand:y_rand + height, x_rand:x_rand + width, z_rand:z_rand + depth, :]
    img_fu_f = img_fu[:, y_rand:y_rand + height, x_rand:x_rand + width, z_rand:z_rand + depth, :]
    mask_f = mask[:, y_rand:y_rand + height, x_rand:x_rand + width, z_rand:z_rand + depth, :]
    flip_x = random.choice([True, False])
    flip_y = random.choice([True, False])
    flip_z = random.choice([True, False])

    if flip_x:
        img_bl_f = np.flip(img_bl_f, axis=2)
        img_fu_f = np.flip(img_fu_f, axis=2)
        mask_f = np.flip(mask_f, axis=2)

    if flip_y:
        img_bl_f = np.flip(img_bl_f, axis=1)
        img_fu_f = np.flip(img_fu_f, axis=1)
        mask_f = np.flip(mask_f, axis=1)

    if flip_z:
        img_bl_f = np.flip(img_bl_f, axis=3)
        img_fu_f = np.flip(img_fu_f, axis=3)
        mask_f = np.flip(mask_f, axis=3)

    return (img_bl_f, img_fu_f), mask_f

The tuple in the output isn't solving my problem. Is it possible to modify the return to get my desired output?


texts: Simple example of Parzen window (kernel density estimation)

texts:
I am confused about the Parzen Window question.

Suppose we have two training data points located at 0.5 and 0.7, and we use 0.3 as its rectangle window width. How do we estimate its probability density?

According to the definition, the probability density is $\frac{k/n}{V}$, where $k$ is the number of patterns, $n$ is the total number of points, and $V$ is the window region.

Therefore, is the density for this question $(1/2)/1$?

Then what if we use a triangle instead of a rectangle?


texts: Is there any named entity reconginition algorithm trained for the french language?

texts: I am trying to implement a utility for my mobile application to perform some actions based on user questions. I need an algorithm to extract named entities from a text string (French grammar). I have used nltk's interface to Stanford's NER models but it works only for English (A subset of other languages is supported but i can't find French). I have also used Polyglot but it seems that it doesn't do the work very well (Maybe the models I am using are not very well trained). I don't know if there is any free REST API that can do NER for the French language or any other algorithm or even an already trained model for nltk/Stanford NER.


texts: Domain adaption vs. heirarchical model - when to use which?

texts:
I know a little bit about domain adaption and also about random effects models, but I'm a little unsure if they're at all compatible. Domain adaptive models (based on ANNs) usually try to find some representation of the data such that the source and target domains looks indistinguishable, while random effects models try train over several domains in such a way as to remove the domain knowledge from the final answer. To me they seem to be trying to do quite similar things but in very different ways.

As an example, let's say that I want to know how the probability that there will be a crime varies with unemployment, temperature and rainfall. You collect some data from 3 cities. Since these cities have different mayors, they have different underlying crime rates so it is wrong to simply pool all the data into a simple classification problem.

In this example I'd be tempted to try either a random effects model, or domain adaption. For instance you could treat the cities each as a domain, and try to transfer to your target city.

Another approach I've thought of, but I haven't really seen written down anywhere is to marginalize over the city paramter. For example you train a model with "city" as a feature, so we have $ p(y | X;city) $. Then when we come to make predictions on some new city we just average over all values of "city" that were used in training. E.g.,

$$ p(y|X) = \sum_{city} { p(y|X;city)\, p(city) } $$

Is this a valid thing to do? I tried it on some test data and the results look ok, I'm just wondering why I've not seen it anywhere. It feels a little bit like training a separate model for each city and just ensembling the results but it has the advantage of having a single model.


texts: I am fairly new to RNNs and Im having trouble setting up the desired output from RNN using Keras library. Each datapoint in my dataset consist of a pattern of labels and timestamp of occurrence of each label and based on the pattern of label I want to predict what the next label in the pattern be. I have developed the model which takes series of label as input and outputs the next label in the series but now I want to input labels as well as their timestamp and receive output the next label as well as the timestamp of its occurrence. How can I achieve this with a single model?


texts: Does keras' model.fit() remember learning rate when called multiple times?

texts:
Let's say I'm using the Adam optimizer, and calling fit() on my model multiple times. What parameters does the fit function remember?

From what I've observed, the loss function/metrics seem to resume from the same value (because the same saved weights are iterated on), but does the value of the learning rate reset? Does it resume from the same value it ended on the last fit() call?


texts: Iterate over multiple dataframe rows at the same time

texts:
I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values.

I need to loop over all dataframes at the same time, and compare all row values with the separate dataframes, and then create another dataframe with the results like so:

comparison: sum(row_values_of_dataframe) - sum(row_values_of_reference). In the example below, the cell df_a_ref_a is equal to $(1 + 2 + 3 + 4) - (5 + 5 + 5 + 5) = -10$

Dataframe A (df_a)

col1 | col 2 | col 3 | col 4
1       2       3       4
2       4       6       8
[...]

Dataframe B (df_b)

col1 | col 2 | col 3 | col 4
10      5       2       1
4       4       6       2
[...]

Reference Dataframe 1 (ref_1)
col1 | col 2 | col 3 | col 4
5       5       5       5
5       5       5       5
[...]

Reference Dataframe 2 (ref_2)
col1 | col 2 | col 3 | col 4
3       3       3       3
3       3       3       3
[...]

Final dataframe should be:

df_a_ref_1 | df_a_ref_2 | df_b_ref_1 | df_b_ref_2 | ....
    -10          -2           -2           6        ....
     0           8            -4           4
[...]

This behaviour resembles zip() function in python.

Thanks in advance,


texts: How can I create a classifier using the feature map of a CNN?

texts:
I intend to make a classifier using the feature map obtained from a CNN. Can someone suggest how I can do this?

Would it work if I first train the CNN using +ve and -ve samples (and hence obtain the weights), and then every time I need to classify an image, I apply the conv and pooling layers to obtain the feature map? The problem I find in this, is that the image I want to classify, may not have a similar feature map, and hence I wouldn't be able to find the distance correctly. As the order of the features may by different in the layer.


texts: Data with similar mean, min and max across all columns. What could I do to build a classifier

texts:
I have a data with the following columns
    col1         col2       col3        col4        label
    7669.533073 7669.533073 7669.695497 7669.922593 1
    7669.533043 7669.533072 7669.695487 7669.922596 0

the mean across all the 50 columns are similar and also the min and maximum.
I am trying to build a classifier and the best model(random forest) is giving me a recall of .55 (doesn't seem so good), could there be anything I am missing in this?
I have thought about normalising the data but there seems to be no need as all columns have a similar mean and std.
Is there any statistics technique I could apply to the data to help get an improved result.
Note the data is from a simulated crypto price and I am trying to predict the price movement (up or down)