texts:
I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains both text and input (maybe Amazon reviews where some users share images of the products). I was thinking of designing a classifier in this way:

Assuming each review is a datapoint, we represent that datapoint with a tensor that contains the concatenated values representing the image and the text preprocessed appropriately.
We then input this data into a deep learning model where the tensor is correctly separated and the image part is fed into a CNN and the text part is fed to a pre-trained BERT model.
The outputs of both the CNN and BERT are then concatenated and fed to a linear classifier with a couple of layers.
The output of this linear classifier is the prediction of the model from which we derive the loss and the entire network is trained end-to-end.

I'm looking for thoughts and feedback on this task and approach. Specifically,

Does this approach make sense and do you think its worth trying or is there a better approach that you can point me to?
One issue I can think for some reviews there might be multiple images. How do I merge these multiple images to a single review to form a datapoint?
Do you know of any datasets/tasks that conforms to this modality?
Do you know of any previous work that has tackled this problem?

To point 4, I did some preliminary research and only came across a very limited number of papers:

Combining textual and visual representations for multimodal author profiling Notebook for PAN at CLEF 2018
Multimodal deep networks for text and image-based document classification

Thanks.


texts: Is the probabilistic cutoff in random forest flexible?

texts:
I fit the random forest to my dataset with a binary target class. I reset the probabilistic cutoff to a much lower value rather than the default 0.5 according to the ROC curve. Then I can improve the sensitivity (recall) but meanwhile sacrificed the precision.

Just wanna confirm that the default 0.5 is not much meaningful and a practical probabilistic cutoff was often derived from ROC curve in practice. Am I on the right track on the application of random forest and other tree based models.


texts: How to deal with exception handling in python?

texts:
Lets say I have a variable city which accepts any city name , like
city='London'

or
city='New York'

It accepts only city name but if user mistakenly specifies city name some different like
city='Londo'

How does should I deal with it means I want to use try catch in it , But How ?


texts:
I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like:
model = Sequential()
model.add(Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', input_shape = training_data.image_shape))
model.add(MaxPool2D(pool_size = (2, 2)))
model.add(Dropout(rate = 0.3))

model.add(Conv2D(filters = 64, kernel_size = (3, 3), activation = 'relu'))
model.add(MaxPool2D(pool_size = (2, 2)))
model.add(Dropout(rate = 0.2))

model.add(Conv2D(filters = 126, kernel_size = (3, 3), activation = 'relu'))
model.add(MaxPool2D(pool_size = (2, 2)))
model.add(Dropout(rate = 0.15))

model.add(Flatten())

model.add(Dense(units = 32, activation = 'relu'))
model.add(Dropout(rate = 0.15))

model.add(Dense(units = 64, activation = 'relu'))
model.add(Dropout(rate = 0.1))

model.add(Dense(units = 42, activation = 'softmax'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

However, the training time takes super long and each epoch takes around 35 minutes to run. The accuracy is also very low and increases very slowly.
My jupyter lab will sometimes stop and have to refresh everything again. So is there a way to train in smaller batches? Or a way to improve the training speed? Any help is appreciated. It is a very huge dataset.
Epoch 1/15
2307/2307 [==============================] - 3999s 2s/step - loss: 3.5377 - accuracy: 0.0687 - val_loss: 3.3247 - val_accuracy: 0.1223
Epoch 2/15
2307/2307 [==============================] - 3764s 2s/step - loss: 3.2884 - accuracy: 0.1239 - val_loss: 3.1065 - val_accuracy: 0.1739
Epoch 3/15
2307/2307 [==============================] - 2204s 955ms/step - loss: 3.1435 - accuracy: 0.1562 - val_loss: 2.9825 - val_accuracy: 0.2069
Epoch 4/15
2307/2307 [==============================] - 2193s 951ms/step - loss: 3.0526 - accuracy: 0.1778 - val_loss: 2.9059 - val_accuracy: 0.2171


texts: Is it always possible for validation accuracy to be as high as training accuracy?

texts:
I have a very small dataset (40 training examples, 10 validation examples, 120 classes) for which I'm getting very high accuracies with a very simple model in Keras (batchnorm, flatten, and dense layers only).

My training accuracy is 94-95% and validation accuracy is 76-78%. I know it's overfitting and I have tried a few things. The data is not images, so I cannot augment the data. I also cannot add data because it's a specific type. I'm using two dropout layers with 0.5 levels, and the architecture is very simple so I don't think I can reduce the architecture complexity. I can paste the model if anyone likes.

My question is: Is there ever a situation where validation accuracy cannot be as high as the training accuracy? Is there a limitation based on the size of the dataset? Or is it ALWAYS possible for validation accuracies to match training accuracies and the network just needs the right parameters?


texts: What algorithm can predict structured outputs of arbitrary size?

texts:
I have a collection of graph objects of variable size (input) which are each paired to another graph of variable size (output). The task is, given an input graph, produce the most likely output graph.

I have been looking at 'structured output' techniques such as SSVM but as far as I can tell, they all act on outputs of fixed size (or at least a size that matches the input size, i.e. sequence tagging). Are there any tools that can map an input to a structured object of arbitrary size?


texts: Keras Model question for Pre trained model extension

texts:
I want to add a few more layers to a Resnet50 model and my question is - do I need to compile it and train it on new data or can I just use it as it is? Will it just give me the Resnet50 results?

Here is what I am trying:

def base_model():
resnet = resnet50.ResNet50(weights="imagenet", include_top=False)
x = resnet.output
x = Conv2D(128, (3, 3), activation='relu',padding='same')(x)
x = Conv2D(128, (3, 3), activation='relu',padding='same')(x)
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
x = Conv2D(256, (3, 3), activation='relu',padding='same')(x)
x = Conv2D(256, (3, 3), activation='relu',padding='same')(x)
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
x = Conv2D(512, (3, 3), activation='relu',padding='same')(x)
x = Conv2D(512, (3, 3), activation='relu',padding='same')(x)
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
x = GlobalAveragePooling2D()(x)
x = Dense(4096, activation='relu')(x)
x = Dropout(0.6)(x)
x = Dense(4096, activation='relu')(x)
x = Dropout(0.6)(x)
x = Lambda(lambda  x_: K.l2_normalize(x,axis=1))(x)
return Model(inputs=resnet.input, outputs=x)

And then use it like this :

enhanced_resnet = base_model()
img = image.load_img(file, target_size=(224, 224))
img = image.img_to_array(img)
x = resnet50.preprocess_input(img)
x = np.array([x])
feature = enhanced_resnet.predict(x)

What I want as return value is the features from the image and not a prediction, as I am using a distance equation to tell the similarity between images.


texts: Linear machine learning algorithms "often" have high bias/low variance?

texts:
In this blog, which explains the meaning of bias and variance in machine learning, there's a line under the heading "Bias-Variance Trade-Off"
which says:

  Parametric or linear machine learning algorithms often have a high bias but a low variance.

I know that there's an "often" in the first sentence, but how can it be true?
If a linear ML algorithm has high bias how can we expect it to have low variance?


texts: Making an ensemble model for high F1 score

texts:
I presently have 2 algorithms that have a numerical output. Using a threshold of 0.9, I get the classification output. Let's say they are:

P (high precision, low recall)
R (high recall, low precision)

Individually, they have poor F-1 scores. Is the naive way of creating a classifier C as:
C(*) = x.P(*) + (1-x).R(*)
And optimizing for x and threshold a good approach to improve the F-1 score?
Or is there some alternate approach I must try.
Note: I can't vary functions P() and R(). Their outputs are provided as a black-box function.


texts: Data system that manages aggregates over time intervals

texts:
I am looking to know if there is a data system that handles the following use case.
To keep it simple, the data is a set of homogeneous enties E.
E contains named numeric properties that the app code increments as the case may be over the life cycle of the application.
I will want to query the state of E for a set of time intervals. To keep it simple, let's say this is today, this month, this year and lifetime. I don't want to use a time series DB because I do not want the calculations done at query time, no matter how efficient.
I just want Application code to increment the named values of E. But I don't want to manage the timeframe slices.
Here is how I see this working. E would be modeled by, say, a json document.
E(today) is a document keyed by the DATE OF TODAY IN TIMEZONE OF E = T(today)
App code increments properties in E(today) only.
E(today) is set to expire at the end of T(today). Data system manages document expiration. On Expiry, the system increments E(month) according to the values of E(today).
E(month) is set to expire at the end of T(month). On Expiry, the system increments E(year) according the values of E(month).
E(year) is set to expire at the end of T(year). On Expiry, the system increments E(lifetime) according to the values of E(year).
The app can query E for (today, month, year, lifetime), where E(today|month|year|lifetime) are all documents.
today = E(today)
month = E(today + month)
Year = E(today + month + year)
Lifetime = E(today + month + year + lifetime)
At any given time no more than 4 aggregate documents represent E.
The query cost is no more than the cost of tallying up the properties for the set of E documents in the interval.
I know that we can build this model using a document store that provides managed document expiration and eventing. But implementing this model using these systems is rife with durability and safety concerns.
I am primarily looking to avoid managing rolling the documents over. I want to increment the counters once during the app lifetime, and have the system manage the temporal aspects of the aggregations on it own.


texts: MR images segmentation for feature extraction

texts:
I have datasets of brain MR images with tumours, the tumours are already selected manually by a physicist using Image J.

I have read about segmentation, but I still couldn't understand how do they extract features from a segmented image.

should the images have only the tumor with a black background as shown in the below images, so the feature extraction will be processed on the whole image?
or do they extract features only on the region of interest using overlay, or layer that specifies the ROI?

and would Discrete wavelet transform DWT descriptor be a good choice for a descriptor?


texts: Why is an activation function notated as "g"?

texts:
In many cases an activation function is notated as g (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as sigmoid.

However, where does this convention come from? And for what reason did g start to be used?


texts: What can be the cause of a sudden explosion in the loss when training a CNN (Deeplab)

texts:
I am training the following deeplab CNN:
https://github.com/tensorflow/models/tree/master/research/deeplab

During training I see the following loss:

The first 50k steps of the training the loss is quite stable and low, and suddenly it starts to exponentially explode. I wonder how this can happen. Of course there are many reasons a loss can increase, such as a too high learning rate. But what I do not understand is the following:

I use a batch size of 16 and I have 24k images, so 24k/16=1500 steps are used for a full pass on the train data
Only after 50k steps the loss starts exploding, before that it is remarkably stable.
So around the 34th iteration through my train set the loss starts to increase all of a sudden. Why only now? How can it be stable for so long and suddenly increase sharply?


texts: Do I classify all types if they are mutually exclusive

texts:
I am trying to classify an image that can represents 3 states. Up, down or Middle.

If the image does NOT represent Up or Down, then it is by default Middle.

Should I train my CNN with a dataset including all three, or just Up and Down?

Which would this make classification more or less accurate?


texts: From regression neural network to generative one

texts: I made a neural network with word embedding, RNN and some dense layers to predict the score of computer game reviews based on the title of the game. I used Keras in Python. I was wondering if it would be somehow possible to transform my model into a generative one that would produce titles with high (or low) predicted score. It could be possible to do brute forcing and run all combinations through my network but maybe there is a smarter solution.


texts: How to prepare training data for deep learning models

texts: I am working on a project which involves the application of deep learning models. I have collected training data. In collected images, I have more than one object in interest. I am not very clear how to label these images. Should I crop and then label or just feed as it is? I would appreciate any kind of help. In the attached image, I have marked two objects.


texts: What to do if my target variable is column of lists?

texts:
How I can transform my target variable(Y)?

As it is list, I cann`t use it for fitting model, because I must use integers for fitting.


texts: Batch norm: why the initial normalization?

texts:
I'm a beginner in NNs and the first thing I don't understand with batch norm is the following 2 steps:

First we normalize the batch data on a z parameter to Mu=0, sigma^2=1
Then we change z via the coefficients of Mu, sigma^2 (usu. alpha, beta) by updating them as learnable parameters.

I don't understand why the first step is necessary if we change the distribution in the second step anyway. Could someone explain please?


texts: TypeError: object of type 'int' has no len(), LogisticRegression()

texts:
I'm trying to fit my Logistic Regression model, but I'm running into an error that I don't understand. Looked around and haven't found a straight answer.

Shape of independent features (X): (495,30)

Shape of dependent feature (y): (495,)

The dependent feature holds binary values (0,1) and the values are currently of type "int". This is probably where the problem is occurring, but I don't understand what I need to do to fix the issue.

This is the error I'm getting when I try fitting to sklearn LogisticRegression():

c_param = [1,10,100,1000]

rand_grid = {'kbest__k':list(range(5,10)),
            'log__C':c_param,
            'log__random_state':42}

log_rand = RandomizedSearchCV(pipe_opt('log',LogisticRegression()),rand_grid,n_iter=100,cv=3,
                              random_state=42,n_jobs=-1)
log_rand.fit(X_train,y_train)


texts: hidden state of each sequence of mini-batch

texts:
I am new to Pytorch and trying to implement a lstm character level seq2seq model.
What I am trying to do is:
Each sequence is a list of the characters of a particular word and several words will create a minibatch, which also represent a sentence. Now, in my understanding for each sequence (list of character embedding, in my case), there will be a final hidden state. So, if there is two character sequence (two words), there will be two hidden state each representing a word.
I am not even considering the variable-length sequence. I also don't understand why it should be a problem if the sequence length is variable. Should not the lstm loop until there are elements in each particular sequence? The number of iteration should not be static, right?
Here is my code I tried:

character_embedding = nn.Embedding(17,5)
#LSTM with input embedding dimention 5, and expected hidden state dimention 3
lstm = nn.LSTM(5,3)
#each vector is a word and there are two words with same number of character
char_embeds=character_embedding(torch.tensor([[1,2,3,4,5],[4,5,6,7,8]]))
#out will contain all the hidden states for each character and hidden sould contain final hidden state for each sequence
out, hidden=lstm(char_embeds)
print("char_embeds: ")
print(char_embeds)
print("hidden: ")
print(hidden[0])

Output:

char_embeds:
tensor([[[ 1.0157, -0.2197,  1.6615, -1.2916, -0.6116],
         [ 0.5630, -0.9618,  0.7287, -0.5727,  1.6796],
         [ 0.9902, -0.5408,  0.9785, -1.1090,  1.1126],
         [ 0.7472,  0.0440,  1.0629, -0.7375,  0.0828],
         [ 0.6632, -0.4523,  0.5051,  2.6031,  0.3798]],

        [[ 0.7472,  0.0440,  1.0629, -0.7375,  0.0828],
         [ 0.6632, -0.4523,  0.5051,  2.6031,  0.3798],
         [-0.6522, -3.2626,  0.7967, -1.0322,  0.4667],
         [-0.5086,  0.5142, -0.7141, -1.5352,  0.4177],
         [-0.0582,  1.3398, -0.2829,  0.1392,  1.0709]]],
       grad_fn=<EmbeddingBackward>)
hidden:
tensor([[[-0.2774,  0.0724, -0.4297],
         [-0.4580,  0.1563, -0.5811],
         [-0.5492, -0.2314,  0.3473],
         [-0.0772,  0.2474, -0.1026],
         [-0.1042,  0.4394, -0.3582]]], grad_fn=<StackBackward>)

Here, I would expect two hidden states, as there are two sequences. But I am getting 5 hidden states. What is that? What I am missing?

The second question is, Why can't LSTM can not handle variable-length sequences?


texts:
Has anyone defined a spline function (i.e. defineFuction) in PMML?

There are quite a few parameters that need to be defined, with a fairly lengthy math. For example, for a predictor with a simple 3-knot restrictive cubic spline, I have seven parameters (including two coefficients, three knots, one reference point and one predictor name). The math includes a bit of exponential, max statements, etc.

For that matter, is there a library of functions other than those described on the PMML website documentation? It seems like it would be helpful to have a library of descriptions for even relatively simple functions like polynomials.


texts: Running a query in R after establishing dbconnect

texts:
I do not seem to figure out what is wrong it the following statment. The connection to the DWH is establishe but the query statement in R seems not to work, with the following error :

LR=dbGetQuery(con, "select id as ID,
       date_c."Professional_Status" as Prof_Status,
       case when talk_sec >= 5 then 1 else 0 end as Established_Connection
from id_collect as id_c
left join date_conncet as date_c on id_c.date=date_c.date
where date::date = '2018-01-19' and country = 'IT' and type = 'shop'
              and 'district' = 'South' and interaction is true")

This piece of query runs in the SQL editor (using Postgresql) but in R I get the error meassage: Error: unexpected symbol in "  or unexpected numeric constant. I do use the following:

install.packages("RPostgreSQL")
install.packages("DBI")
library(RPostgreSQL)

I believe it is not about the syntax but the quote I use, whats the proper quote ethiquette in R when using numeric vs. categorical values? The whole select statment must begin and end with "select ...." correct?

Also to add how to establish a connection to your DTW (add your DWH infos):

con <- dbConnect(RPostgreSQL::PostgreSQL(),
                 dbname="DBNAME",
                 host= "HOSTNAME",
                 port=0000,
                 user="youruser",
                 password="yourpassword")


texts: Streaming sequence detection (Binary Classification) LSTM/GRU

texts:
I am currently trying to implement a model which can detect a specific sequence according to the training data which looks like the following:
x_train = [...] # shape: (timesteps, features)
y_train = [[0], [0], [0], [0], [1]]  # shape: (timesteps, 1)

The model:
    rnn = GRU(units=32,
        activation="tanh", recurrent_activation="sigmoid",
        recurrent_dropout=0.0, stateful=True,
        return_sequences=True,
        unroll=False, use_bias=True, reset_after=True,
        dropout=0.2)

    dense_pre = Dense(units=32, activation=relu)
    dense_post = Dense(units=1, activation="sigmoid")

    # Input
    input_shape = (39, 20)
    inputs = Input(
        shape=input_shape,
        batch_size=batch_size,
        dtype=tf.float32
    )

    # Feed input through the network

    outputs = dense_pre(inputs)
    outputs = rnn(outputs)
    outputs = dense_post(outputs)

    outputs = tf.reshape(outputs, tf.shape(outputs)[:-1])

    adam_opt = Adam()
    metrics = ["accuracy"]

    model = Model(inputs=inputs, outputs=outputs)
    model.compile(optimizer=adam_opt,
        loss=binary_crossentropy,
        metrics=metrics)

Through stateful=True the model should learn to support "streaming" capabilties, such that by the time we are doing inference, we can feed the network data chunkwise without manually resetting the state.
With return_sequences=True we can produce a probability for each timestep.
The problem I am currently facing is that it somehow converges too fast while reaching low loss very quickly (I would assume that the low loss would result into the stagnation of the learning process). I am wondering whether this approach is somehow sub-optimal or whether I am missing something very crucial here.
I also have the suspicion that the loss function/y_true might not be optimal here.


texts: VAE will always results in somewhat different latent vectors for same input?

texts: Hey I was wondering if my intuition is correct that for the same input in a VAE we will get a slightly different vector every time we feed it through the network, due to the random sampling operation?


texts: How to get vector representations(or embeddings) of time series?

texts: Even if a time series is constructed up of numbers only, finding abstract fixed-dim vector representation would be interesting for classification/clustering purposes. As we can learn & find abstract representations/embeddings of text/images, can we do something similar on Time series? Finding such ways would result in better clustering & related tasks instead of traditional ways using some statistical measures like Pearson correlation etc. All thoughts are welcome.


texts: Is there an alternative to nltk in golang?

texts:
Golang is one of my favourite languages and I want to use it for a personal NLP/ML project. Is golang's ecosystem good and rich enough for this?

Is there an alternative package for nltk in golang?


texts:
In a classification problem when the response variable has multi-class, e.g., "sunny","rainy","cloudy", how should we encode it? I know that for predictors like this, usually we do One Hot Encoding, and if a predictors have too many classes, then we might just use the LabelEncode.

When this multi-class issue occurs in the response variable, I guess we can just use LabelEncode() rather use using One Hot encoding right? Because if we use One Hot encoding, then we will have 2 variables as the response variable, and the machine learning algorithm in sklearn usually expects the response variable not to be a vector right? (I mean it expects a long 1D vector with length equal to the number of observations but not a 2D matrix). But on the other hand, if we just map "sunny", "rainy","cloudy" to {1,2,3} or {0,1,2}, or whatever 3 numbers, that will create a less than or greater than relationship among "sunny", "rainy","cloudy", which is not inherited in the original problem.


texts: Why is KNN better at K-Fold Cross Validation than XGBoost or Random Forest?

texts:
I've been running K-Fold cross validation multiple times for KNN, random forest and XGBoost.
KNN can complete sklearn's cross_val_score, so much faster consistently.
They all use the same preprocessed data, all have the same test/train split with a random state, etc.
When ran individually the timings are within 10% of each other.
When ran using sklearn's cross val score, KNN can do 50 k-folds in less than 10 minutes, when the other two take over an hour each.  (this is consistent, i've ran it 5 times for all 3 algorithms)
Mathematically, or maybe due to sklearn's code, is there a reason for this? (The dataset is 200,000 by 100 columns, at a 25% test/train split)


texts:
I have only just started to look into graph neural networks and I am a little confused on the node embedding process. Here is my understanding, please let me know if i misunderstood:
Given unlabelled data, we try to construct a graph $G=(V,E)$ and generate a mask that contains important features of each node such as shape, size, intensity etc. Then let's say the aim is to try and classify the nodes. So, this is where we use node embedding, it seems that it takes all of those aforementioned features and tries to condense them into a embedding matrix. And the reason we do this is because this removes all unneccessary information from the adjacency matrix and only contains all the important features of the nodes. Then afterwards we use this matrix with graph convolutions to ultimately classify unseen images. Is this right?
So let's take an example, I've been looking into GraphSage, as this seems one of the best methods (is it?). It seems it can learn embeddings from a local neighbourhood, but i want to know how does that work?


texts: What is a good objective function for allowing close to 0 predictions?

texts:
Let's say we want to predict the probability of rain. So just the binary case: rain or no rain.

In many cases it makes sense to have this in the [5%, 95%] interval. And for many applications this will be enough. And it is actually desired to make the classifier not too confident. Hence cross entropy (CE) is chosen:

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

But cross entropy practically makes it very hard for the classifier to learn to predict 0. Is there another objective function that does not behave that extreme around 0?

Why it matters

There might be cases where it is possible to give the prediction 0% (or at least something much closer to 0 like $10^{-6}$). Like in a desert. And there might be applications where one needs this (close to) zero predictions. For example, when you want to predict the probability that something happens at least once. If the classifier always predicts at least a 1% chance, then having rain at least once in 15 days is

$$1 - (1-0.05)^{15} \approx 54\%$$

but if the classifier can practically output 0.1% as well, then this is only

$$1 - (1-0.001)^{15} \approx 1.5\%$$

I could also imagine this to be important for medical tests or for videos.


texts: Why Restricted Boltzmann Machine(RBM) is better than random initiating?

texts:
To train a Deep neural network, why training Restricted Boltzmann Machine(RBM) layer-wise first and then apply BP algorithm through the network?

Comparing to random initiating all the weights+bias in all layers and then apply BP algorithm, is there a quantitative reason(proof) that pretrained RBM is better than random weights assignment? I found a lot of case(datasets) studies about these two training methods saying RBM is better than random.

(Some documents argued that RBM also could help avoiding local minimizers when running gradient descent, but didn't offer any proof)


texts: How do you effectively predict the top 20% most likely customers to churn from a dataset?

texts:
I am looking to work out that if I have a dataset with 100,000 existing customers who didn't churn and 20,000 previous customers that churned in the past and the business objective is to target the 20% of customers most likely to churn within the business, how would that be done?
For example we would have to take this dataset and split it into a training and test set. Lets say the split is an 80/20 ratio for the training and test set respectively. That means that when we build our model on the 80% of the data, we can no longer use this data to see if any of the existing customers are likely to churn as we cannot evaluate a model on the data we have used to train it as it. We can only use the remaining 20% of the test set in order to evaluate our model and which customers are more likely to churn or not by considering the probabilities of each customer within the test set. What if there are existing customers who have a high chance of churning that we miss because they were in the training set?
Is what I have said above correct? Its very different from predicting, for example, if someone will default on their loan as you can train on all past data and then use this data to predict on new customers coming into the bank etc but with churn you want to predict on the customers who exist in the bank at the moment to avoid them leaving.
Any answers are greatly appreciated.


texts: What is the difference between float64 and double in TensorFlow?

texts: In storing floating point values both overflow and underflow problems cause loss of data. In machine learning tasks underflow is a common problem. I wanted to know if double is better than float64 in TensorFlow or not and if there is any difference between them?


texts:
I have seen this in a kaggle notebook. I understand we add some weight to classes. what I don't understand is how those weights are generated. below is the code. Can you explain why it's useful and how weights are produced?
p = 0.369197853026293
pos_public = (0.55410 + np.log(1 - p)) / np.log((1 - p) / p)
pos_private = (0.55525 + np.log(1 - p)) / np.log((1 - p) / p)
average = (pos_public + pos_private) / 2
print (pos_public, pos_private, average)

w1 = average / p
w2 = (1 - average) / (1 - p)

dtrain = xgb.DMatrix(X, label = y)
def weighted_log_loss(preds, dtrain):
    label = dtrain.get_label()
    return "weighted_logloss", -np.mean(w1 * label * np.log(preds) + w2 * (1 - label) *
              np.log(1 - preds))


texts: Error when trying .transform for OrdinalEncoder from Scikit Learn

texts:
I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out.
for i in range(len(ordinal_orders)):
   ord_en = OrdinalEncoder(categories = {0:ordinal_orders[i]})
   X_train.loc[:,ordinal_features[i]] = ord_en.fit_transform(X_train.loc[:,ordinal_features[i]].values.reshape(-1,1))

This works fine but when i try and apply this transformation to the test set i get an error.
for i in range(len(ordinal_orders)):
    ord_en = OrdinalEncoder(categories = [ordinal_orders[i]])
    X_test.loc[:,ordinal_features[i]] = ord_en.transform(X_test.loc[:,ordinal_features[i]].values.reshape(-1,1))

i get the value error
AttributeError                            Traceback (most recent call last)
<ipython-input-45-24cb27da6829> in <module>
      1 for i in range(len(ordinal_orders)):
      2     ord_en = OrdinalEncoder(categories = [ordinal_orders[i]])
----> 3     X_test.loc[:,ordinal_features[i]] = ord_en.transform(X_test.loc[:,ordinal_features[i]].values.reshape(-1,1))
      4

/opt/anaconda3/lib/python3.8/site-packages/sklearn/preprocessing/_encoders.py in transform(self, X)
    698             Transformed input.
    699         """
--> 700         X_int, _ = self._transform(X)
    701         return X_int.astype(self.dtype, copy=False)
    702

/opt/anaconda3/lib/python3.8/site-packages/sklearn/preprocessing/_encoders.py in _transform(self, X, handle_unknown)
    105         X_mask = np.ones((n_samples, n_features), dtype=np.bool)
    106
--> 107         if n_features != len(self.categories_):
    108             raise ValueError(
    109                 "The number of features in X is different to the number of "

AttributeError: 'OrdinalEncoder' object has no attribute 'categories_'

The ordinal features and orders are
ordinal_features=['LotShape','ExterQual','ExterCond','BsmtQual','BsmtCond',
'BsmtExposure','BsmtFinType1','BsmtFinType2','HeatingQC','KitchenQual',
'FireplaceQu','GarageQual','GarageCond','GarageFinish','Fence','PoolQC']

ordinal_orders=[
    #LotShape
['Reg','IR1' ,'IR2','IR3'],
    #ExterQual
['Fa','TA','Gd','Ex'],
    #ExterCond
['Po','Fa','TA','Gd','Ex'],
    #BsmtQual
['None','Fa','TA','Gd','Ex'],
    #BsmtCond
['None','Po','Fa','TA','Gd','Ex'],
    #BsmtExposure
['None','No','Mn','Av','Gd'],
    #BsmtFinType1
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #BsmtFinType2
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #HeatingQC
['Po','Fa','TA','Gd','Ex'],
    #KitchenQual
['Fa','TA','Gd','Ex'],
    #FireplaceQu
['None','Po','Fa','TA','Gd','Ex'],
    #GarageQual
['None','Po','Fa','TA','Gd','Ex'],
    #GarageCond
['None','Po','Fa','TA','Gd','Ex'],
    #GarageFinish
['None','Unf','RFn','Fin'],
    #Fence
['None','MnWw','GdWo','MnPrv','GdPrv'],
    #PoolQC
['None','Fa','Gd','Ex']
]

train and test set is just a generic dataset from kaggle, theyre not an issue here.
Any sort of clarification would be nice!


texts: Unseen samples(Rare category) during prediction?

texts:
Say if I train on these features(combination of categorical and numerical data),

we could see that in feature x2, sample 4 has a rare entry 'b'. if during my train test split, I end up not getting that rare sample in training data, I could use cv , But what if some unseen rare sample's like 'C' shown above in the picture ends up in my prediction data? Can someone please explain me how to tackle these situations and how to get good prediction results out of these?


texts: Additive vs Multiplicative model in Time Series Data

texts: The above time series plot is a daily closing stock index of a company. I want to know which model between additive and multiplicative best suits the above data. I know what the two models are, but i haven't been able to figure out the correct model for the above data. Also, is there any way other than simple visualisation which can help me decide the correct model?


texts:
I am looking for a dataset containing a large number of NLP research papers and abstracts. Are there any open access datasets like that?

If so, can you please share the details?


texts: Including column of indices as predictor for model?

texts:
I have a small dataset consisting of 1000 observations (rows), 11 predictors + 1 response (12 columns). It is a toy dataset used for learning purposes in a machine learning class at university -- binary classification (heart disease vs. no heart disease).
I have fitted a logistic regression, SVM, and KNN.
However, I am using python and I was deleting some rows corresponding to missing values and outliers I did not want to consider -- doing it with df.drop() and then reset_index(). If you do not set the drop flag of reset_index to true you will get a column in your data frame with the indices you wanted to reset. I accidentally forgot to set the flag to true and when fitting the 3 mentioned models included the column with these indices. This gave me an f1-score of 99-100%. When I realized that I had accidentally included this column, I dropped the column, re-fit and achieved ~ 60-80 percent F1 score with the respective models.
Is there a reason why including this column of indices gets you to 100% F1 score or even accuracy? Is this merely a coincidence that apparently works with the dataset?


texts: Keras - Why Validation data produce good results, while unseen data is performing poorly

texts:
I've built a feedforward net that predicts 2 classes (0,1). I've used the validation_split attribute like so:

history = model.fit(X, Y, validation_split=validation_split_ratio, epochs=700, batch_size=10, verbose=1, callbacks=[tb, checkpoint])

I got strange results (validation performs better than the train data):

I then took the validation data used for the training (20% from the last samples in my dataset) and tested the model to build a confusion matrix:

But when I trying to build a confusion matrix from new unseen data, my model performs poorly.

Why is that? I thought the validation data used by Keras is also "unseen" by the model.

UPDATE Visualization code:

from keras.layers import Dense, Dropout
from keras.models import Sequential
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import classification_report, confusion_matrix
import itertools

def create_model():
    ## Create our model
    model = Sequential()
    # 0.81250

    model.add(Dense(44, input_dim=18, kernel_initializer='uniform', activation='relu'))
    model.add(Dropout(0.8))
    model.add(Dense(44, activation='relu'))
    model.add(Dropout(0.4))
    model.add(Dense(3, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(1, activation='sigmoid'))
    return model

def load_trained_model(weights_path):
    model = create_model()
    model.load_weights(weights_path)
    return model

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

dataset = np.loadtxt("./validation/validation.csv", skiprows=1, delimiter=",")
model = load_trained_model('./checkpoints/weights.best.hdf5')
X = dataset[:, :18]
Y = dataset[:, [18]]

Y_pred = model.predict(X)
print(Y_pred)

y_pred = model.predict_classes(X)
print(y_pred)

p = model.predict_proba(X)  # to predict probability

target_names = ['class 0','class 1']
print(classification_report(Y, y_pred, target_names=target_names))
cnf_matrix = confusion_matrix(Y, y_pred)
print(cnf_matrix)
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=target_names,normalize=True,
                      title='Confusion matrix, without normalization')
plt.show()

Dataset sample:

0,1,0,0,0,0,0,1,0,0,0,5,1,6.01,4.86,43.0,1,2,1
1,0,0,0,0,0,0,0,1,0,117,3,1,5.02,4.86,54.0,0,2,1
0,1,0,0,0,0,1,0,0,0,1,2,1,6.01,4.86,70.0,1,2,0

Last column is the label (1 or 0)


texts:
What is the correct number of biases in a simple convolutional layer? The question is well enough discussed, but I'm still not quite sure about that.

Say, we have (3, 32, 32)-image and apply a (32, 5, 5)-filter just like in Question about bias in Convolutional Networks

Total number of weights in the layer kernel trivially equals to $3x5x5x32$.
Now let us count biases. The link above states that total count of biases is $1x32$, which makes sense because weights are shared among all output cells, so it is natural to have only one bias for each output feature map as a whole.

But from the other side: we apply activation function to each cell of output feature map separately, so if we will have different bias for each cell, they do not sum together, so the number $0x0x32$ instead of $1x32$ makes sense too (here 0 is the output feature map  height or width).

As I can see, first approach is widely used, but I also saw the second approach in some papers.

So, ($3x5x5 + 1)x32$ or  $(3x5x5 + 0x0)x32$?


texts: RNN Layer with multiple independent sequences for same label

texts:
I have a question about the use of longitudinal data (reoccurring sequences of data). Let's imagine we have multiple independent longitudinal features (e.g. visits to doctor and purchases in online store, both can appear 0:n times) within one label/person. Here is an example for the visits to the doctor, data for purchases would look similar:

In addition there are numerous categorical and numerical features, which are all fed into a Neural Net with these 3 kinds of inputs:

Numerical -> 1:1 into vector
Categorical -> Embedding Layer
Longitudinal -> RNN (LTSM or GRU) Layer

The results of the layers are then concatenated and put into a Dense Layer.
I see 2 options to deal with the longitudinal data:

Concat independent sequences (fixed length with padding for each sequence) and process in a single RNN Layer
Use different RNN Layers for each input and concat the results

I am wondering if both are theoretically applicable and if so what are benefits/drawbacks for these approaches? Are there better approaches to handle this data?


texts:
I don't understand where this formula for Mean Squared Error is coming from.

How do we arrive at:

$$MSE = \frac{1}{m}||y' - y||_2^2$$

from:

$$MSE = \frac{1}{m}\cdot\sum_i(y'_{i} - y_{i})^2$$

(The source is deeplearningbook)


texts: How to predict future prices with Keras LSTM time-series prediction model?

texts:
I have a trained and tested LSTM model which is meant to predict Ethereum close prices using all time csv data (24h steps). How do I now go about inputting an empty dataframe with future dates to recieve a plot of the future price trend.
Price data csv format example:
Currency,Date,Closing Price (USD),24h Open (USD),24h High (USD),24h Low (USD)

ETH,2015-08-09,0.909046,1.749289,1.91654,0.794497

ETH,2015-08-10,0.692321,0.909046,0.909046,0.692321

ETH,2015-08-11,0.668067,0.692321,0.692321,0.654331
...

code:
from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout, LSTM
import matplotlib.pyplot as plt
import numpy as np
from numpy.core.numeric import NaN
import pandas as pd
import seaborn as sns
from sklearn.metrics import mean_absolute_error

def train_test_split(df, test_size=0.2):
    split_row = len(df) - int(test_size * len(df))
    train_data = df.iloc[:split_row]
    test_data = df.iloc[split_row:]
    return train_data, test_data

def line_plot(line1, line2, label1=None, label2=None, title='', lw=2):
    fig, ax = plt.subplots(1, figsize=(13, 7))
    ax.plot(line1, label=label1, linewidth=lw)
    ax.plot(line2, label=label2, linewidth=lw)
    ax.set_ylabel('Price [USD]', fontsize=14)
    ax.set_title(title, fontsize=16)
    ax.legend(loc='best', fontsize=16)

def normalise_zero_base(df):
    return df / df.iloc[0] - 1

def normalise_min_max(df):
    return (df - df.min()) / (data.max() - df.min())

def extract_window_data(df, window_len=5, zero_base=True):
    window_data = []
    for idx in range(len(df) - window_len):
        tmp = df[idx: (idx + window_len)].copy()
        if zero_base:
            tmp = normalise_zero_base(tmp)
        window_data.append(tmp.values)
    return np.array(window_data)

def prepare_data(df, target_col, window_len=10, zero_base=True, test_size=0.2):
    train_data, test_data = train_test_split(df, test_size=test_size)
    X_train = extract_window_data(train_data, window_len, zero_base)
    X_test = extract_window_data(test_data, window_len, zero_base)
    y_train = train_data[target_col][window_len:].values
    y_test = test_data[target_col][window_len:].values
    if zero_base:
        y_train = y_train / train_data[target_col][:-window_len].values - 1
        y_test = y_test / test_data[target_col][:-window_len].values - 1

    return train_data, test_data, X_train, X_test, y_train, y_test

def build_lstm_model(input_data, output_size, neurons=100, activ_func='linear',dropout=0.2, loss='mse', optimizer='adam'):
    model = Sequential()
    model.add(LSTM(neurons, input_shape=(input_data.shape[1], input_data.shape[2])))
    model.add(Dropout(dropout))
    model.add(Dense(units=output_size))
    model.add(Activation(activ_func))

    model.compile(loss=loss, optimizer=optimizer)
    return model

np.random.seed(42)
window_len = 5
test_size = 0.2
zero_base = True
lstm_neurons = 100
epochs = 20
batch_size = 32
loss = 'mse'
dropout = 0.2
optimizer = 'adam'

hist = pd.read_csv(r"C:\Users\tristan\Documents\ETH-USD\ETH_USD_2015-08-09_2021-01-18-CoinDesk.csv")
del hist['Currency']
hist = hist.set_index('Date')
hist.index = pd.to_datetime(hist.index)
hist = hist.reindex(columns=['Closing Price (USD)','24h High (USD)','24h Low (USD)','24h Open (USD)'])
target_col = 'Closing Price (USD)'
train, test, X_train, X_test, y_train, y_test = prepare_data(hist, target_col, window_len=window_len, zero_base=zero_base, test_size=test_size)
model = build_lstm_model(X_train, output_size=1, neurons=lstm_neurons, dropout=dropout, loss=loss,optimizer=optimizer)
history = model.fit(X_train, y_train, epochs=epochs, batch_size=batch_size, verbose=1, shuffle=True)
targets = test[target_col][window_len:]
preds = model.predict(X_test).squeeze()
print(mean_absolute_error(preds, y_test))
preds = test[target_col].values[:-window_len] * (preds + 1)
preds = pd.Series(index=targets.index, data=preds)
line_plot(targets, preds, 'actual', 'prediction', lw=3)
plt.show()

Thank you!


texts:
I'm scraping data from Bing search results for (non-commercial purposes, of course) on Python using BeautifulSoup. I've entered an Indian dessert name, called 'rasmalai' as the word that I am focusing on. The code I'm using returns the title and a description of the web page. I've also extracted the links for the results. Here is the code I used:

from bs4 import BeautifulSoup
import urllib, urllib2

def bing_search(query):
    address = "http://www.bing.com/search?q=%s" % (urllib.quote_plus(query))

    getRequest = urllib2.Request(address, None, {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 Chrome/65.0.3325.162 Safari/537.36'})

    urlfile = urllib2.urlopen(getRequest)
    htmlResult = urlfile.read(200000)
    urlfile.close()

    soup = BeautifulSoup(htmlResult)

    [s.extract() for s in soup('span')]
    #unwantedTags = ['a', 'strong', 'cite']
    #for tag in unwatedTags:
        #for match in soup.findAll(tag):
           # match.replaceWithChildren()

    results = soup.findAll('li', {"class" : "b_algo" })
    for result in results:
        print "# TITLE: " + str(result.find('h2')).replace(" ", " ") + "\n#"
        print "# DESCRIPTION: " + str(result.find('p')).replace(" ", " ")
        print "# ___________________________________________________________\n#"

    return results

if __name__ == '__main__':
    links = bing_search('rasmalai')

Now that I have the links, web page title, and a short description, I want to extract keywords using NLP. In the end, I'd like to produce a CSV file with the dish name and associated keywords. Could someone guide me to some resources on how to do this part?

Thank you so much in advance.


texts:
I'm using the newest version of xgboost package in python 2.7 and based on my problem, I'm going to change xgboost cost function to use my own defined cost function. Couple of questions:

In which part of the xgboost package should I find and change the cost function?
After changing the cost function, how can I add the updated code to the xgboost package to use the new training function instead of xgboost's pre-defined function?


texts:
If you look at this:
>>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

I suppose fist row of array means "predicted ant" and first column is "actually is ant" second column is "actually is bird" etc.
So first row first col 2 i read like "predicted ant, is ant", first row second col 0 i read as "precited ant is bird" is 0 which fits, and third column is "predicted ant is cat" is 0 but should be 1.
What i am doing wrong while understanding the confusion matrix.
Another example is this
>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

Where is not even clear, what is the order of classes.
Source: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
edit: Unless it is swapped. First row is "is ant" not "predicted ant". Only that on wikipedia the system is that row is the prediction.


texts: Approach to labelling new data for training

texts:
Overview:
Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server.
I don't want to go through all the new cat and dog photos labelling whether each one is a cat or dog.
So how should I split the new data into categories to be trained on. The neural network may have identified inaccurately. What is a sure fire approach to labelling new data to be trained?

Imagine the user doesn't know whether their pet is a cat or a dog!


texts:
I need to save the results of a fit of the SKlearn NearestNeighbors model:

knn = NearestNeighbors(10)
knn.fit(my_data)

How do you save to disk the traied knn using Python?


texts: convert predict_proba results using class_weight in training

texts:
As my dataset is unbalanced(class 1: 5%, class 0: 95%) I have used class_weight="balanced" parameter to train a random forest classification model. In this way I  penalize the  misclassification of a rare positive cases.

rf = RandomForestClassifier(max_depth=m, n_estimators=n_estimator,class_weight = "balanced")
rf.fit(X_train, y_train)

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

In my case the classes frequencies are:

fc = len(y_train)/(len(np.unique(y_train))*np.bincount(y_train))

10000/(2*np.array([9500,500]))
array([ 0.52631579, 10.        ])

I apply my model over a test dataset using predict proba function:

y_predicted_proba = rf.predict_proba(X_test)

The second column presents the probabilities of being 1 to the input samples. However I understand this probability must be corrected to be real.

If I divide them by class_weight values these new probabilities don't sum one...

How can this correction be achieved?


texts: CNN with XGBoost as an output layer, is it better?

texts:
Context:
I have been experiencing with some Kaggle datasets to learn more about image classification. So, in this binary image classification task, I tried something that I thought would increase the accuracy of the model but then It didn't. I don't understand why exactly that did it decrease so I am just looking for some sort of explanation.
Problem statement:
I used a CNN sequential model with 7 conv layers followed by 1 dense layer and finally 1 sigmoid unit for the output.
After some hyperparameter tuning, the model reached 90% accuracy on the test set.
Then I thought about using and XGBoost model in place of the last output layer. I achieved that by using feature extraction from the Keras model. My intuition was that at worst the accuracy would stay the same as before, however the accuracy on the test set dropped to 88%!
I couldn't understand why exactly does the accuracy drop since the last output layer in the CNN was basically a sigmoid unit which works as if we have a logistic regression model, and from my minimal experience, XGBoost usually gives better results than logistic regression.
Any insight or explanation would be appreciated, and thanks!


texts: Semantic segmentation: mean IOU in presence of missing classes

texts:
It seems to me that the mean IOU is a poor metric in the presence of unbalanced classes. E.g., suppose I have 10 classes but one image has only 2 classes present in its label. Consider the prediction where the 2 classes are inverted, the IOU for these classes is 0, but the IOU for the 8 other classes is 0/0. If we consider it to be 1 (since no prediction for those classes is the correct prediction), with uniform weights the resulting mean IOU is 0.8 which makes it sound like the prediction was quite good.

It seems that the mean IOU weighted by the proportion of pixels corresponding to each class in the target image solves this problem. Are there cases where using the later over uniform mean IOU is problematic? Why is the uniform mean IOU the more popular of the two metrics?


texts: how to reconstruct image from feature space of convolutional neural network?

texts:
lets say i have fed an image into VGG19 pre-trained on imagenet as follows:
from tensorflow.keras.applications.vgg19 import VGG19
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg19 import preprocess_input
from tensorflow.keras.models import Model
import numpy as np

base_model = VGG19(weights='imagenet')
model = Model(inputs=base_model.input, outputs=base_model.get_layer('block5_pool').output)

img_path = 'C:/shared/a5/images/training/n01518878_8432.JPEG'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

features = model.predict(x)

i then extract the values at layer5 max pool layer, which are determined by the input image, and unique for every different image that is fed into the network.
does anyone know how to reconstruct the original image based on the features extracted from a specific layer? thank you


texts: YOLO annotation guildelines: overlapping and partially visible objects

texts: Can you please advise on how I should annotate objects that are partially visible or overlapping?


texts: Should I normalize the dependent variable in a penalized linear regression model?

texts: When I compute penalized regression on the data without normalizing using the glmnet package in R, the lambda values and RMSE generated in lasso, ridge, and elastic net are unreasonably large. The RMSE generated is in thousands. However, when I normalize the response variable, I see that the lambda, RMSE, and R^2 values are all within reasonable range, < 1. Are we supposed to normalize the response variable? I tried scaling the predictor variables, but it still generates large values for lambda, RMSE, and R^2. The response variable is numeric, number of shares of online articles and the values range from 1-840,000 with a mean of 3500.


texts: "Binary Encoding" in "Decision Tree" / "Random Forest" Algorithms

texts:
Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities?
Some facts about my dataset:

My dataset has ~170,000 rows
One of the categoric variables has 1,700 unique values.
Another one has 3,000 unique values.
Note that it is not practically possible to group the values of those variables into more aggregate levels.

As a domain expert,I am sure those categorical columns with high cardinalities are strong candidates as predictors.
On the other hand, binary encoding surely decreases model's interpretability.
Else than interpretability, after binary encoding, is it just alright to build a decision tree / random forest model on the newly formed dataset with new variables which only indicating bits?
Click for a good post on encoding categorical features


texts:
I have some question concern similarity measure

Suppose that we have a matrix M where M(i,j) is the similarity measure between user i and user j .

Each user is characterised by : id-user | country | id-artist | id-track

For this I choose to use Jaccard similarity metric.

Jaccard is determined to compute similarity between users based on the tracks that they listened.
My question is : is it possible to take account both id-track and id-artist to measure the similarity between users?

Thank you


texts: Representing similar states in reinforcement learning?

texts: Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state having 10 spades, 4 hearts, 6 clubs on the table and holding King and Queen of hearts had already been visited, I would like it to affect the weights of similar states, like the same cards with different suits. How do I accomplish this?


texts: Manipulate the nltk.word_tokenize to remove the stopwords and assign to two dataframe

texts:
I'm trying to manipulate an imported list of keywords with about 1000 factors from a CSV, tokenizing the list while, at the same time, removing the stop words.
A dedicated function, returning a tuple, was expected to memorise the value of the innermost objects in the two additional dataframe, but so far I've been failing in my attempts.

Below what I've been doing. So long I do something like the below, everything is good
data["unigrams"] = data["Keywords"].apply(nltk.word_tokenize)

However, the resulting dataframe still contains the stopwords.

I therefore written a small function that manipulate the value and return a tuple, but I then fail to assign to the dataframe with the following error:
TypeError: expected string or bytes-like object

The assignment I do, is something like this. I couldn't get any other syntax working, but this is not what I'm willing to figure out now.

data = data.assign(Tokenized = lambda x: doIt(x['Keywords']),
                   Filtered = lambda y: doIt(x['Keywords']))

The doIt function code is:

def doIt(keyword):
    filtered = []
    tokenized = nltk.word_tokenize(keyword)

    for w in tokenized:
        if w not in stop_words:
            filtered.append(w)

    return tokenized, filtered

When recalled individually the doIt("this is my string") code works fine, though, returning the following tuple: (['this', 'is', 'my', 'string'], ['string'])

In an attempt to debug the function, I then realised that something but be connected with the length of the inner object. In fact, a call like this

vals = tokenAndRemoveStopWords("this is my string")
dfObj = pd.DataFrame(vals, columns=['A', 'B'])

returns the following error: ValueError: 2 columns passed, passed data had 4 columns

As far as I understand then, the problem seems to be with the inner object that is later unpacked to be fitted in the dataframe that is evidently smaller. However, my attempt was to get the first object in the tuple and memorised as a list; the same for the second.

Not sure whether I was able to explain the concept properly though, but I'm in my infancy with this Data Science bits.

Thanks
Andrea


texts: In Word2Vec, how to the vector values translate back to related words?

texts:
Apologies for the newbie question: I've just downloadd the GoogleNews word2vec bin file and used convertvec to convert it to a text file to look at the vector values. I see they are values between -1 and 1. Can somebody explain how these values map back to related terms?

I would eventually like to write some code to take this bin/text file, transform it and load it into a MongoDB database for further manipulation.


texts: How can I find synonyms and antonyms for a word?

texts:
I found some code online where I can feed in a word, and find both synonyms and antonyms for this word.  The code below does just that.

import nltk
from nltk.corpus import wordnet   #Import wordnet from the NLTK
syn = list()
ant = list()
for synset in wordnet.synsets("fake"):
   for lemma in synset.lemmas():
      syn.append(lemma.name())    #add the synonyms
      if lemma.antonyms():    #When antonyms are available, add them into the list
          ant.append(lemma.antonyms()[0].name())
print('Synonyms: ' + str(syn))
print('Antonyms: ' + str(ant))

My question is, how can I choose a word, like 'and', and find all synonyms and antonyms based on comments that exist in a field in a dataframe?  Here is a sample of the first 10 lines from my dataframe.

feels weird; may be a fake!
package came end missing box. since itâ€™s gift i update actual fit.
birkenstock amazing shoe!!!! i wish i ten pairs!
delivered advertised.... shoe looks fake.
second pair i had. nothing beats them.
they totally fake ðŸ˜¡. they felt weird i finally noticed â€œmade germanyâ€ logo above. they also smell like glue leather. infuriating happen!
i've birkenstock wearer 35 years. i wear 10 women's size ones i ordered wear big. i need size down.
great brand, packaging good, dad likes them. the size clearly needs fixed others don't order bigger sizes wanted. some people don't want bigger size actually wear.
false advertising.
a bit loose compared birkenstocks size. still like though. very comfy


texts: Class weights for imbalanced data in multilabel problems

texts:
I am trying to train a CNN for a multiclass - multilabel classification task (20 classes, each sample can belong to 1+ labels) and the dataset is highly imbalanced. In single-label cases I would use the compute_class_weights function from sklearn to calculate the class weights in order to help the optimizer to account for the minority class. However, for the multilabel case I feel its not working as supposed to, because it considers as number of samples the number of times all classes occur, while the actual number of samples are less (since its multilabel). Is anyone familiar with a function, or even a formula, to calculate the class weights in this case?

Thanks


texts: the models in deep-learning on a little corpus

texts:
I have a little corpus (around 450 observations with approximately 50 words) with domain specific words. I have to classify each observation with a target variable with 5 classes. I tokenized and did the preprocess on my text and now I want to do the classification.
I tried random forest and perceptron and I get an accuracy around 0.75 which is pretty good!! I saw on the web deep learning was a good tool to work with when we have textual data (that's why I tried perceptron) but I'm pretty new in this field.
Do you have any idea which model in deep learning is good considering my corpus?


texts: Revealing the causal structure in time-dependent data

texts:
We have a data table that accumulates the control and monitoring parameters of the High-Temperature Superconductor (HTS) production process: such that the rows represent the observations and columns represent the parameters mentioned above.
Due to the nature of the production process, there are time dependencies between the rows of our data sets. Thus the columns, are, indeed, time series. (Which boils down our data to time-dependent data.)
Now the question arises: whether we can apply induced causation methods, explained in sections 2.5 and 2.6 of [1], also publicly available and explained here, and implemented in this GitHub repo, to reveal the causal structure of the time-series data set, or are there better options?
[1] Pearl, J., 2009. Causality. Cambridge university press.


texts:
I ran the Neat Python XOR example and a neural network it found has the following properties:

  Best individual in generation 22 meets fitness threshold - complexity:
  (2, 8) Number of evaluations: 3450

  Best genome: Nodes:
          NodeGene(id=0, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)
          NodeGene(id=1, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)
          NodeGene(id=2, type=OUTPUT, bias=0.0, response=4.921960812206318, activation=sigmoid)
          NodeGene(id=3, type=HIDDEN, bias=-2.340619398413843, response=4.930091408143276, activation=sigmoid)
          NodeGene(id=4, type=HIDDEN, bias=0.4674809078722945, response=4.918937894580367, activation=sigmoid) Connections:
          ConnectionGene(in=0, out=2, weight=1.0534262789427578, enabled=True, innov=0)
          ConnectionGene(in=1, out=2, weight=-1.6479324848766828, enabled=True, innov=1)
          ConnectionGene(in=0, out=3, weight=-3.5479647367966765, enabled=True, innov=2)
          ConnectionGene(in=3, out=2, weight=5.970619032226201, enabled=True, innov=3)
          ConnectionGene(in=1, out=3, weight=4.634338591600734, enabled=True, innov=4)
          ConnectionGene(in=0, out=4, weight=0.19203601809123594, enabled=True, innov=5)
          ConnectionGene(in=4, out=2, weight=-0.6588709650159562, enabled=True, innov=6)
          ConnectionGene(in=4, out=3, weight=-3.0010419817357072, enabled=True, innov=8) Node order: [4, 3]

  Output: expected 0.00000 got 0.11990
  expected 1.00000 got 1.00000
  expected 1.00000 got 0.92939
  expected 0.00000 got 0.00395 {2} {2} {0,
  1, 3, 4} {0, 1, 2, 3, 4}

However, when trying to implement the neural network in Python, I don't get the same results:

import math

def sigmoid(x):
    f = 1 / (1 + math.exp(-x))
    return f

node0 = 1
node1 = 1
node2 = 0

node3_bias = -2.340619398413843
node4_bias = 0.4674809078722945

connection_0_2 = 1.0534262789427578
connection_1_2 = -1.6479324848766828
connection_0_3 = -3.5479647367966765
connection_3_2 = 5.970619032226201
connection_1_3 = 4.634338591600734
connection_0_4 = 0.19203601809123594
connection_4_2 = -0.6588709650159562
connection_4_3 = -3.0010419817357072

def main():
    activated_node4 = sigmoid(node4_bias * 1 +
                              node0 * connection_0_4)
    activated_node3 = sigmoid(node3_bias * 1 +
                              activated_node4 * connection_4_3 +
                              node0 * connection_0_3 +
                              node1 * connection_1_3)
    activated_node2 = sigmoid(activated_node4 * connection_4_2 +
                              activated_node3 * connection_3_2 +
                              node1 * connection_1_2 +
                              node0 * connection_0_2)

    print("The value is: ", activated_node2)

if __name__ == "__main__":

    main()

For nodes 1 and 1 I get:

The value is:  0.30957048732254944

which does not match the expected output above.

Can anyone please help me with some pointers as to what I am getting wrong?

Thank you.


texts: Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

texts:
A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples.
This is difference than NCE Loss, which doesn't use a softmax at all, it uses a logistic binary classifier for the context/labels. In NLP, 'Negative Sampling' basically refers to the NCE-based approach.
More details here
I have tested both and they both give pretty much the same results. But in word embedding literature, they always use NCE loss, and never sampled softmax.
Is there any reason why this is? The sampled softmax seems like the more obvious solution to prevent applying a softmax to all the classes, so I imagine there must be some good reason for the NCE loss.


texts: Decomposing time series data results in error

texts:
library("tseries")
library("xts")
library("scales")
library(astsa, quietly=TRUE, warn.conflicts=FALSE)
library(ggplot2)
library(knitr)
library(printr)
library(plyr)
library(dplyr)
library(lubridate)
library(gridExtra)
library(reshape2)
library(TTR)
cc <- ts(Mobile_Payments,frequency = 12,start = c(2010,1))
cc
names(Mobile_Payments)
View(cc[,5])

plot.ts(cc[,5], main="Trend and Seasonal Monthly Data", ylab="Total Agent Cash in Cash Out (Volume Million")
x <- xts(Mobile_Payments$`Total Agent Cash in Cash Out (Volume Million)`,as.Date(Mobile_Payments$Dates),frequency = 12, start =c(2010,1),end =c(2021,2))
x
decomposex <- decompose(x,"multiplicative")
autoplot(decomposex)

Error in decompose(x, "multiplicative") :
  time series has no or less than 2 periods


texts: Gradient Descent Python Implementation isnt converging

texts:
I'm trying to implement gradient descent in Python and following Andrew Ng course in order to follow the math. However, my implementation isn't working as I expected. It would be great if the community can help me to identify my mistake.
When I increase the range from 3 to higher number, it does not converge, rather thetas move from very positive to very negative and finally get nan because they get extremely small.
Code is given below:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression

X = pd.DataFrame(load_boston().data, columns = load_boston().feature_names)
X['theta0'] = 1
y = load_boston().target
y = pd.DataFrame(y, columns = ['target'])
theta = pd.DataFrame(np.random.randn(X.shape[1]),columns = ['target'], index = X.columns.values)

print('theta shape',theta.shape)
print('X shape',X.shape)
print('y shape',y.shape)
print(theta)

def predict(X,theta, ycol = 'target'):
    return X.dot(theta)

mse_values =[]
alpha = 0.01
for i in range(10000):
   error = predict(X,theta) - y
   theta = theta - ((alpha)* (1/len(X)) * X.T.dot(error))
   mse= np.sum(error**2)/len(X)
   print('mse: ', mse.values)
   mse_values.append(mse)
   print('+'*5)

plt.plot(mse_values)
plt.show()


texts: sklearn package with AttributeError: 'MissingValues' object has no attribute 'to_list'

texts:
I am currently trying to reproduce this tutorial on building a CNN based time series classifier for human activity recognition.
My setup is:

Windows 10, Pycharm IDE with a new project for this tutorial, Python3.6, freshly installed the needed packages.

For reproducing, you need to download the activity data here and place it in the project directory under ./Data
The code executes the graphs well until this position:
df[LABEL] = le.fit_transform(df["activity"].values.ravel())

and throws following error:
Traceback (most recent call last):
  File "C:/Users/bobin/PycharmProjects/Mussel/cnn_musseltest.py", line 226, in <module>
    df[LABEL] = le.fit_transform(df["activity"].values.ravel())
  File "C:\Users\bobin\PycharmProjects\Mussel\venv\lib\site-packages\sklearn\preprocessing\_label.py", line 117, in fit_transform
    self.classes_, y = _unique(y, return_inverse=True)
  File "C:\Users\bobin\PycharmProjects\Mussel\venv\lib\site-packages\sklearn\utils\_encode.py", line 31, in _unique
    return _unique_python(values, return_inverse=return_inverse)
  File "C:\Users\bobin\PycharmProjects\Mussel\venv\lib\site-packages\sklearn\utils\_encode.py", line 133, in _unique_python
    uniques.extend(missing_values.to_list())
AttributeError: 'MissingValues' object has no attribute 'to_list'

Related threads that have not helped me so far:
Link1
Link2


texts: Calculate likelihood of a date being worked based on location, hours and rate of pay

texts:
I am just getting into trying some machine learning. As a C# developer I have played around with ML.NET. I'd now like try and see if it can  help with a real world problem.

We have a system that invites staff members to work shifts in multiple hospitals, if a staff member is interested they contact our admin team and are then assigned to the shift.

Many shifts are difficult to assign staff to due to the date/day, location, the hours and of course the pay available. We therefore often have to increase the rate paid per hour. When the rate is increased there is a much better chance the shift will be filled.

My shift class is displayed below.

    // SHIFT CLASS
    int shiftID
    int locationID
    int shiftGradeID

    DateTime shiftStartDate
    DateTime shiftEndDate

    decimal hourlyRate
    decimal hours

    int assignedLocumID

Using the data above I'd appreciate any advice or insight to help work out;

Based on location, date (day, month), start time, end time, grade, hourly rate and hours, how can I work out the percentage chance of a shift being filled.

Ideally I'd like to try something out using ML.NET, however I'm not sure where to start. I have 5 years of data to train the model. Appreciate any help you can provide.


texts:
I am trying classify an input graph (a 2D point sequence) into one of the predefined graphs (A,B,C etc) using machine learning. The goal is to identify which type of graph the input graph belongs to.

I have done classification of single data points before, but I have never classified sequences of data like graphs before. The only way I could think of is calculating 'mean squared error' between input & each of A,B,C graphs and choose the category with the lowest error.

3 example outputs might look like this.

input graph belongs to type A (confidence: 82%)
input graph belongs to type C (confidence: 68%)
input graph doesn't belong to any type (max confidence: 12%)

How can I achieve this using classification techniques or any other accurate way?


texts: What does it mean when we say most of the points in a hypercube are at the boundary?

texts: If I have a 50 dimensional hypercube. And I define it's boundary by $0<x_j<0.05$ or $0.95<x_j<1$ where $x_j$ is dimension of the hypercube. Then calculating the proportion of points on the boundary of the hypercube will be $0.995$. What does it mean? Does it mean that rest of the space is empty? If $99\%$ of the points are  at the boundary then the points inside the cube must not be uniformly distributed?


texts: Validation loss is lower than the training loss

texts:
I am using autoencoder for anomaly detection in warranty data.

Architecture 1:

The plot shows the training vs validation loss based on Architecture 1.

As we see in the plot, validation loss is lower than the train loss which is totally weird.

Based on the post Validation loss lower than training loss, I understood that it is because of the dropout layer in my model. So I ran the model with dropout layer.

Architecture 2:

Based on the above architecture, I plotted the training vs validation loss. Now validation loss is a bit higher than the training loss.

Clearly it is because of the dropout layer.

Now my question is, whether the model based on Architecture 1 is correct or not? If it is not correct, what kind of changes I could do to make it work?

Thank you ! Any help is much appreciated!!


texts: what is convolutional implementation of sliding window in Faster RCNN? Also how RPN proposes region?

texts:
do we need to train this model on some standard dataset separately? and then use it? can we use pre trained model such as VGG etc for this?
also How RPN is used to propose regions?? unlike segmentation what technique RPN uses to propose regions?


texts: Which model is better, one just before overfitting with higher accuracy or one with no overfitting and lower accuracy?

texts:
I am training a CNN model.
In the first one I got a training accuracy of 87%(0.29 loss) and validation accuracy of 87%(0.30 loss) at 5th epoch, I kept training it for total of 15 epochs and as expected it started overfitting with training accuracy increasing to 97%(0.01 loss) and validation remaining at 87%(0.35 loss).
In the second model, I used Data Augmentation and Dropout layer to deal with overfitting (trained for total 10 epochs). These are the results : 5th Epoch : Train Accuracy 77% (0.45 loss) and Validation accuracy 77% (0.41 loss). 10th Epoch : Train Accuracy 82% (0.38 loss) and Validation accuracy 82% (0.35 loss)
From the loss and accuracy graph which you can see below, it's clear that Model is overfitting in first scenario, but in second it's not overfitting.
Scenario One

Scenario Two

My question is, which model is better in real world based on accuracy ? Model one stopped at epoch 5 with 87% accuracy or model 2 which has not overfit with 82% accuracy (validation)? I understand just based on accuracy model 1 sounds better but it's ultimately overfitting, but if I stop the training using early stopping or something similar, will this be a better model than my second one ?


texts: How to prune the features using recursive feature elimination?

texts:
I have some kind of spatial data for nearly 1000 locations and at each location around 5000 features. I am doing neighborhood analysis to identify which features are predominant in local neighborhood.
I read that using recursive feature elimination only strong features can be kept and weak features can be eliminated. For some particular location I get neighborhood data X of numpy shape (14, 5000) where 5000 is features and 14 are local neighbors.
I also get target vector of 14 length having 3 class labels.
I tried to perform following things for pruning the features but getting value error. I am new to ML community and not sure whether doing the right things or not. Any suggestion will be helpful.
from sklearn.model_selection import StratifiedKFold
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier

#X.shape (14, 5000)
#target.shape (14,)
#len(np.unique(target)))  3

rfc = RandomForestClassifier(random_state=101)
rfecv = RFECV(estimator=rfc, step=1, cv=StratifiedKFold(5), scoring='accuracy')
rfecv.fit(X, target)
print('Optimal number of features: {}'.format(rfecv.n_features_))

error
ValueError: n_splits=5 cannot be greater than the number of members in each class.


texts: Create data visualization for unstructured data - Basket Market Analysis

texts:
I have this dataset (just a sample):

product1,product2,product3
product1,product4
product1,product2
product4,product3,product1,product2

The products are grouped by transaction. I want to create some data visualization using this dataset but I don't any tool or any type of visualization that allows creating some visualization with this structure...

Anyone can suggest me an option?

Thanks!

I feel desperate because they can not find anything that suits this data structure


texts: Stratified selection based on the y response creates a bias in information (Berkson's bias)?

texts:
I have a database which has individuals in different months and a target variable which indicates wheter an event happened or not, let's say:

Id: 1, Month: 1, Event: 0
Id: 1, Month: 2, Event: 0
Id: 1, Month: 3, Event: 1
Id: 2, Month: 1, Event: 0
Id: 2, Month: 2, Event: 0

As my sample is so big (think of the number of combinations), I would like to choose a sample n much lesser than N (one month per individual).

For the individuals who had the event, the selected month is always the one which had the event. Always. There is no randomness.

Instead, for individuals which did not have the event, any month could be randomly selected.

These facts...

Selecting one month per individual.
Difference between selection methods for positives and negatives (I think is called Berkson's bias).

...affect the performance of the classification model?


texts: What is the meaning of "probability distribution of p(x)" of something uncountable?

texts:
I'm studying VAE and new to both of the neural network and the statistic.

After some researches, I could understand the rough concept of VAE.
But what makes me confused is, the meaning of probability distribution p(x) itself.
When the x is an image data, what is the meaning of "probability distribution of the image"? and what is "probability distribution of the latent space"?

When I learned the probability distribution in school, x in p(x) was always something numerable(the value of dice, the number of apples, ..), and so I could get some value like p(x=1).
But I couldn't understand the meaning of p('image data'), p('latent space').
Although many websites explains the concept and logical flow of VAE magnificently, I'm stuck by the lack of my knowledge.

Anyone help me?


texts: How can I get variable values from a plot?

texts:
I would like to know if there is any way in python to automatically determine the values of point a and b marked in the graph below. Said a different way, finding the values when my plot shows a marked decrease or after a marked increase.

This is the source code of my plot if it is helpful:

#load data

min_duration = data['Time_Secondes'].min()
max_duration = data['Time_Secondes'].max()
xr_ = list(range(min_duration,  max_duration,  60))
y_ = []

for i in range(0,(len(xr_)-1)):
    Time_proba_ = np.logical_and(data['Time_Secondes'].values >= xr_[i], data['Time_Secondes'].values < xr_[i+1])
    Time_proba_1_ = np.logical_and(np.logical_and(data['Time_Secondes'].values >= xr_[i], data['Time_Secondes'].values < xr_[i+1]), data['outcome'].values==1)
    y_.append(sum(Time_proba_1_ )/sum(Time_proba_))

import matplotlib
matplotlib.pyplot.plot(xr_[1:len(xr_)], y_)

```


texts: Populating Knowledge Base - Stanford DeepMind Alternatives

texts:
I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or Question-Answering application.

During my research I formulated following solution approach:

Parse doc, docx and pdf documents to raw text. Pre-process with common NLP tools.
Consult with client and decide upon several categories for entities.
Train customised NER model to obtain entities from texts.
Obtain entity-level-relations.
Populate knowledge base
Migrate knowledge base data to knowledge graph and set up the queryable system.

For the implementation, I was looking into Stanfords Deepdive architecture, however they stopped support in 2017 and I am not sure if it is still up to date.

Alternatively, I intended to use spaCy to train my own NER model, but it is unclear to me if this allows to obtain the entity-level-relations for populating a knowledge base.

Could you advise me to alternatives to Deepdive or any other resources that demonstrate how to build up such a system from scratch?


texts: Choose-many categorical features: alternatives to one-hot encoding?

texts:
I'm building a model to predict the lifetime value of a client based on the relational data we have on them.  The user table has a bunch of one-to-many child tables that might be predictive.  Grossly simplified, the child features boil down to things like:

a list of item categories that they've bought in the past
a list of the predominant colors in ads they've clicked on
etc, etc

In each case, the obvious feature comprises a list of ~ 0-10 choices from a categorical variable.  I have several of these features, some of which have as many as ~10k discrete values, so one-hot encoding would get very wide, very fast.
Aside: if there a term of art for this kind of "list-of-tags feature" that I'm referring to as "choose many categorical", please tell me.
Question: Is there an dense encoding scheme that works with choose-many categorical features?


texts: How do I iterate over my images in dataset?

texts:
I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like 0.jpg, 1.jpg, 2.jpg.....

I tried to develop an iterator to iterate over all my images but the problem arises that when I convert all of my 124 images in a single training_data array the model responds that it expected a single array yet 124 arrays were given to it. Can anyone tell me how I should write the iterator? I tried using the keras flow_from_directory function from the "Machine Learning mastery" website but it shows 0 images from 0 classes.

Here is my code:-->

import tensorflow as tf
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
from keras.models import Model
from keras import backend as K
from keras.callbacks import TensorBoard
import numpy as np
from PIL import Image

i = int(0)
images_dir = "/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/{}.jpg".format(i)
training_data = []

while i < 125:
    print("working on ", i, 'file')
    image = Image.open(images_dir)
    pic_array = np.asarray(images_dir)
    training_data.append([pic_array])
    i += 1

input_img = Input(shape=(600, 400, 3))  # adapt this if using `channels_first` image data format

x = Conv2D(48, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(24, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(24, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

x = Conv2D(24, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(24, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(48, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

autoencoder.fit(training_data,
                epochs=50,
                batch_size=128,
                shuffle=True,
                callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])

  Also I want the images to retain the 'color' feature so I am using the input shape as (600,400,3) because RGB is on 3 channels. is it correct?
   I would have simply used my iterator but it is my understanding that I need a different function that communicates to the model and gives it images one-by-one while I am just loading them all in a single variable. So can anyone help me with this?

Here is the full TraceBack:-

Traceback (most recent call last):
  File "autoencoder.py", line 46, in <module>
    callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])
  File "/home/awesome_ruler/.local/lib/python3.7/site-packages/keras/engine/training.py", line 1154, in fit
    batch_size=batch_size)
  File "/home/awesome_ruler/.local/lib/python3.7/site-packages/keras/engine/training.py", line 579, in _standardize_user_data
    exception_prefix='input')
  File "/home/awesome_ruler/.local/lib/python3.7/site-packages/keras/engine/training_utils.py", line 109, in standardize_input_data
    str(len(data)) + ' arrays: ' + str(data)[:200] + '...')
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 125 arrays: [array([['/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/0.jpg']],
      dtype='<U76'), array([['/home/awesome_ruler/Documents/Atom projects/Compression_enc/Images/IMG/0.jpg']]...


texts:
I am using TF Eager to train a stateful RNN (GRU).

I have several variable length time sequences about 1 minute long which I split into windows of length 1s.

In TF Eager, like in Keras, if stateful=True, "the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch." (source)

Thus, how should I design my batches? I obviously cant sample random windows from random sequences. I also cant split a sequence into windows and place adjacent windows in the same batch (e.g. batch 1 = [[seq1 0-1s], [seq 1 1-2s], [seq1 2-3s], ...]), as the state from the previous window wont get passed to the next window, as is the point of a stateful RNN.

I was thinking of mixing sequences in the same batch as in:

batch 1 = [[seq1 0-1s], [seq2 0-1s], [seq3 0-1s], ...]
batch 2 = [[seq1 1-2s], [seq2 1-2s], [seq3 1-2s], ...]
...

However, there the issue is that the sequences have different length, and thus some will finish before others.

So what is the best way to implement this?

(FYI, I couldn't find anything in the academic literature or blogoshhere which discusses this, so refs would be great)

Thanks!


texts: HDBSCAN cluster: still unclear to me how to chose 'min_cluster_size`

texts:
Hdbscan is an excellent technique to find the "optimal" number of clusters within your data when you have little a priori idea how many clusters should exist. This makes the method great for exploratory analysis:

http://hdbscan.readthedocs.io/en/latest/comparing_clustering_algorithms.html

Here's my problem: All results using hdbscan with the python implement in the link above rely on the crucial min_cluster_size

http://hdbscan.readthedocs.io/en/latest/parameter_selection.html

If users have a priori little idea how many clusters best fit the data, what is the correct approach above? Isn't there a metric one uses to decide what the optimal number of clusters is?


texts: Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question)

texts:
I'm a n00b in Python Pandas.
I want to do a particular aggregation / grouping / cross-tab but I know so little of the terminology that I don't even know how to look this up.
But here's what I want.
Say I have a table like this
Bob, Oranges,  5
Bob, Apples,  10
Bob, Bananas, 12
Tim, Oranges,  3
Tim, Apples,  20
Tim, Bananas,  5

I want to groupby fruit to find the total for each type of fruit. But produce another field containing a string which has the details sorted by their value in another column.
So I want an output something like this.
Oranges,  8, "Bob(5), Tim(3)"
Apples,  30, "Tim(20), Bob(10)"
Bananas, 17, "Bob(12), Tim(5)"

Where the string aggregates the values from the names column in a list sorted by the associated numeric value.
I know that there isn't something out of the box to do this, but what is this kind of operation or aggregation or pivoting (where you take one of the columns and turn it to be "horizontal" if that makes any sense) actually called?
How would I go about implementing it in Pandas functions?


texts: Why is my classification tree is showing only one node, and how to resolve this issue?

texts:
I also asked here: https://stats.stackexchange.com/questions/287587/reason-of-getting-only-one-node-while-implementing-classification-tree-in-matlab

Hello, I have $N=1520=19*80$ samples, each of which are $p=1764$-dimensional. So $N < p$ here. I want to perform a $19$-class classification of these samples. The training data consists of $80$ samples from each of the $19$ classes. I performed classification tree in matlab (sorry but this is the only language I know well) using fitctree$(X,Y)$ where $X=$matrix of covariates of size $N \times p$, $Y=$ vector of responses of size $N \times 1$.I'm getting only one node each time.

Just to see if the problem solves when I reduce the number of features significantly, I tried to take the first $100$ features by $X=X(:, 1:100)$, but still got only one node.

Next, my knn and naive Bayes classifiers are also predicting all samples to be in one class, just like the tree above.

What could be the problem, and how could I possibly solve it? Am I making a syntactical mistake?

Thanks a lot in advance!


texts:
I have a little more general question. My dataset consists of N sequences of events. Example of one sequence could be [A,B,C,D,X,Y] and another [A,B,Z], where letters represent different events. The sequences are at most 80 steps long.

The idea is to predict next letter or next step from known previous events. For very simple example maybe after A will always come B. Next step would be measuring time of each event and the ultimate goal is to predict how long until process reaches specific event.

I tried N-gram, MLP neural network and lastly LSTM network, which had around 80% accuracy.

That would not be bad if the events were balanced in the dataset. To account for that I used weighted loss function in training of the LSTM and then the overall accuracy is around 66%. However the less frequent classes have much much higher accuracy (still not perfect, but higher). How can I create model that will have the best of both? That will learn the less frequent AND the most frequent at the same time.

Also I have read that tree base methods perform very good on unbalanced dataset. However all examples always consider one big timeseries data. My data are many short timeseries. Is it possible to train RandomForest on such data? How?

If you know about different algorithm/method that could be applied to such data please post it :)

Thank you.


texts:
I'm trying to predict rare events, meaning less than 1% of positive cases. I basically try to predict if a subject will have 0, 1, 2 ... , 6, > 6 failures (there are cases in all those categories).

I've tried several algorithms:

decision trees
random forest
adaboost
grouping using k-means clustering and finding associations with failures (which group has most failure)

In any case, learning either goes to no failure or has too much variance (leading poor reasults on C.V. set).

Do you know any machine learning algorithms which are better suited for rare events?

Or is it surprising that I get those bad results using those algorithms, which means that my features list is not good?

Thanks a lot.


texts: Recommendation system with active learning

texts:
I have data where companies ask users to score a bunch of questions but some items may be randomly chosen while others are personalized. Users score higher in personalized questions on average. I have a user ID, question ID, corresponding score of the question by the user, and whether the question is random or personalized.

I want to build a recommendation system that incorporates the feature of a question being random or personalized.

I assume that for a personalized item to appear there must be some learning of previous random questions the company learned about the user beforehand.

But I got quite lost in terms of how to have a recommendation system that incorporates the dynamic structure.

I know basic recommendation includes matrix factorization or embedding for a user-item matrix but I don’t know how to accommodate the learning.

I would appreciate any insights/pointing to relevant literature/relevant code.


texts:
Say I have two email addresses and I would like to see if it is likely that they belong to the same person. For example, Cameron_M_Thompson@company.com and cthompson1024@personal.com is likely to be from the same person (it doesn't have to be certain, providing the likeliness would be sufficient).
I had two directions in mind to achieve this, one is a string comparison between the two email addresses and the other is to first extract the names from the email addresses then compare if they might be the same person. Like in the example above, the names extracted should be Cameron M Thompson and c thompson.
I am also wondering if given that one of the email addresses is guaranteed to contain the full name (usually company email addresses have the full name), would that help the extraction of name in the other email address (personal email addresses might not always contain the full name), or would that help on the comparison of the two email addresses.
I have had a hard time trying to figure out if any of the above two directions would be feasible. Especially when email addresses might not have separators and names can vary a lot that a listing might not be sufficient to find a match.
How should I proceed in solving this problem? Would machine learning / deep learning help or I should go with something else simple like regex and fuzzy string match?
UPDATE:
I have a dataset that has two columns, email address and name, and about 2k rows there. I believe this could be used for the second direction (name extraction). For the first direction (string comparison similarity), I am thinking of modifying the dataset to three columns (email address 1, email address 2, label of whether they are the same person), which should give about 1k rows of data.


texts:
So, I am currently learning machine learning and data analysis. I have created for my self a problem that is:
Who will win a match of soccer
Now, I have narrowed it down as being a Binary Classification problem as I only want to figure out who will be the winner of a match.
For this I have some data containing the following data points:

Team One
Team Two
Duration
Goal (Team One)
Goal (Team Two)
Winner

Now, this is where the trouble begins (for me at least) I am unsure which features to choose for my model and also kind of off as to how to manage the data.
Say that I have two teams playing (Liverpool vs Chelsea) now I do have around 5000 data points for all matches played however only around 82 points where it is Liverpool against Chelsea.
Which dataset should I use?
Also sometimes their position in the dataset changes meaning that sometimes Liverpool is Team One and sometimes Chelsea is Team One does this matter or should I process the dataset to always match one team at a certain position?
In general, what is the best way to train my model? Should I use the small dataset containing only the matches between these two teams or should I go for all matches in general?
I am sorry for the beginner question I really hope someone can help me out :)


texts: Question of pretraining text-generation task, it seems that pretraining is not work for a small model?

texts:
My task is to generate keywords from sentences.
I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens.
Pretraining batch_size = 8 and step = 1000000
I haven't observed improvement from pretraining. BLEU score is 10.5 for not pretraining, BLEU score is 9.5 for pretraining.
Code
I take the python code from
https://github.com/google-research/pegasus/blob/master/pegasus/models/transformer.py#L38
hidden_size = 512
num_encoder_layers = 3
num_decoder_layers = 3
Discussion
The task is to generate keyword from sentences.
The keyword may not appear in the sentences.
So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.
Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.
Am I right? Is it the reason that pretraining do not improve the BLEU score?
Thank you very much.


texts: Train-Test split for Time Series Data to be used for LSTM

texts:
values = df.values
train, test = train_test_split(values)

#Split into train and test
X_train, y_train = train[:, :-1], train[:, -1]
X_test, y_test = test[:, :-1], test[:, -1]

Executing the above code splits the time series dataset into training- 70% and testing 30%. I want to control the train-test split as 80-20 or 90-10. Can someone please help me understand what train[:, :-1] does in this context?
It is borrowed from https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/.
Note : I cannot split the dataset randomly for train and  test and the most recent values have to be for testing. I have included a screenshot of my dataset.
If anyone can interpret the code, please do help me understand the above. Thanks.