texts: Identify objects (bus) on the map based on coordinates (lat, lon)

texts:
Let's say I have an android app that frequently sends current GPS location of the user. If person is driving with bus, I can easily get GPS location of the bus and display it on the map and update it accordingly every 30 seconds.
Let's say that instead of one bus there are 3 buses driving along same route. My problem is to display real time location of every bus on the route. In this case there are X users on every bus all having my android application installed on their phones. Here is the picture that could represent the problem:

As you can see, every mobile application will send GPS location of the user to my server. Server receives and saves data.
Based on the server data I should create a web app that should display map whith position of every bus in real time.
One possible way to do this is to to try to cluster GPS data and based on clusters to conclude where is position of every bus. This approach has few problems:

This algorithm could be slow if there is lot of data (then I can't
update position of every bus every 30 seconds) Second problem is that
I don't know number of clusters (buses). There might be 2, 3, 4 or N
buses

Is there any other better approach to this problem?


texts: What is learnt by a model having multiple LSTM layers, each getting a slice of the same input data?

texts:
I came across the following architecture: -
Generation of training and testing data
# Some pre-processing of ASCII data

# Essentially, given 512 previous letters (X) predict the next letter (Y)
X = [...] # shape = (100,000 * 512)
Y = [...] # shape = (100,000 * 1)

Model
sequence_length0 = 512
sequence_length1 = 64

# first part of model
input_0 = Input(batch_shape=(batch_size, sequence_length0))
embedding_output_0 = Embedding(..., batch_input_shape=(batch_size, sequence_length0))(inputs_0)
lstm_0 = Bidirectional(LSTM(recurrent_units=32,return_sequences=True,return_state=True))(embedding_output_0)

# secondpart of model
input_1 = Input(batch_shape=(batch_size, sequence_length1))
embedding_output_1 = Embedding(..., batch_input_shape=(batch_size, sequence_length1))(inputs_1)
lstm_1 = Bidirectional(LSTM(recurrent_units=32,return_sequences=True,return_state=True))(embedding_output_1)

lstm_cat = Concatenate()([lstm_0, lstm_1])

# some fully connected network

output = Dense(..., activation='softmax')(...)

# model is constructed the following way. Notice that inputs is an array i.e inputs[0] will go to
# first part of model (lstm_0) and inputs[1] will go to other (lstm_1).
model = Model(inputs=[input_0, input_1], outputs=output)

Training loop
for batch_index in range(...):
  # Take a chunk of X equal to batch_size and put in X1
  X1 = X[start_position: start_position + batch_size] # shape = (batch_size, 512)

  # Take a chunk of X1 (take all rows and take 64 columns from right) and put it in X2
  X2 = X1[:,-64:] # shape = (batch_size, 64)

  model.train_on_batch([X1,X2], Y)

I was wondering what kind of information is learnt by this model? Like for example, the first part of the model might specialize in long term dependencies while other in short term (I may be wrong, just my intuition). Is the second LSTM layer worthless as it has a slice of the same input which the first part has?


texts:
I am a newbie in ML world, but very curious and enthusiastic about it. Have gone through articles and some hands-on too. Still got a silly doubt.

In sample datasets (like Iris or diabetes or breast cancer etc.) and exercises, I find the data to be well-formed and ready to be digested by ML model (algos). But, in reality, it is much more than that.

For example, Amazon's feature of suggesting (cross-selling) products. Based on my previous searches and orders, it probably would be looking into several features and predicting further items. I'm curious -

How does it work in real time?
How my product searches (they are not numbers) are fed into the ML model under-the-hood?

Please share your thoughts.

Regards,
RB
(using Python 3.6 & latest-1 versions of NumPy, Scikit and Pandas)


texts: Converting ML Azure random forest to Python - what is "Number of random splits per node"?

texts:
I want to convert a random forest model from ML Azure to Python (using SKlearn.ensemble.RandomForestRegressor. However there is a hyperparameter called "Number of random splits per node". There is no equivalent in SKlearn's random forest implementation.

Do you have any idea what it might refer to in SKlearn's random forest?


texts: Using Keras how and what do I need to export to use my classifier independently?

texts:
I have a basic question that I can't seem to find an answer to.

I built and trained with good results (above 90% accuracy) a NLP Log classifier that takes in a UTF-8 payload and classifies it into 32 distinct categories but I am having a hard time writing a simple script that loads all the necessary info from my training and testing session (model.h5 and ?).

This is the structure of my code.

# load data logs and split it 80-20 for training and testing
vocab_size = 500
tokenizer = text.Tokenizer(num_words=vocab_size)
tokenize.fit_to_text(trainRawLogs)
x_train = tokenize.text_to_matrix(trainRawLogs)
x_test = tokenize.text_to_matrix(testRawLogs)

encoder = labelBinarizer()
encoder.fit(trainRawLogs)

#Model build is simple ReLu - Softmax

model.compile..

model.fit..

model.evaluate..

Now here is my question.

Out of all of this process what do I need to save to build a lightweight classifier? The model? The model and the labels? Anything else? I tried loading the model


texts: Interpreting the results of randomized PCA in scikit-learn

texts:
I'm using scikit-learn to do a genome-wide association study with a feature vector of about 100K SNPs. My goal is to tell the biologists which SNPs are "interesting".

RandomizedPCA really improved my models, but I'm having trouble interpreting the results. Can scikit-learn tell me which features are used in each component?


texts: What should I use if I have millions of possible values for a feature in a sklearn predictive model?

texts:
I am trying to create a large model. One of the features is categorical, and it has almost 100 million entries.

I have looked at sklearn LabelEncoder, but I am concerned that it will still create an ordering in my labels which I would like to avoid. If I use one hot encoding I would end up with really large vectors with 100 million dimensions. What are my options?


texts: Can I make kaggle kernels read directly from my computer?

texts: For some reason jupyter notebooks never work the same as kaggle kernels for me. I want to use kaggle kernels but the downside is I don't know how to make it read from a file on the computer like a regular jupyter notebook


texts: LSTM training/prediction with no starting sequence

texts:
ML newbie here. As an exercise, I'm trying to build a character based language model based on a simple 1 layer LSTM. Based on what I've learned about LSTMs, a common usage is to take in a sequence of characters and then predict the next character. What I don't fully understand is how I would go about predicting the very first character when there's no preceding sequence yet (or, by extension, predicting a character when there's not a long enough preceding sequence as input to the LSTM units).

The best solution I can thing of is to reserve a special character in the vocabulary to represent the abscence of any characters. A toy example:

Full training corpus: "foo"
LSTM unit count: 2
"Absent character" symbol: ABSENT
"End of file" symbol: EOF
Training inputs:
    sample: [ABSENT, ABSENT] label: 'f'
    sample: [ABSENT, 'f'] label: 'o'
    sample: ['f', 'o'] label: 'o'
    sample: ['o', 'o'] label: EOF

My question is: what's the best practice for doing this type of thing? Am I on the right track?


texts: What to do when facial recognition fails to find a face?

texts:
I am currently implementing a CNN to recognise the identity of people given a portrait picture of the person.
The objective is to maximise the clf.score function in sklearn, the database is composed of 144 images of 70 different individuals. The program will be tested on a database of images of the same 70 individuals, which I do not have access to.

For most of the pictures, my facial recognition algorithm (openCV) will detect and crop out the background, leaving only the face, but in certain cases, it does not find a face.

What solution will return the best classification score?

Options I've considered are:

Leaving the picture as is (Useless background info isn't removed)
Cropping a predefined area (Could risk cropping out parts of the face)


texts: How to tackle too many outliers in dataset

texts:
I boxplot all of my columns with seaborn boxplot in order to know how many outliers that i have, surprisingly there're too many outliers and so i can remove the outliers because i'm afraid with too many outliers it will have bad impact to my model especially impacting the mean,median, variance which will further impact the performance of my model. Then i found this QnA about too many outliers the answer said that the case is not really an outliers. The answer makes sense but i'm still afraid it will give bad impact to the performance of my model. What is the best thing should i do?

These are two of examples of what i mean, 9 out of 10 columns of my data are exactly like this and i'm really worried about this because you see, it's not only there're too many outliers, it also happens in almost every (9 of 10 columns in total) columns.


texts: Lambda Architecture - How to implement the Merge Layer / Query Layer

texts:
I am reading up about lambda architecture.

It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.

So we have our entire data set. in our system. very good.

but the architecture diagram shows that the merge layer is able to query both the batch layer and the speed layer in one shot.

How to do that?

Your batch layer is probably a map reduce job or a HIVE query. The speed layer query is probably a scala program which is execution on the spark.

Now how will you merge these?

Is there any guidance.


texts: What is syntax V and S standing for nominal subject?

texts:
I was reading the recent paper https://www.aclweb.org/anthology/P19-1580.pdf and noticed that in section 5.2, the syntactic relation is studied in terms of the "direction between two tokens". In table 1, the result is further shown with direction like $$v \to s, s \to v$$ for nsubj(nominal subject).

However, what does V and S refer to here and where I can find more materials about this ?


texts: Train only Region Proposal Network in faster RCNN architecture

texts:
I am looking for a way to used my pretrained EfficientNetv2 model and turn it into an object detection.
Is there anyway, I can put my pretrained model as a backbone and only train the region proposal net work in any R-CNN family of architecture, preferably in Pytorch


texts: Handling huge dataset imbalance (2 class values) and appropriate ML algorithm

texts:
I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem is that there are only 725 OLD instances in the train set and 95 in the test.

The only algorithm which succeeds for me to handle imbalance is NaiveBayes in Weka (0.02 precision for OLD class), others (trees) classify each instance as NEW.
What is the best approach to handle the imbalance and the appropriate algorithm in such a case?

Thank you in advance.


texts: assignment of observations to known size classes

texts: Assume I have some observations and I want to predict their classes. For each observation my classification algorithm spits out probabilites for each class. Now assume I know exactly what distribution my prediction has to have. For example I know that 20 of my observations have to be in class 1, 15 in class 2 and so on. What is the best way to assign my observation based on their probabilities of belonging to a class. Does this problem have a name?


texts: Logistic regression prediction changed after executed couple of time

texts:
I noticed that after each time I execute the following lines of code, my results are different. Any idea why?
I think  that the main issue here is this line of code
But I dont understand why?

test_data['prediction']=sentiment_model.predict_proba(test_matrix)[:,0]

What is the best way to add new column with .predict_prob()?

test_data['prediction'] = sentiment_model.predict_proba(test_matrix)[:,0]
test_data['prediction_label'] = sentiment_model.predict(test_matrix)
test_data['prediction'] = test_data['prediction'].apply(lambda x: round(x,2))
test_data.sort_values(by='prediction', ascending=False, inplace=True)
test_data[test_data['name'] == 'Britax Decathlon Convertible Car Seat, Tiffany']
test_data.head()


texts: Algorithm for multiple extended string matching

texts:
I need to implement an algorithm for multiple extended string matching in text. Algorithms to match regular expression would be perhaps too slow.

Extended means the presence of wildcards (any number of characters instead of a star), for example:

abc*def //matches abcdef, abcpppppdef etc.

Multiple means that the search is going on simultaneously for multiple string patterns (not a separate search for each pattern), for example:

abc*def
abc
whatever
some*string

QUESTION:

What is the fast algorithm that can do multiple extended string matching?

Preferably, optimized for SIMD instructions and multicore implementation. Open source implementation (C/C++/Python) would be great as well. I'm interested in 10 Gbps+ performance on a single core of a modern CPU.

Thank you


texts: Need of maxpooling layer in CNN and confusion regarding output size & number of parameters

texts:
In my CNN architecture for binary classification, I have 2 convolutional layers, 2 maxpooling layers, 2 batchnormalization operations, 1 RELu and 1 fullyconnected layer.

Case1: When the number of channels, $d=1$:

In the first layer an input of size $[28*28*d]$, $d=1$ channel is convolved with $M_1=20 $ number of filters applied over all the input channels of size {$f_h \times f_w \times d$} = $[3\times 3\times 1]$ having the step size (stride) as 1 that creates a feature map of size ${(h-f_h+1) \times (w - f_w +1)\times d \times M_1} = (28-3 +1)\times(28-3+1)\times 10 = [26\times 26\times 20]$.

The second convolutional layer contains twice the number of filters = 40 of same size $[3\times 3 \times 1]$. So, the number of parameters becomes $[23 * 23 * 1 * 40]$ as the output from the second convolutional layer. So total number of parameters =  $[26\times 26\times 20]+  [23 * 23 *  40]$

Case 2: When $d=2$ and all other sizes are same. The filter size become $[3 \times 3 \times 2]$. The output of the first convolutional layer will contain: $(28-3 +1)\times(28-3+1)\times 2 \times 20 = [25\times 25\times 40]$. For the second convolutional layer, the output will contain $[23 \times 23 \times 2 \times 40]$ parameters.

Question1) Is my calculation for each case above correct?

Question2) I have read that the purpose of maxpooling is to reduce the dimensionality of the feature map. In my case each maxpooling is of size 3 and stride 2. What does it mean by reducing the dimensionality and then what will be the size of the output from each layer upon considering that there is a maxpooling operation after each convolutional layer?


texts: Why would removing a variable in adaboost decrease error rate?

texts:
I was trying to classify an outcome on some data using adaboost (the ada package in R) and I was playing around with the training data set of descriptors when I realized that removing a column in the descriptor matrix increased the accuracy of the output on the training data. Specifically, the number of false negatives dropped/true positives increased.

Aside from removing a single column in the descriptors, I left everything else the same, including number of iterations.


texts:
I have been implementing a convolutional neural net for Character recognition (MNIST) and I'm having troubles when trying to train it (basically the CNN doesn't learn). So I was wondering if the topology that I used is wrong...

CNN topology:

Input -> Conv layer 3*3 (10 filters)  -> Maxpool 2*2 -> Conv layer 4*4 (30 filters) -> Maxpool 2*2 -> Fully Connected layer -> Fully Connected layer -> Fully Connected layer (the result of this layer is the output).

As you see, I'm not using softmax, but I think that not using it should not be the problem.
I also use root-mean-square error as loss function.
Another thing I want to add is that, when I say that it doesn't learn, I mean that 10% of the time the output is correct, but I noticed that the output that is correct is when the image is a 9 (and not any other number).

If anyone could give me some advice about some CNN structure or about an existing CNN, I would thank you a lot.


texts: Why is large decision tree likely to overfit

texts:
My lecture slide told me that if we don't prune the regression tree, then the tree likely to over-fit.
So, I wonder why would that happen?
Is that because if the tree grows too large, we would end up with very little instances on each leaf nodes of the tree so the estimated mean value on each leaf node will be not accurate?


texts: Which model may be best for outcome of a surgery?

texts:
New to data science and am trying to be a self-starter and implement advanced data analytics in my subspecialty of surgery. Below is a description of my data set. I know that I will have to explore multiple methods, but wanted to get your take on which you think may be best. I will most likely be using R to achieve this analysis.

Have a data set with about 200 patients (rows)
Each patient has about 10-15 variables (preoperative and intraoperative)
Each patient has undergone either nonoperative or operative management
Success in nonoperative or operative management is determined by a questionnaire that patients fill out 1 year after they are seen. This questionnaire gives a binary outcome on whether they (1) Benefited or (2) Did not Benefit from the surgery.

My questions for the study are as follows:

In the surgery group, I am trying to find out which variables lead to patients (1) Benefit vs (2) Do not benefit from surgery, and create a model which can better help predict which patients we can operate on (I have left out some details such as patient population, type of surgery, etc).
In the second study, I would like to determine which patients we should operate on. In other words, I would like to find out which preoperative characteristics make some patients more likely to benefit from (1) Operative treatment vs (2) Nonoperative treatment and in this case the outcome will also be binary from the questionnaire.

I have tried linear and logistic regressions for this which have not been very good, hence why I am trying to learn more advanced models.

Models which are easier to comprehend by clinicians are more valuable which is why I haven't delved into neural nets. I appreciate any and all advice that can be provided. In addition, if I expand this data set to 600 people, would you use another model? I don't have access to large servers so most of this will be done on my laptop though I can use online resources if necessary (Azure etc).

Thank you all for your help and input.


texts: How do Classification Algorithms such as Catboost and Random Forest parse test data?

texts: I would like to know how classification works with the algorithms listed above. My specific question is this, say I have a high signal continuous feature which has a certain distribution and I train a model according to some training data and it finds the best split for that feature. When I use the model on test data, would it split according to a specific number or by distribution? i.e if the number '10' provides the best split for the training data which happens to be 75th percentile. When it comes to test data will it split according to the percentile or the number 10? I hope that is clear.


texts: How regularization helps to get rid of outliers?

texts:
I have heard regularization helps to get rid of outliers, how so?
'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'.
Could you shed some more light on it?


texts: Spark ALS-WR giving the same recommended items for all users

texts:
We are trying to build a recommendation system for a supermarket with diverse item types (ranging from fast-moving grocery to low-moving electronic items). Some items are purchased more frequently in high volume and some items are purchased only once.

We have purchase history data of 4 months from 25K+ customers across 30K+ SKU's from 100+ departments. We ran ALS-WR in Spark to generate recommendations. To our surprise, we are receiving top 15 recommendations for each customer quite generic without much variation.

We have tried several means to diversify the recommendations as below-
- computed "rating" = normalized # of purchase
- computed "rating" = log of # of purchase
- computed "rating" = 1 (if purchase # > 1)
- We have used following combination of parameters - lambda = 0.01 to 300, alpha = 5 to 50, rank = 10, 20, 30 and # of iterations = 10, 20
- preference considered is explicit.

Do you think that ALS can be used for such heterogeneous data? If yes, what modifications will make the recommendations diverse & personalized?


texts: Don't know how to preprocess my dataset for image classification

texts:
I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy.

I believe the issue might be with my data preprocessing. My dataset directory contains a Training and Test folder but no validation folder (I have to split the dataset myself) and they are layed out like this:

Training
├── class0
│   ├── image1
│   ├── ....
│   └── image20
│
├── ....
│   ├── image1
│   ├── ....
│   └── image20
│
└── class9
    ├── image1
    ├── ....
    └── image20

Here is my AlexNet model, the code is below:

model = tf.keras.models.Sequential([
    #Conv_1          #original model was built for input shape of 224X224
    tf.keras.layers.Conv2D(96, (11,11),strides=4, padding='valid', activation='relu', input_shape=(224, 224, 3)),
    # Pooling_1
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    # Batch Normalisation_1
    tf.keras.layers.BatchNormalization(),
    # Conv_2
    tf.keras.layers.Conv2D(256, (11,11),strides=1, padding='valid', activation='relu'),
    # Pooling_2
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    #Batch Normalisation_2
    tf.keras.layers.BatchNormalization(),
    # Conv_3
    tf.keras.layers.Conv2D(384, (3,3),strides=1, padding='valid', activation='relu'),
    # Batch Normalisation_3
    tf.keras.layers.BatchNormalization(),
    # Conv_4
    tf.keras.layers.Conv2D(384, (3,3),strides=1, padding='valid', activation='relu'),
    # Batch Normalisation_3
    tf.keras.layers.BatchNormalization(),
    #conv_5
    tf.keras.layers.Conv2D(256, (3,3),strides=1, padding='valid', activation='relu'),
    #pooling_3
    tf.keras.layers.MaxPooling2D((2, 2), strides=(2,2),padding='valid'),
    #Batch Normalization_4
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Flatten(),
    #Dense layer_1
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    #Dense layer_2
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    #Dense layer_3
    tf.keras.layers.Dense(1000, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(1, activation='sigmoid')
    ])

As for what I did to preprocess my data:

train_dir = '/content/gdrive/My Drive/x/Training'
test_dir = '/content/gdrive/My Drive/x/Test'

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest',
      )

test_datagen = ImageDataGenerator(rescale=1./255, validation_split=0.4)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        test_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary',
        subset='validation')

model.compile(loss='binary_crossentropy',
              optimizer=RMSprop(lr=1e-4),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=10,
      validation_data=validation_generator,
      validation_steps=10,
      #callbacks = [checkpoint],
      verbose=2)

Any and all help will be immensely appreciated! Thank you to anyone who may reply.


texts: Unbalanced class: class_weight for ML algorithms in Spark MLLib

texts:
In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.

However, I do not find such parameter for the MLLib algorithms. Is there a plan of implementing class_weight for some MLLib algorithm? Or is there any approach in MLLib for unbalanced data? Or we actually have to handle all the up/downsampling ourselves in MLLib?

Thanks!


texts: Numpy element wise comparison for a particular value in two arrays

texts:
If I have two arrays as shown below:

a = numpy.array([0, 0, 1, 0, 1, 1, 1, 0, 1])
b = numpy.array([1, 1, 1, 0, 0, 1, 1, 0, 0])

Is there an easy way using numpy to count the number of occurrences where elements at the same index in each of the two arrays have a value equal to one. In the above two arrays, the elements in position(zero-indexed) 2, 5 and 6 are equal to 1 in both the arrays. Thus I want to get a count of 3 here.

Thank you for any help that you may be able to provide.


texts:
I am fitting mixture models to data and assessing how mixtures with more or less components will fit the data. To do this, I am going to plot the cdf of the empirical data and the cdf of my mixture model with k components. As an example, here is a cdf of the empirical data plotted beside a mixture of lognormal distributions with 2 components.

My question is: how do I use scipy's kstest to determine the goodness of fit for the mixture model on the empirical data?
ss.kstest(Y,y_cdf)

Above is the code that I tried where Y is the data I used to fit the model and make the empirical cdf and y_cdf is the cdf of the mixture model.
I am unsure if this is correct as the value for D returned seems quite high.


texts: unsupervised learning in medical systems and intelligent systems?

texts:
I have a dataset which belongs to a hospital. It contains data about patients and healthy people. The problem is separating healthy ones from patients. I add some new features to dataset to solve this problem. When I reduce the dimensions of data including the new features and visualize the data, the patient and healthy individuals are distinguishable(visually separable). Now if one asks what is the relation between the used approach (feature extraction, visualization, using the human ability, unsupervised methods) and expert system, what can I say? How can I use this method to build an expert system?

Thank you in advance.


texts:
Suppose we want to predict what customer will buy during his next visit to the Electronic Shop based on his past purchase history. I know that it is a very broad question, but I am new to machine learning and don't have much idea about how to approach this problem.

The simplest thing that comes to my mind is to find the most frequent items that customer has bought and suggest it. However, I don't think that this is a very robust approach as it doesn't consider this scenario:

Computer (1st Purchase) -> Mouse (2nd Purchase) -> Mouse Pad (3rd Purchase)

I am looking for a simple model to get started and scale in terms of features and training data. I would love to hear suggestions of experienced Data Scientist as it is a most common problem.

Thank you.


texts: Error on custom RNN/LSTM with multiple inputs

texts:
I want to implement a custom RNN/LSTM model similar to this. The model should take two separate vectors as input and process them. I was following keras tutorial to implement a custom keras layer and inputting two vectors a and b as a list [a,b] to the layer as shown below.

import keras
from keras.layers.recurrent import RNN
import keras.backend as K

class MinimalRNNCell(keras.layers.Layer):
    def __init__(self, units, **kwargs):
        self.units = units
        self.state_size = units
        super(MinimalRNNCell, self).__init__(**kwargs)
    def build(self, input_shape):
        print(type(input_shape))
        self.kernel = self.add_weight(shape=(input_shape[0][-1], self.units),
                                      initializer='uniform',
                                      name='kernel')
        self.recurrent_kernel = self.add_weight(
            shape=(self.units, self.units),
            initializer='uniform',
            name='recurrent_kernel')
        self.built = True
    def call(self, inputs, states):
        prev_output = states[0]
        h = K.dot(inputs[0], self.kernel)
        output = h + K.dot(prev_output, self.recurrent_kernel)
        return output, [output]

# Let's use this cell in a RNN layer:
cell = MinimalRNNCell(32)
a = keras.Input((None, 5))
b = keras.Input((None, 5))
layer = RNN(cell)
y = layer([a,b])

But I am getting the error TypeError: 'NoneType' object has no attribute '__getitem__' at

self.kernel = self.add_weight(shape=(input_shape[0][-1], self.units),
                                          initializer='uniform',
                                          name='kernel')

Also the type of the input_shape is showing as <type 'tuple'>, not a list.
Whats wrong I am doing and how to overcome this error. Please help


texts: How to apply dataset balancing techniques whilst using Pipeline in Sklearn?

texts:
I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's Pipeline class to ensure that my model is not prone to data leakage. I am building an ML model that attempts to predict the trend Buy, Hold, Sell for the next hour.

However, my multi-class classification dataset is extremely imbalanced. Whilst it is not necessarily a concern that the test set is imbalanced, it is important that the train set is balanced. However, I have researched properly but I cannot find an answer as to where this dataset rebalancing step should be conducted. Should it be done before scaling or after? Should it be done train/test split or after?

I cannot figure out where this crucial step should be done. For simplicity's sake, I will not be using SMOTE, but rather random minority upsampling. Any answer would be greatly appreciated.

My code is as follows:

#All necessary packages have already been imported

x = df['MACD', 'MFI', 'ROC', 'RSI', 'Ultimate Oscillator', 'Williams %R', 'Awesome Oscillator', 'KAMA',
    'Stochastic Oscillator', 'TSI', 'Volume Accumulator', 'ADI', 'CMF', 'EoM', 'FI', 'VPT','ADX','ADX Negative',
    'ADX Positive', 'EMA', 'CRA']

y = df['Label']

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state = 0)

pipe = Pipeline([('sc', StandardScaler()),
                 ('svc', SVC(decision_function_shape = 'ovr'))])

candidate_parameters = [{'C': [0.0001, 0.001, 0.01, 0.1, 1, 2, 3],
                        'gamma': [0.0001, 0.001, 0.01, 0.1, 1, 2, 3], 'kernel': ['poly']
                        }]

clf = GridSearchCV(estimator = pipe, param_grid = candidate_parameters, cv = 5, n_jobs = -1)

clf.fit(X_train, y_train)```


texts: Average n 2D clusters into one finale result

texts:
I have n clusters run on 3D data, resulting n 2D clusters, I couldn't run the clustering model on a one year satellite mages time-serie. So I did chunks of one month and run clustering which finally gave: 12 sets of 2D clusters like:

Set 1:
lat, lon, class
10, 10, 1
10, 20, 2
20, 10, 3
...

Set 2:
lat, lon, class
10, 10, 2
10, 20, 2
20, 10, 3
...

...

I would like an approximate for what it would be to run the clustering model on 365 data, in other words the best average of the clusters. Even if it does not take into consideration the time order between these clusters. It would be less correct for a clustering on all data but still fair.


texts: How to visualize categorical data over time?

texts:
I'm visualizing results from lab tests. Most of these results are numerical and therefore easily communicated using line graphs, where the x-axis is time and the y-axis is determined by the range of possible values, like [0, 100].

EDIT: To be clear, I'm not measuring frequency. I'm displaying an individual's lab test results for one specific test (like cholesterol) over time. Since most people don't get more than 1 or 2 lab tests a year, most people's graphs will have less than a dozen (x,y) values.

I'm less sure how to visualize this information when the y-axis values are categorical:

Sometimes the possible values are themselves ranges. A test value will be something like 2-6, meaning that the value falls between 2 and 6 — and that's all we need to know. Would it make sense to graph these values with something like error bars? Or would it make more sense to graph the average of the range (in this case, 4) as a regular line chart?
The other types involve categories which are non-numerical. Sometimes these are ordinal. In the ordinal case the categories might be something like [none, few, moderate, many]. Does it make sense to just assign an index to each [0, 1, 2, 3] and use a standard line graph? If not, what would make more sense?
Lastly, non-numerical, nominal types. These are categories with no clear rank hierarchy, like colors: [yellow, orange, red]. In this case, I think it might be misleading to use a standard line graph, because it would imply that certain colors are "higher" or "better" than others. What would be appropriate?


texts: How do i perform feature selection for clustering?

texts:
I am new to Data science and trying to learn clustering? I have to partition the given dataset into different clusters into customer clusters based on their purchasing habits? How do I select the features that important for clustering?

Heirarchy_1, Herarchy_2 etc describe the product.


texts:
I am trying to classify a series of timestamps using RNN with LSTM. The data consists of timing information extracted from the uplink packets recorded during a website fetch. The dataset contains 100 individual fetch samples of each website in a set of 100 websites, this gives me 10,000 samples.

I would like to teach a neural network to classify sequences of timestamps to which website they belong.

An example sequence would look like:

0.0, 0.25420099, 0.70250899, 0.7434534, 0.8746745, ... 2.54634634

Each of these values represents an offset from the start of the website fetch.

These all differ in lengths and are between ~300 and ~4500 timestamps.

I have tried training the LSTM network in Keras like below:

modelClass = Sequential()
modelClass.add(LSTM(100))
modelClass.add(Dense(1, activation='sigmoid'))
modelClass.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
modelClass.fit(X_tr, y_tr, epochs=3, batch_size=64)
print(modelClass.summary())
scores = modelClass.evaluate(X_tst, y_tst, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))

Training data is in the shape of (9000,  maximum_sequence_length, 1), however this gave me very bad results. I am new to machine learning and don't fully understand how to find appropriate algorithms for specific tasks. Google searches did not come up with any ways to classify such data.

I would appreciate any suggestions and directions, please let me know if more information is required.


texts: Extracting useful features for k-means clustering

texts:
So say suppose I have a data-set with features being either present or not i.e. 0or 1. Now I want to identify the features which really help in  the clustering.

Like say I have 4 training examples. Now say I have a feature which is  present i.e = 1 in all the training example, thus I can remove the feature as it dies not help me. Now let's talk about 2 more features, if the number of training examples they are present in common is high, they also do not help much in clustering (think of 2 highly overlapping circles in a Venn Diagram). So in this way I want to find features which has a significant impact on the clustering i.e mostly non-overlapping features.

Is there any good way to do this? (my features are all represented in binary, either it is there or it is not).


texts: In machine learning how to find feature interdepencies?

texts:
Given a data set of N features, wherein some the features in this set were derived from other features from the same set, I am trying to discover inter dependencies between features (something like this Input feature(s) -> output feature(s)).

Note that,there can be multiple dependencies in the same feature set. Can someone suggest some technique to approach this problem.


texts:
I'm training a Neural Network for pattern recognition. I have a matrix of examples of size ($N$x$4$) with $N$ examples and $4$ variables.

When I train the Network the Number of examples used for training vs. Cross-entropy, the performance of the Network it's not improving when adding new examples.

I suspect that the examples are highly correlated between them and adding new examples is not giving new information to the network, but I don't figure how to measure that correlation.

Each row of the matrix can be labeled $\mathbf{X}_{i}$. Suppose zero mean, the covariance matrix between examples would be $\mathbf{R}_{ij} = E[\mathbf{X}_{i} \mathbf{X}_{j}^T]$.
I can also calculate the covariance matrix of $\mathbf{X}$, whose elements are $\mathbf{R}_{ij}$ but in that case I would have a matrix of $\sim 300000$x$300000$.

Is there a way of calculate some average correlation or an other measure of correlation between examples?.

Thanks.

EDIT: I don't want to know the correlation matrix of variables, I want to know the correlation between examples.

EDIT2: This is the learning curve:


texts: Comparison of dispersion measures of different samples with different subsamples

texts:
Context: I have data on movie ratings and I want to find what factors affect the rating the most. I have $k$ features $F_1, F_2,\ldots,F_k$ characterizing a movie - for example, director, lead actor, genre, etc.

Now for each feature value, I need to get a representative rating. For example, I'll fix the "value for director feature" (say $F_1$) as Steven Spielberg and calculate the median of the ratings of all movies by Spielberg (avoiding mean due to potential outlier ratings). This will give me a representative rating of a movie directed by Spielberg. I can do the same thing for other directors, getting representative ratings for movies directed by each.

Notationally, let the values of $F_1$ be $F_1^1,F_1^2,\ldots,F_1^{n_1}$ (different directors). More generally, the different values of feature $F_i$ are $F_i^1,F_i^2,\ldots,F_i^{n_i}$. For each feature value, we calculate the medians of all movies having that feature value: $M_i^1,M_i^2,\ldots,M_i^{n_i}$. Next, I calculate the coefficient of variation of these medians for each feature. e.g. for $F_1$, I'll calculate CV of $M_1^1,\ldots,M_1^{n_1}$.

If the CV of medians for $F_1$ is high, that means the representative rating varies substantially as we change the directors, which in turn means that director is probably an important factor in deciding movie rating. On the other hand, if $F_r$ feature corresponds to the distributor, and if the CV of medians $M_r^1,\ldots,M_r^{n_r}$ is low, that means even if we vary the distributor, it doesn't impact the representative rating that much, and hence distributor is probably not an important factor for rating.

Question 1: Some features may have several possible values (e.g. director) - in that case I'll have enough representative values, and I'm comfortable taking the CV because I have a good enough "sample of medians". But what about features with very few values? e.g. broad genre category may only have $5$ or $6$ or $7$ values. I may even have a feature like "International / Domestic" that only has $2$ values. Any suggestions on how to deal with such cases?

Question 2: What other caveats should I be looking out for? Any problems or major fail conditions with the overall approach? Again, would appreciate any tips/suggestions or modifications to the approach.


texts: Clustering mixed data: Find optimal number of clusters

texts: What do you say about this plot to find the number of cluster for kmean or kproto for mixed data. Where is the elbow to identify? I would say 5? I have 11 feautures.


texts: Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

texts:
It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model algorithm on it (e.g. kNN, regression models, etc) using something like Pandas, R or Matlab. In other words one operates within the DBMS and then operates outside of it.

One can use User-defined Functions (UDF) to implement the algorithm of choice in many languages (build a UDF in C using e.g. PostgreSQL) and remain operational within the DBMS environment, i.e. one does not give up the benefits obtained of operating inside of a DBMS. Furthermore, one can seamlessly integrate the UDF with the SQL construct, which is very powerful if one requires to perform subsequent SQL operations on the data which passed through a model.

I was wondering what is the reason behind operating outside of the DBMS given that such functionality is available and that the DBMS is, ironically, good at managing data (supported by many years of research).

There are other discussions which seem to conclude that SQL is for preprocessing and Pandas is for data analysis; to clarify I am building on this conclusion: why did things end up this way, when the DBMS is very well-suited for data analysis as well?


texts: attach "1" for the bias unit in neural networks. what does it mean?

texts: In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network?


texts: QuantileTransform before or after long-to-wide format transformation?

texts:
I perform the following steps:

Perform quantile transform
Transform it from wide to long format
Train the anomaly detection model on the wide-format data

Will there be some significant change if I, instead, first perform a wide-to-long transform, and after perform the quantile transform?

Since the normalization is column-based, I understand, that the average will be slightly different across the columns, so the normalization might produce a bit different results. But would it really affect the results a lot, or it does not really matter?

Example of data:

Features: daily temperature, daily number of routes taken by the machine, daily movements in some parts of the machine (eg. number of switches).

Data contains a lot of outliers, that was motivation to select Quantile Transform.

Analysis will be done on 4-days window, hence the transformation to the wide format.

For example, consider one feature in the long format. If I perform some normalization procedure on this feature in the long format, all the numbers will be taking into account for computing std/mean/quantiles/...

Temperature
===========
15
21
23
30
24
14
30
19
20
23
31
19

However, if I transform to the long format:

Temperature_1   Temperature_2   Temperature_3   Temperature_4
=============   =============   =============   =============
15              21              23              30
24              14              30              19
20              23              31              19

So, now, if apply the same normalization procedure, it would calculate the statistics (std/mean/quantile depending on the chosen method) on each column separately.

Would it be safe to assume that in case of relatively big amount of data (eg. 100,000-500,000 rows) these differences would not matter, or is it safer to first perform the normalization, and then transform the data to the wide format?

Thanks.


texts: Looking for a machine learning algorithm that can classify line plots into two groups?

texts:
I have a dataset of many 2-D line plots that represent height versus position measured from an AFM.
I am interested in training a model by classifying these lines into two groups: presence of a defect and no presence.
I believe a neural network would be appropriate for this since there are other features in the line plots that I want to be able to ignore.
The problem is that I am just starting by taking a certification course on machine learning but that hasn't given me enough information to develop one on my own. I have worked with a segnet based neural network but at a very user oriented level.
So I was wondering if anyone knew of an open-source algorithm that I could feed these lines into (maybe as vectors?) to train a model?


texts:
I have a dataset that looks something like this;

ID    | Location | Job_title | blue_jumper | red_jumper | yellow_jumper | green_jumper | Target(purple_jumper)
-------------------------------------------------------------------------------------------------------------------
B7372 | Rome     | Builder   | 2           | 1          | 0             | 9            | 1
D9823 | London   | Lawyer    | 0           | 1          | 8             | 3            | 0
E8718 | Rome     | Teacher   | 3           | 0          | 2             | 3            | 1

etc etc.....

What I would like to do is to use this information to predict whether a person will have a Purple jumper (Target 1 or 0)

Things to note about this dataset I think are the following;

I have an ID that relates to the individual
I have a number of catergorical features
I have some information relating to features that are of the same type (jumpers) but differing by some aspect (colour)
These features are of the same type as the Target (e.g jumpers)
The target is binary (e.g. I am not looking to predict how many Purple jumpers a person has, just whether they have one or not)

As the Target is binary I know I could use a classification method but I have decided to use Multiple Linear Regression. I like this algorithm because it gives me a measure out that is equivalent to how much like a 1 or 0 my record is.

I have generated dummies for my categorical features but what I am struggling with is whether or not to scale my other features in a situation like this.


texts:
On pg. 206 of Barto and Sutton's Reinforcement Learning, there is a curious statement about the result of a scalar product:

As I interpret it, A is the expectation of a scalar product of two d-dimensional vectors: which should be a scalar, right? So how do they get a dxd-matrix from it? Is it a shorthand for a scalar matrix (diagonal with the repeated coefficient, namely this scalar product)?


texts: How to best represent rate or proportion as a feature?

texts:
Given two features in a data set: NFObs (Number of failed observations) and NObs (Number of total observations), I'd like to better understand how I should represent these data for downstream use in binary classification problem using a decision tree model.

Should the features be left "as-is" or substituted for a new feature FailRate equivalent to (NFObs / NObs)?  My concern with the rate/proportion idea is that it the model will consider an instance originally 1/1 the same as an instance originally 500/500.  However, someone interpreting the results of the model would value an instance with 500 observations much more than an instance with 1 observation.

NFObs < NObs and NObs >= 1

Problem Description

Each instance contains:

A unique id uid, representing some widget.

Two independent, quantitative measurements feat1 and feat2.

class is 1 if an instance (observation) "failed" and 0 if the instance "passed".

NFObs is a count of how many times the uid has previously been reported 1. (stored in some legacy db without access to feat1 or feat2).

NObs is a count of how many times the uid has previously been reported either 1 or 0.

Hypothesis is that a model can predict whether a widget will fail using historical fail rate and newly collected feat1 and feat2

In [2]: df.head()
Out[2]:

class   uid  feat1  feat2   NFObs   NObs
   1    001   63     21      500    500
   0    002   31     29        1      1
   1    003   38     56        1      1
   0    004   33     86        5     20
   0    005    7     35       13     77

In [3]: dfRate.head()
Out[3]:

class   uid  feat1  feat2   FailRate
   1    001   63     21       1.0
   0    002   31     29       1.0
   1    003   38     56       1.0
   0    004   33     86       0.25
   0    005    7     35       0.16

Any insight or suggested reading would be helpful.


texts: Augmentation on test dataset and validation dataset

texts:
I'm training a segmentation model (computer-vision).
Thus, my dataset contains images and masks (binary segmentation of objects).
I'm augmenting the training dataset (applying random crop, rotation or shift etc.) to get a larger dataset. I don't apply augmentation on test and validation dataset.
Should I use augmentation on the validation dataset or the test dataset too ?


texts: How does one-sided label smoothing make the discriminator more robust by reducing the confidence in correct class?


texts: Scikit-Learn - Learned model description?

texts: Is there a way I can "look inside" a model once it's trained? For example, if I train a spam filter with a multinomialNB, is there a way I can extract which words are most likely to make an email classify as spam? I'd like to see how the models determine the outcome once fitted.


texts:
I have the following sample dataset (the actual dataset is over 10 million records)

     Passenger    Trip
0       Mark     London
1       Mike     Girona
2    Michael      Paris
3        Max     Sydney
4     Martin  Amsterdam
5     Martin  Barcelona
6     Martin  Barcelona
7       Mark     London
8       Mark      Paris
9     Martin   New york
10       Max     Sydney
11       Max      Paris
12       Max     Sydney
...
...
...

And I wanted to get the destination frequently travelled by a passenger !

I was playing around in Jupyter and got the expected data with the following approach

series_px = df_px_dest.groupby('Passenger')['Trip'].apply(lambda x: x.value_counts().head(1))

df_px = series_px.to_frame()

df_px.index = df_px.index.set_names(['UName', 'DEST'])

df_px.reset_index(inplace=True)

def getNextPossibleDestByUser(pxname,df=df_px):
    return df.query('UName==@pxname')['DEST'].to_string(index=False)

While the response is fine. I have few doubts now

1) What's the best way to expose the method (say in this case getNextPossibleDestByUser) as a API (pass customer name as input and get the destination as output) ?

2) Whenever the API is being called , does that mean all the 10 million records gets processed each time ? Are there anyway to optimise that ?

3) Rather than dataframe (pandas) query approach can I consider some ml models or utility functions from say scikit to solve the same problem ?


texts:
I'm dealing with textual data. Imagine a certain number of statements (they could be descriptions), each one referring to a certain process, where each process ends in one f three categories: A, B, C.

I want to characterize the descriptions of type A, the ones of type B and the ones of type C. Una possible way is to focus on the terms contained in the description.
How can i represent them? One possibility is to compute a document term matrix and once each term is characterized by a vector, compute the distance among all the terms. Starting from these distances it is possible to find a minimum spannign tree connecting the terms and recostructing someway the concepts contained in the original descriptions. Doing that only for the description of type A, etc., we could reconstruct graphically what the questions of type A, etc. are about.

The question is: once the concepts are reconstructed... are there other evidences that can be extracted by a network analysis?


texts: Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which contains a few hundred images.


texts:
Given a time-series prediction with a Recurrent Neural Network (doesn't matter if LSTM/GRU/...), a forecast might look like this:

to_predict (orange) was fed to the model, predicted (purple) is the forecast resulting from the RNN-model and correct (dashed blue) is how it should have benn forecasted correctly.

As can be seen, to_predict (as well as all the training-data) is quite "spiky", while the forecast is much smoother. The smoothness is presumably the result of the models architecture etc.; anyhow, my question aims somewhere else (even though connected to this):

Is a smooth forecast that more or less middles the zigzag of peaks and valleys of the correct data a hint for...

... a generalizing model, ...
... or is the absence of such zigzags a sign that its forecast-quality is low ...
... or neither (respectively no conclusion can be drawn at all)?

I intentionally did not mention any criteria like MAE, MAPE etc. since I am only concerned with this graphical interpretation.


texts:
I am considering the following hypothetical situation: I have a time series of data. In general, 'the public' should have access to features of this data. However, making the time series available would constitute a privacy leak. I am considering making a moving average available instead.

Can anyone recommend either some literature on this, or some alternative methods?

I understand that this is a case by case question. However, I think there should be a general answer available along the following lines:

1) Privacy leaks occur because you can match up the time stamp to an individual, by using outside information.

2) Therefore, you want to make it so that each window aggregates the data of several individuals. (The data is of a form where the mean is a meaningful quantity.)

There are of course ways to break this privacy, if one is sufficiently determined. I think in this case no one is. So I'm looking for literature that deals with some real world case studies, if possible.

(This situation is hypothetical. I do not have access to the data. I am 'the public' that wants the data, and I want to suggest a reasonable approach for aggregation.)

I also asked this question over on cross validated: https://stats.stackexchange.com/questions/324204/privacy-through-moving-averages


texts: How to visualize results/errors of multilabel classifiers?

texts:
For multiclass classification you would normally choose a confusion matrix to plot the error of predicted classes against the target classes.

What is the best way to visualize errors of multilabel classifiers? As multiple classes are predicted at once a mapping of prediction against target is not always possible so confusion matrices are generelly not suitable.

My first idea is to plot a bar chart where each class has a bar for missed and one for false predictions. But is there any standard for visualizing the errors which includes more information just like a confusion matrix for multiclass problems?


texts: What do I call this visualization? Specific point co-ordinates don't matter as long as they are within the correct labelled area

texts:
I have a list of products along with the stage of development they are in - there are several possible stages.

I would like to display the products as labelled points arranged into an area which represents the stage of development. As long as the products are within the boundaries of the area, the specific arrangement of points within the area is irrelevant. However, the points will need to be labelled with the product name, and these labels should be legible.

The areas would be ideally arranged to show progression of the product stage. Two examples of this would be:

Boxes arranged in a line, with product stage increasing left to right
Concentric circles with product stage increasing from the outer
circle to the inner circle

I have two questions:

What do I call this visualization?
Is there a program that will create this visualization for me from a
list of product info and associated stages?

I would prefer to pull the data in from Excel and have a GUI, but I am happy to Google some code if necessary

Example:

 _______________________________
| Prod7    |         |          |
|          | Prod3   |Prod1     |
|   Prod2  |         |          |
|Prod4     |  Prod5  |    Prod6 |
|__________|_________|__________|
   Stage 1   Stage 2   Stage 3


texts: What is the best method for classification of time series data? Should I use LSTM or a different method?

texts:
I am trying to classify raw accelerometer data x,y,z to its corresponding label.

What is the best architecture for best results?

Or, does anyone have any suggestions on LSTM architectures built on keras with input and output nodes?


texts:
I have a question about types of RNN. Ian Goodfellow in his book Deep Learning writes:

  Some examples of important design patterns for recurrent neural
  networks include the following:

  • Recurrent networks that produce an output at each time step and have
  recurrent connections between hidden units, illustrated in figure
  10.3.

  • Recurrent networks that produce an output at each time step and have recurrent connections only from the output at one time step
  to the hidden units at the next time step, illustrated in figure 10.4

  • Recurrent networks with recurrent connections between hidden units,
  that read an entire sequence and then produce a single output,
  illustrated in figure 10.5.

  CHAPTER 10. SEQUENCE MODELING: RECURRENT AND RECURSIVE NETS

Next I read about RNN article by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/ and his describe rnn architecture like relation model.

My question is: description of types RNN by Ian Goodfellow are equal to Andrej Karpathy? If not what it the diffrence beetween this descriptions?


texts: Spark MLlib recommendation - restaurant/ item similarity - issues/improvement

texts:
Use Case

Recommend similar items (restaurants) to diner.

Solution:

We have used apache spark MLlib ALS algorithm. Values for lambda and rank has been obtained by iterating through all permutations of combinations on selected range of ranks (9-20) and lamda(0.0001-3). We have kept iterations constant (10). In the end, we evaluated the rank-lamda for which the error is minimum  for our dataset and used it .

  Note - Training set - booking matrix for time range say November
  2015-March 2016. Test Set/Cross Validation Set - booking matrix for
  the same diner set for time range April 2016 - May 2016. This is just
  example, we have followed 70/30 ratio.

Approaches followed:

Approach 1 -
input data consists dinerID,restaurantID,rating. rating in this case is the number of time a restaurant has been booked. 0 rating if it has not been booked.

Approach 2 -
Instead of number of times a restaurant has been booked, we have rating as 1 if diner has booked a restaurant.
Using Approach 2 as it had far less error rate.

Considerations/Tail cutting:

Only active diners and restaurants considered
Diner who has booked more than 1 restaurant has been considered
Restaurant which has been booked more than once is considered.

Implementation:
These 16 latent factors (rank - 16 in our case) for every restaurant/item is obtained and persisted in database. Since our dataset is not big (~25k items), on runtime euclidean distance for each is calculated and top 5 are recommended for each restaurant.

Problem:

Restaurants are city specific and so are diners. But while building model as well as creating input matrix, city or any other explicit factor is not taken. I believe spark should be identifying a pattern and one of the factor could be city as well.
Now, out of X number of cities we operate in, 70%-80% of our business is driven from a city A. A small number from rest 2-3 cities and very few from the rest.
Recommendation for city A is very good. Also, the recommendations for other 2-3 cities seem to be decent to normal. Other cities, honestly I have not evaluated.

Pain point:
Since model is trained irrespective of city, I assume the 16 factor vector for each restaurant is a good metric to compare item item similarity. So, if my last booking is a restaurant abc in city A and next time I am in city B (I am new to this city btw) and looking for similar restaurants. What I have implemented is item-item for restaurant abc and restaurants in city B.
Whenever similar restaurant to any restaurant from city A(the major driver) is calculated for any other city, irrespective of which restaurant from city A has been selected, recommendations are same. However, there are slight variations if instead of city A there are comparisons between city b/c/d/e.

What could be the possible reasons to this?
Is it because of heterogeneous/skewed data?
Ways to improve recommendations?
Can we work on individual latent factors (for example if it is identified that factor2 is city, factor 3 could be cuisine, then can we manipulate??)
Any other way to improve or test model?


texts:
I have gathered a large amount of qualitative data and am now looking to cluster it so as to make sense of it. For this, I am using Biolab's Orange.

In my data, specific values may co-occur in a given feature, or they may not. I am wondering how I could cluster the data (either in Orange or other software), where values that co-occur would be seen as two values, rather than one string.

To make matters clearer, imagine I have a feature X, with the possible values A and B. The values can occur in the following way: A, B, A and B.

Question: how can I cluster my data without "A and B" being treated as a separate string, but rather "A" and "B" co-occurring?


texts: Keras Loss Function for Multidimensional Regression Problem

texts:
I am new to DL and Keras. I am trying to solve a regression problem with multivariate outputs (y shape (?, 2)) using Keras (tensorflow backend). I am having my confusion about how the loss is calculated. I use mean absolute error as the loss function. However, since my target data has 2 dimensions, is the loss value calculated as the reduced mean on all dimensions (a scalar as the result)? I checked the Keras source code, it uses K.mean(..., axis=-1) for MAE calculation. If K.mean is the same to numpy.mean, "axis=-1" should do the column mean (for my case, it should return a tensor with shape (?,2) but not a scalar).  If this is the case, how could the loss value be a single number (as outputed in the training process log)?

If the MAE return is indeed a scalar (reduced mean), this gives me another problem. The data from each dimension of my target is not in a same range. A reduced mean would be biased towards the high value dimension. Shall I change my model to a multi-task learning model then?

Thanks a lot for your help on this.

L.


texts: Forcing a multi-label multi-class tree-based classifier to make more label predictions per document

texts:
I'm been experimenting with tree based classifiers for multi-label document classification. All the trees I've created, however, tend to predict only one or two labels per document. Whereas the training set has about 4 labels per document on average. Furthermore, in my particular application, false positives are much less costly to the business than false negatives. So, if anything, I'd like the tree to be making about 6 or 7 predictions per document.

I'm not entirely sure which parameters control this. I've tried experimenting with tree size but without effect. I'd ideally like to just set a threshold for when a prediction is included, and lower this.

I'm using sklearn (and playing with skmultilearn). Here's an example of a forest configuration:

 from sklearn.ensemble import RandomForestClassifier

    clf = RandomForestClassifier(
        n_estimators=20,
        criterion='gini',
        max_features=0.5,
        max_depth=68,
        min_samples_split=4,
        min_samples_leaf = 2,
        n_jobs=3
        )


texts: What observations can you perceive from the following table?

texts:
I am preparing my thesis presentation and I want to give a clear understanding of the data that I have collected for the jury members. I understood the data very well, but I feel like I am lacking a simple language to explain it clearly, because the jury members are probably not from a quantitative background.
PS: Data is not normally distributed.


texts: how to package PMML files for versioning?

texts:
I am looking to package PMML file generated to export tree model.
Are there any existing standards which describe how to package and version PMML files ?


texts: How Can I Solve it? TypeError: fillna() got an unexpected keyword argument 'implace'

texts:
I am trying to replace NaN values in a given dataset with this

import pandas as pd
import quandl
import math

df.fillna(-9999, implace=True)

But I keep on getting this error:

**

Traceback (most recent call last):

File "regression.py", line 44, in
    df.fillna(0,implace=True)

File "/home/compname/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 3790, in fillna
    downcast=downcast, **kwargs)

TypeError: fillna() got an unexpected keyword argument 'implace'

**


texts: BertPunc (punctuation restoration with BERT)

texts:
I've found the script for punctuation restoration. And I have one question about this method.
I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), PERIOD (1), COMMA (2), QUESTION (3). Further, all words are converted to BERT tokens. Here is an example:
  2045  0
  2003  0
  2200  0
  2210  0
  3983  0
  2301  0
  2974  0
  1999  0
  2068  2

Next, we do padding. We set a segment (e.g. eight words) and for each word we take two words before a token and four words after a token. Also, we add a padding token after each word. For the very first word, there are no words before. Therefore, a word are taken from the end. Similarly, for the last word, there are no words after and therefore a word are taken from the beginning.
Here is an example of it.
[1999, 2068, 2045, 0, 2003, 2200, 2210, 3983] 0
[2068, 2045, 2003, 0, 2200, 2210, 3983, 2301] 0
[2045, 2003, 2200, 0, 2210, 3983, 2301, 2974] 0
[2003, 2200, 2210, 0, 3983, 2301, 2974, 1999] 0
[2200, 2210, 3983, 0, 2301, 2974, 1999, 2068] 0
[2210, 3983, 2301, 0, 2974, 1999, 2068, 2045] 0
[3983, 2301, 2974, 0, 1999, 2068, 2045, 2003] 0
[2301, 2974, 1999, 0, 2068, 2045, 2003, 2200] 0
[2974, 1999, 2068, 0, 2045, 2003, 2200, 2210] 2

The first column contains tokens, and the second column contains punctuation symbols. In first column, "0" corresponds to a padding. Next we do TensorDataset, and than DataLoader. In the second column, '0' corresponds to "other", and '2' corresponds to a "period". Finaly we train a model:
for inputs, labels in data_loader_train:
       inputs, labels = inputs.cuda(), labels.cuda()
       output = model(inputs)

The algorithm works well, but I do not understand the following.
What's the point of putting padding in the middle? Maybe we can do punctuation restoration with BERT in a more simple way?


texts: Prove Reccurrent Neural Network can exhibit oscillatory behavior

texts:
I understand how recurrent neural networks work, however I'm trying to build a deep intuitive understanding of their behavior which is difficult for me because they exhibit such complex behaviors.

The most difficult thing for me to understand is how a Reccurrent Neural Network can exhibit an oscillatory behavior along with the notion of exploding weights. For one, I'm guessing that oscillatory behaviors are only possible for certain activation functions and configurations. Here are my following questions:

1) Can sigmoid activation functions exhibit oscillatory behavior? I've convinced myself that they don't since they have a positive range between 0 and 1 which doesn't allow for negative derivatives, but maybe I'm wrong. Is there a formal proof out there for whether it can or cannot?

2) Which activation functions can exhibit oscillatory behaviors and are there proofs out there for them? I believe tanh might have this behavior but I'm not sure.

3) What are exploding weight derivatives and how do they occur? There is polar opposite of exploding weight's which seems to be where the weights do not learn and stay stagnate. What causes these issues? I imagine this is dependent upon the activation function as well.

Lastly, I began reading this article:

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCsQFjAB&url=http%3A%2F%2Fwww.researchgate.net%2Fprofile%2FChristian_OReilly%2Fpublication%2F52004955_Permanent_oscillations_in_a_3-node_recurrent_neural_network_model%2Flinks%2F0c96052464be171d68000000.pdf&ei=X_E1Vf7ULNGsogSo9YHIBQ&usg=AFQjCNE0fex0s7hY2w_upMNkmaxIseUKww&bvm=bv.91386359,d.cGU

but got confused around page 10 because the author omit's explaining opaque assumptions such as why we are defining the matrix measure as is and so forth(it's a good paper nonetheless). Is there a more transparent paper or lecture out there that can shed light on this subject, or maybe simple explanation for what the author is trying to get at?


texts:
I have a list
[1.0, 0.488, 0.300, 0.213, 0.163, 0.127]

Plot (dont have enough reputation to post an image)
I need to extrapolate this function for 15 more points further, asymptote is 0.
I figured out that maybe I need a custom kernel for SVR (something like y=1/x), but I havent found any examples on how to implement it.
I will be very grateful for any examples/links/or tips


texts: Reinforcement Learning Vs Transfer Learning?

texts:
I recently saw a video lecture from Jeremy Howard of fast.ai in which he states that transfer learning is better than reinforcement learning. But I was unable to understand the reasoning behind it.

Can someone explain to me or point to any evidence stating which is better and why?


texts:
I'm found an interesting problem on kaggle and more or less solved it with my limited knowledge of machine learning. I was curious how other people solved and checked the solution with the highest vote.
The solution includes a line where is implementing and training a model.
params = {'num_leaves': 8,
         'min_data_in_leaf': 42,
         'objective': 'binary',
         'max_depth': 5,
         'learning_rate': 0.01,
         'boosting': 'gbdt',
         'bagging_freq': 5,
         'feature_fraction': 0.8201,
         'bagging_seed': 11,
         'reg_alpha': 1,
         'reg_lambda': 4,
         'random_state': 42,
         'metric': 'auc',
         'verbosity': -1,
         'subsample': 0.81,
         'num_threads': 4}
oof_lgb, scores = train_model(X, y, params=params, folds=folds, model_type='lgb', plot_feature_importance=True)

But it is nowhere mentioned what kind of model he is using, which causes the error.
NameError: name 'train_model' is not defined

Does someone of you know which model he is using, so that i can run his notebook on my computer.


texts: What does online learning mean in Topic modeling (LDA) - Gensim

texts:
I came across this line in the Gensim Documentation- Gensim LDA  - "The model can also be updated with new documents for online training."

So my assumption on what it means is - 'Once we have a model trained on one corpus, we can add new data and continue to train the model with new data thereby adding more vocabulary and enriching results. Is this correct?

Is this the same approach discussed in the paper - Online Learning for LDA ? Help me understand this technique.


texts: Replace a value in a column if that value appears only once

texts:
I have a dataframe, I want to replace the values in one column by "other" if the value count of that value in that column is exactly 1

i        Food_group
0          Flake
1          Flake
2          Flake
3         Almond
4          Drink
5          Drink
6          Flake

I have tried,

data["food_group"] = data.food_group.apply(lambda x: "other" if x.value_counts()==1 else x)

I got error

AttributeError: 'str' object has no attribute 'value_counts'


texts: Question About Coming Up With Own Function for Distance Matrix (For Clustering)

texts:
Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game.

A lot of the features I plan on using are unique to this game (data that can only be analyzed if one knows the game well), and thus I believe that it is best for my data that I come up a new function to generate the distance matrix that I plan on using in the various clustering algorithms later on.

My data is just a mixture of continuous and categorical data, and instead of using Gower Distance for the reasons specified above, I wanted to come up with my own formula.

For example, when comparing the download date (a time period measured by the datetime object from pandas), I would like to cross reference the download date with the time periods of the various promotions that went on and adjust the similarity score for the download date accordingly.

Now, if I have 35 million different user data, and around 20 features (for each of the 35 million different users), is this a feasible method of calculating the distance matrix?

I'm asking because I've read that using the gower distance to calculate the dissimilarity matrix for a large dataset is not feasible due to the time complexity.

I was wondering if the method I described earlier is feasible in terms of time complexity, or am I just wasting my time?

Thank you. As someone who doesn't have much experience with data science and clustering, any input would be grateful!

Thank you.


texts:
I am analysing if 15 books can be grouped according to 6 variables (of the 15 books, 2 are written by an author, 6 by an other one, and 7 by an other one).
I counted the number of occurrences of the variables and I calculated the percentage. Then I used Orange software to use PCA. I uploaded the file. selected the columns and row. And when it comes to PCA the program asks me if I want to normalize the data or not, but I am not sure about that because I have already calculated the percentage - is normalize different from calculating the percentage? Moreover, below the normalize button it asks me to show only:... and I have to choose a number between 0 and 100 but I don’t really know what it is.

Could you help me understand what I should do?
Thank you in advance


texts: way to add target delay on TimeseriesGenerator from keras.preprocessing.sequence

texts:
from deep learning with python book, it created function for data generator.

I thought I can do the same using TimeseiresGenerator from keras package but I was not able to add target delay. Is there any way to do it?

EDIT: I found a way by using np.roll on target data. But if there is a better way please let me know.


texts: How to get Real life Social Network Data?

texts:
I am trying to build a case study for Insurance fraud detection using SNA. I have already looked around and build my model using Stanford's SNAP and other similar databases available on the internet for research. However, I am just wondering, what are the data sources from which I can get Social Network Data apart from the Facebook, Linkedin, Twitter connections.
Can I get (buy) phone call records, or any other social network dataset?
Also, do FB, linkedin sell their connections data?


texts:
I'm currently building an app that strives to predict how the users uses different apps and give the user a suggestion based on which apps it think the user will currently use (a ranked list based on the user's current conditions). I've been collecting some data over the past week now, and I'm not really sure which approach to take. I've been thinking about using multiple seasonalities (correct me if I'm using the wrong terminology) such as time of day, day of week, week of month, month and quarter. I also want to use location, and other sensor data (such as the user state "walking" or "sitting" later on).

I've summarised the usage over the last week on an hourly level for some apps.
The bars represents each time the app was opened during that time, and the green line is a weighted moving average which has a weight of 0.5 on its closest neighbours.

Now I see several challenges in front of me and would greatly appreciate some input from others, or some good resources to find further information.

Do you think my model is a good one for this problem
How do I account for ageing data?
How do add up the different seasonalities/states/location? Multiply them?
Does it make sense smoothening the curve as I do?

Here some data for the last week on an hourly basis:


texts: Can I consider this pattern of data as a linear and use parametric multiple linear regression?

texts:
In the data, there are 355 observations including one continuous dependent variable (Y: ranges from 15-55) and 12 independent variables (continuous, categorical, and ordinal). The X1 (2 levels) and X6 (3 levels) are considered as categorical variables. Here are some questions that I have:

Can I assume that all the coefficients (except X1 and X6 which are categorical) are linear with respect to Y?
Can I consider X5 as continuous variable; however, it is ordinal and ranges from (1-7)?
Can I get the X7 (year) as continuous variable; however, it’s ordinal and rages from 2002-2006 (In fact, year of data per se does not improve the response; it is the other factors occurring in the same time period which result in improvements and we don’t know those factors), does this approach seem logical?.
In general if I use different transformations on independent variables such as log, squared, square root, and inverse, do I need to standardize the data also?

Here is the scatter plot:

Any feedback and insights would be highly appreciated. Thank you


texts: I read this interesting kernel on kaggle. I learned that you can make a 3D plot starting from CT sections. Is there a way to calculate the volume of the lungs or other structures you plot using 3d_plot? I couldn't find any reference so I need help.


texts:
i m try to apply k-means with Python 3 to my dataset (Amazon review)  for classify similar user (from review).

I just have a TF and TF-IDF matrix (and i have a row(user) and columns(words) value in distinct csv file)

I m starting with sklearn from this sketch

https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html#sphx-glr-auto-examples-text-plot-document-clustering-py

but in my case i don't have label or category, and i don't understand how k-means can cluster different items from this matrix TD-IDF.

How can I group similar words from my dataset, from the matrix
without having any information?
How do I show these n-clusters in the chart?
And how do I show the similar words of this cluster, if I do not
have any information (like label or category)?

For now this is my code :

k = 5
km = KMeans(n_clusters=k, init='k-means++', max_iter=100, n_init=5)
km.fit(Y) ##Y is my TD-IDF matrix

original_centroids = svd.inverse_transform(km.cluster_centers_)
print(original_centroids.shape)
for i in range(original_centroids.shape[0]):
original_centroids[i] = np.array([x for x in original_centroids[i]])
svd_centroids = original_centroids.argsort()[:, ::-1]

i would like have a set like this(with similar word near)

I m very lost, because i m new in this word.


texts: Why do we only care about convex functions when doing Gradient Descent/SGD?

texts:
I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of the function.
However, there are also functions which are not convex, but for which local minima are also global minima, for example, a function which looks like this:
Isn't there a way to characterize every function which "works well" with gradient descent? Something like "if f has a local minima, then it's also a global minima", which would be a weaker hypothesis than convex?
PS: When writing this, I also realize that we have a problem with functions like $x \rightarrow{} x^3$, for which gradient descent could stop on a "plateau". Anyway, my question remains the same: Isn't it possible to define a more general class of functions for which gradient descent works better than the convex functions?


texts: Is it possible to use Word2vec for text paraphrasing?

texts:
After reading several papers I am not sure if it is possible to some how generate text with the same meaning (paraphrase it) using only Word2vec.

I found out other approaches that use sequences of sentence pairs, and they train Neural nets to find the most similar, but this is hard to maintain and it will be hard to generate relevant content like this.

I would like to give raw text to Word2vec powered algorithm that gives paraphrased text.


texts: I am trying to do a small project on my own to find out job openings using twitter data. I saved data using flume and converted it to .csv for analysis. My problem is i don't know how to classify tweets, whether it is a job vacancy or just some news on say machine-learning.I read online about neural networks and word2vec but i am not sure if it will solve my problem. Can anyone suggest some ways to do this based on tweet text and hashtags.I don't have training and test data, i just have tweets stored using flume.Also what kind of analysis will i be able to do with it.I am new to data science :/


texts: What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

texts:
This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm?

I understand that EDA helps us generate good and helpful insights into the data, which is crucial in data understanding. If we leave aside standard checks and manipulations like - removing outliers, scaling, removing constant value columns, removing null/'zero' value columns, etc. and if we have 20-30 features.
How will EDA help me in producing good and sensible clusters? Is it even necessary to do the EDA before clustering?

Note: I am using k-means


texts: Satellite Change Segmentation using Unet

texts:
Hi StackExchange community I am working on to train a Unet for satellite change segmentation. My dataset consists of images(before change),images(after change) and the corresponding change segmentation ground truth mask. Right now I am concatenating both the satellite images on channel axis and passing to Unet also I am changing the pixel values of mask which consists of [0,1,254,255] to only [0,1].I am using dice coefficient and dice loss to monitor the performance but my dice coefficient value is not increasing neither the model is able to segment properly .Any suggestions on what wrong I am doing and how can I improve the performance. Or any other approach that I should use in order to do satellite change segmentation.
Thank you very much


texts: is it possible determine a best neural network classifier by considering only accuracy

texts: Below, you have the accuracy plots for training and testing set for 6 different neural networks. is it possible to say, which of the following neural network classifier is better?. Having this little known information (training and testing accuracy by number of epochs plot) ? . Personally, I believe, the first classifier (top-left corner) is better because it shows that the accuracy of the testing data is stable as the number of epochs increases.


texts: Which statistical method to use for feature selection between numerical inputs and categorical input?

texts:
I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and understanding of ML workflow tells me some features are redundant and don't provide much predictive insight to my model.

I am using Sklearns SelectKBest() package. However there are numerous statistical tests to conduct such methods. So my question is, given my inputs and outputs, which statistical method is best for my situation?

Furthermore, I wish to implement this step in Sklearns pipeline() package with GridSearch. My pipeline is quite rudimentary at this point but I still don't know which step this should be conducted. Would appreciate your answer guys.

My pipeline code is as follows:

pipe = Pipeline([('sc', preprocessing.MinMaxScaler()),
                 ('SVM', svm.SVC(decision_function_shape = 'ovr', kernel = 'poly'))])

    candidate_parameters = [{'SVM__C': [0.01, 0.1, 1, 10], 'SVM__gamma': [0.01, 0.1, 1, 10]}]

    print('Fitting model...')
    clf = GridSearchCV(estimator = pipe, param_grid = candidate_parameters, cv = 5, n_jobs = -1)
    clf.fit(X_train, y_train )


texts: How can conclusions be drawn from recommendation systems evaluation?

texts:
From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many type of recommendation systems, such as collaborative filtering and content-based.

One important aspect, is that evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error, and also there exists others such as precision an recall.

My question is after I developed a recommendation system, and after evaluating using those metrics, how can I consider that my system provides good quality recommendations, or in others words, what should be the tresholds or MSE or RMSE values to be considered a good recommendation system.

Thanks. Any suggestions are welcomed.


texts: Why does using Gradient descent over Stochatic gradient descent improve performance?

texts:
Currently, I'm running two types of logistic regression.

logistic regression with SGD
logistic regression with GD

implemented as follows
SGD= SGDClassifier(loss="log",max_iter=1000,penalty='l1',alpha=0.001)
logreg = LogisticRegression(solver='liblinear', max_iter=100, penalty='l1', C=0.1)

nevermind the hyperparameters as I've used GridsearchCV and tried multiple combinations.
When calculating accuracy logistic with GD performs better than SGD. I want to understand why this is the case, is using GD instead SGD one way to mitigate underfitting model?


texts: How to study the effect of eps in sklearn.cluster.DBSCAN?

texts:
I posted this question on stackoverflow.com and have not received any answer. In case I get an answer from one of them, I will inform on the other.

I have a dataset and is requested by my professor to study the effect of eps in sklearn.cluster import DBSCAN? Could you please suggest some ways to measure its effect? I generated eps = np.arange(start = .1, stop = 1, step = .1), but I don't know what should I compare.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import DBSCAN
%config InlineBackend.figure_format = 'svg' # Change the image format to svg for better quality
don = pd.read_csv('https://raw.githubusercontent.com/leanhdung1994/Deep-Learning/main/donclassif.txt.gz', sep=';')

kmeans = DBSCAN(eps = .2)
y_kmeans = kmeans.fit_predict(don)

plt.figure(figsize = 1.5 * np.array(plt.rcParams['figure.figsize'])) # This is to have bigger plot
plt.scatter(don['V1'], don['V2'], c = y_kmeans, cmap = 'viridis')

eps = np.arange(start = .1, stop = 1, step = .1)


texts: How do you find the standard deviation of a dataset over a certain period?

texts:
I'm new to R.
I could get the std. dev. of the whole data set,
data <- read.csv("Berkshire.csv")
sd1 <-sd(data$BRKRet, na.rm = TRUE) #0.0675
mean<- mean(data$BRKRet, na.rm = TRUE) #0.0189

but I'm not sure how to find the std. dev. of a dataset over a certain time period. For example, from Nov. 17, 2010 ~ Nov.16, 2020.
Thanks for the help.


texts: Problems with Concatenating Embedded Categorical and Numerical variables for LSTM use

texts:
I am new to here and new to Deep Learning too, so apologies in advance for any ill formatted code or wordings. I have a data set where I track 4 variables with 2 categorical and 3 numerical fields, over 4 time steps.
I was able build a dataframe like this:

cat1 - SckitLearn Label Encoded variable names
cat2 - SckitLearn Label Encoded time steps
num1 : num3 - SckitLearn Normalized variables

I got it turn into a list of numpy arrays like this:

I want to embed and concatenate all these fields before feeding it into a LSTM in Keras, using this function:
def build_concat(df):
  global inputs
  inputs = []
  global embeddings
  embeddings = []
  cat_cols = df.filter(like='cat')
  num_cols = df.filter(like='num')
  for cat_col in cat_cols:
    cat_input = Input(shape=(4,4), name=cat_col)
    unique_cat = cat_cols[cat_col].nunique()
    embedding_size = min(np.ceil((unique_cat)/2), 20)
    embedding_size = int(embedding_size)
    cat_dim = unique_cat + 1
    inputs.append(cat_input)
    embeddings.append(Embedding(cat_dim, embedding_size, input_length = (4,4),
        name=str(cat_col) + "_emb")(cat_input))
  for num_col in num_cols:
    num_input = Input(shape=(4,4), name=num_col)
    inputs.append(num_input)
    embeddings.append(num_input)

But I get this error :

ValueError: Shape must be rank 3 but is rank 2 for '{{node so2_model/concat/concat}} = ConcatV2[N=5, T=DT_FLOAT, Tidx=DT_INT32](so2_model/cat1_emb/embedding_lookup/Identity_1, so2_model/cat2_emb/embedding_lookup/Identity_1, IteratorGetNext:2, IteratorGetNext:3, IteratorGetNext:4, so2_model/concat/concat/axis)' with input shapes: [2,4,2], [2,4,2], [2,4], [2,4], [2,4], [].

This is my embedding list :

[<KerasTensor: shape=(None, 4, 4, 2) dtype=float32 (created by layer 'cat1_emb')>, <KerasTensor: shape=(None, 4, 4, 2) dtype=float32 (created by layer 'cat2_emb')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num3')>]

This is my input list, before embedding :

[<KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'cat1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'cat2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num3')>]

Thank you for your help!


texts: Which Machine Learning methods would be good for a beginner-level Data Science project for described dataset?

texts:
I want to do my first  beginner-level project regarding Data Science. I picked my data set which is described below. I want to make a Python script which will make some predictions on our data set using training and test data. That could be for example a prediction which countries will see rise in number of customers and which ones will see a decline, but that's just the first idea that popped in my head. I want to implement a few Machine Learning methods to do the same prediction on our data set and compare their results. Finally comes my question - which Machine Learning methods would you suggest for a beginner Data Science project with this data set which is described below? Thank you in advance for any suggestions. I don't want to pick something very hard as the beginner and I also don't want to pick a method that just doesn't "fit" my data set (because it's rather simple).

EDIT: To specify this project will concern business predictions based probably on time-series. I'm looking for recommendations for which ML methods are easy to implement use and understand for the beginner. Project will be written in Python.

Data Set Description:

Number of Attributes : 8

Invoice : Integer

StockCode : String

Description: String

Quantity : Integer

InvoiceDate : Timestamp

Price : Decimal

CustomerID : Integer

Country : String

Number of records: 541910

Date ranges: 01/12/2010 08:26 - 09/12/2011 12:50

Price ranges: -11062.06 - 38970.00

Number of distinct Countries: 38


texts: feature extraction from single word for classification into nouns and names

texts:
I would like to write a NN that can classify different kinds of words(e.g. nouns,verbs,names) and am struggling to find information on how to do feature extraction on single words.For example, i would like the NN to learn that "street" is a noun, while "How would i go about doing that? I'm really nwe to this and searching for it always seems to yield only examples of text feature extraction, which is not what I'm looking for.

Thank you in advance kind strangers!


texts: How to train LSTM with daily timeseries?

texts:
I have for each day sensor timeseries data. I just ask myself how to train with that a LSTM eg. for classification? Since I would like to have the LSTM train on all examples and not just one?

I just see examples where LSTM are trained on one timeseries


texts: Adding more emphasis on most recent data in CNNs

texts:
I am using a CNN for multivariate time series analysis. The input size is (batch_size, 500, 30) i.e 30 variables and 500 time steps. I would like to put more emphasis on recent data and less on the past data.

What is the best way to achieve this?


texts: Weird distribution of neural network outputs

texts:
I've faced an unusual behavior during training a neural network.

The problem is to predict if a sample of 1st class or 2nd class. (2-class classification). Classes are imbalanced (~ 5 / 95). I use weighted crossentropy.

What do I see is probabilities of two classes for each sample, and it's seems weird to me that no matter how long I wait (number of epochs), the probabilities are in range [0, 0.9) and there are no samples close to probability 1.0. I guess, the imbalance and weighted loss + ADAM with low starting learning rate could led to this, but I am not sure.

Has anyone faced the same?


texts: Strategy for "forcing" number of labels in seq2seq predictions with Keras?

texts:
I'm trying to train a seq2seq model that for every timestep in a given timeseries sample will output 1 of 6 possible labels.

Furthermore, the training data is constructed in such a way that

Each sample can only contain a maximum of 2 types of labels
Label 1 can co-occur with any other label. No other labels can co-occur.

Is there any way to specifically guide training with rules like these, or is my best bet to hope that the model will pick up on this? In roughly 85-90% of cases the model behaves as hoped, but in a few noisy edge-cases it sometimes oscillates wildly between different labels, throughout the sample.


texts: when x is a vector, derivative of vector diag(f'(x)) is formal notation?

texts:
https://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf

(4)

this note says this

$$
\frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x}))
$$

I know this means make a vector to square matrix which has an original vector as diagonal component.

but, since $\textbf{x}$ is a vector, $f'(\textbf{x})$ can't be a vector, derivative of a vector should be a matrix, right? Is this a formal notation for element-wise derivative of vector?


texts: I am trying to build classification model using Java Weka API. My training dataset have class imbalance problems. For this reason, I want to use SMOTE to reduce class  imbalance problem. But, I do not know how to use it in Java Weka API.


texts: Why ReLU is better than the other activation functions

texts: Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu has a disadvantage and it is its expected value. there is no limitation for the output of the Relu and so its expected value is not zero. I remember the time before the popularity of Relu that tanh was the most popular amongst machine learning experts rather than sigmoid. The reason was that the expected value of the tanh was equal to zero and and it helped learning in deeper layers to be more rapid in a neural net. Relu does not have this characteristic, but why it is working so good if we put its derivative advantage aside. Moreover, I guess the derivative also may be affected. Because the activations (output of Relu) are involved for calculating the update rules.


texts: How does the construction of a decision tree differ for different optimization metrics?

texts:
I understand how a decision tree is constructed (in the ID3 algorithm) using criterion like entropy, gini index, variance reduction. But the formulae for these criteria do not care about optimization metrics like accuracy, recall, AUC, kappa, f1-score and others.

Packages on R and Python allow me to optimize for such metrics when I construct a decision tree. What do they do differently for each of these metrics? Where does the change happen?

Is there a pattern to how these changes are done for different classification/regression algorithms?