texts: CNN seems doing good during training and validation but not really

texts:
I'm doing a simple binary classification using this dataset
import os

train_dir = "/tmp/Dataset/TrainingandValidation"
fire = os.path.join(train_dir, 'fire')
nofire = os.path.join(train_dir, 'nofire')

test_dir = "/tmp/Dataset/Testing/"
tfire = os.path.join(test_dir, 'fire')
tnofire = os.path.join(test_dir, 'nofire')

from tensorflow.keras.preprocessing.image import ImageDataGenerator

test_data = ImageDataGenerator(rescale=1.0/255.)
train_data = ImageDataGenerator (rescale= 1.0/255.)

train_images = train_data.flow_from_directory (train_dir,
                                               batch_size= 20,
                                               class_mode='binary',
                                               target_size=(150,150))
test_images = test_data.flow_from_directory(test_dir,
                                            batch_size=20,
                                            class_mode='binary',
                                            target_size = (150,150))

import tensorflow as tf

model = tf.keras.Sequential ([
                              tf.keras.layers.Conv2D (16, (3,3), activation='relu', input_shape =(150,150,3)),
                              tf.keras.layers.MaxPool2D(2,2),

                              tf.keras.layers.Conv2D (32, (3,3), activation='relu'),
                              tf.keras.layers.MaxPool2D(2,2),

                              tf.keras.layers.Conv2D (64, (3,3), activation='relu'),
                              tf.keras.layers.MaxPool2D(2,2),
                              tf.keras.layers.Flatten(),
                              tf.keras.layers.Dense(512, activation='relu'),
                              tf.keras.layers.Dense(1, activation= 'sigmoid')

])

model.summary()

from tensorflow.keras.optimizers import RMSprop
model.compile(optimizer= RMSprop (lr=0.001),
              loss = 'binary_crossentropy',
              metrics=['accuracy'])

from tensorflow.keras.callbacks import Callback

class myCallback(tf.keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs={}):
    if(logs.get('accuracy')>0.94):
      print("\nReached 94% val_accuracy")
      self.model.stop_training = True
callbacks = myCallback()

history = model.fit(train_images,
          validation_data=test_images,
          steps_per_epoch= 76,
          epochs=15,
          validation_steps= 19,
          callbacks= [callbacks])

import numpy as np

# uploading images to predict
from google.colab import files
from keras.preprocessing import image

uploaded=files.upload()

for fn in uploaded.keys():

  # predicting images
  path='/content/' + fn
  img=image.load_img(path, target_size=(150, 150))

  x=image.img_to_array(img)
  x= x/255.0

  x=np.expand_dims(x, axis=0)
  images = np.vstack([x])

  classes = model.predict(images, batch_size=1)

  print(classes)

  if classes[0]>0:
    print(fn + " fire")

  else:
    print(fn + " no fire")

The model is getting well as below :
Epoch 14/15
76/76 [==============================] - 3s 40ms/step - loss: 0.0524 - accuracy: 0.9812 - val_loss: 0.2918 - val_accuracy: 0.9474

Reached 94% val_accuracy

Although it might look it's overfitting or so, when I upload images to predict them the model almost always predict 1 (fire), even if I ask it to predict training images!
I tried adding a dropout layer before the last layer but not much better.
I feel like there's something wrong with the preprocessing of the images?


texts: Equivalent procedure to Scikit-Learns class_weight=balanced in Keras?

texts:
I want to train a SVM and a CNN with the same unbalanced multiclass-dataset and want to compare the results.

I use Scikit-Learn for the SVM and Keras for the CNN.
My goal is that no class is preferred during the training, every class should be considered equally important.
For the SVM I use class_weight='balanced'.
Would it make sense to use also Scikit-Learns class_weight='balanced'
in Keras for training the CNN?

What is your opinion? What would you recommend?
Many thanks for every hint


texts:
I'd like to perform a textual/sentiment analysis.

I was able to analyse samples with 3 labels: (positive, neutral, negative) and I used algorithms such as SVM, Random Forest, Logistic Regression and Gradient Boosting.

My script works correctly and with the cross validation I can take the best algorithm among the 4.

I use supervised algorithms with the python function "Countvectorizer"

But my boss typed "NLP" on the internet and looked at some articles.

He told me : "These 3 outputs are not enough, I want a complete semantic analysis that can explain the global meaning of the sentence"

He didn't seem to have a preference between supervised and unsupervised algorithms.

He told me that he wanted an algorithm able to tell that "The company president is behind bars" is equivalent to "the CEO is in jail".

So do you have any idea how one could perform that ? And how to implement it in Python? I guess we need a great database full of words, I know this is not a very specific question but I'd like to present him all the solutions.

What scares me is that he don't seem to know a lot about it, for example he told me "you have to reduce the high dimension of your dataset" , while my dataset is just 2000 text fields.

Thank you very much for your answers :)


texts: Ensemble Method using XGBoost and RotationForest python

texts: How can I create an ensemble model using XGBoost and Rotation Forest in Python?


texts: How to make ML model publicly available?

texts:
When I have finished building my machine learning model and want to make it publicly available for other users then what could be the best solution for that?
I mean I'm looking for a solution where there is a UI interface for an input/output and the ML code get executed through this UI.

Simple Jupyter Notebook - The code is not executable.
Jupyter Notebook + MyBinder + ipywidgets - nice UI can be build by the ipywidgets and the ML code is executable but still the user needs to execute all the cell boxes before he/she can use the ML model.
plotly/Dash - So far I have no experience by this. It is said that it can be used for building web applications with highly custom UI's, deploying apps to servers and then share them through URLs. Is it so that when using this then I need to have my own server where the ML code runs?
Is there any solution where I don't need to have my own server and can deploy and run my ML project?


texts:
Suppose we have a list of products categorized into 10 categories. We also have customers order details such as

  order_id, product_name, quantity, order_date

We want to know for a particular month what are the most probable categories from which customer might buy products. How can we approach this problem, Any suggestions.


texts: multi channel feature classification using deep feed forward neural network on tensorflow

texts:
Can any one suggest a dataset that contains multi channel/multi dimensional feature values that can be classified using a deep feed forward neural network?

Please ask me if the question is not clear enough.


texts: Annotating the vocabulary using Word2vec model

texts:
I am trying to label the vocabulary in the corpus.

I have trained the word2vec model on the corpus
I have grouped the words which are related based on the score as key as the first word as the key and remaining words as a list of 2-tuple of word and scores with respect to the key

example:
'coffee'---key
values are

[('tea', 0.8139282),
 ('latte', 0.76456803),
 ('coffe', 0.7607962),
 ('lattes', 0.756057),
 ('starbucks', 0.7158153),
 ('espresso', 0.71386236),
 ('mocha', 0.69999266),
 ('coffees', 0.6816252),
 ('frappucino', 0.67192864),
 ('cuppa', 0.66720986),
 ('cappucino', 0.6664002),
 ('chai', 0.6623157),
 ('decaf', 0.65980726),
 ('frappuccino', 0.65150374),
 ('venti', 0.6486204),
 ('expresso', 0.6369579),
 ('macchiato', 0.6280453),
 ('scone', 0.62476856),
 ('sippy', 0.6236704),
 ('cappuccino', 0.61718297),
 ('iced', 0.6130485),
 ('hazelnut', 0.6023698),
 ('mug', 0.6004759),
'
'
'
'
'

as i  know the coffee is releated to latte ,green_tea ,espresso,starbucks.. from the above data
I would like to label each word as below

latte [COHYPO] green_tea [COHYPO] espresso [HYPO] Starbucks [RELATED] tim_horton [RELATED]

COHYPO-https://en.wiktionary.org/wiki/cohyponym

[HYPO] -https://en.wiktionary.org/wiki/hyponyme

[RELATED] -the word is repeated

[MORPHO]-Morphological variant (example :Computer and computers )

[Partof]- indicates that the annotated word is a part of the word of interest

Any suggestion or ideas by which I can approach this problem


texts: Is linear regression suitable for these data?

texts:
I have a data set predicting a continuous variable, $Y$. I have $15$ to $20$ potential feature variables most of which are categorical, some of which are ordinal or categorical. These have been converted to numerical values. I have two questions.

Is linear regression suitable in this case?
If the variables do not show linear relationships with $Y$, is linear regression still suitable? Otherwise, which algorithms, hopefully existing in scikit-learn, might work?


texts: Semantic Annotation in text with curlie.org

texts:
I came across curlie.org (previously known as the dmoz taxonomy) and I'm interested to see how I could best start tagging a given text, with concepts from that taxonomy:

Are there any tools out there that do semantic annotation based on a taxonomy (I couldn't find any)
How would one go about making such a semantic annotation process

I know this question might be a large to answer in a short reply, but any pointers are greatly appreciated.

Thanks in advance!


texts: How to properly use oversampling without inflating results?

texts:
I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i decided to apply oversampling (SMOTE). Here is what i did:

Using the entire dataset (original 192 samples):
Create synthetic samples for each class using SMOTE, so i get a total of 500 samples per class (2000 in total)

I have a big suspicion about this procedure because when i apply SMOTE i get very high accuracy rates (even 100% in some cases) with the simplest models such as a 15-neuron MLP. So, i have some questions to assert the correctness of my experiments:

Is it ok to oversample the entire dataset or should i apply the SMOTE only in train data (keep in mind that this will leave few samples for test)?

SMOTE is originally used to deal with imbalanced datasets, creating samples for the classes with few sample. Is it ok to use it for generating samples for ALL classes in order to enlarge the entire dataset?


texts:
When reading about SVMs (e.g. on the German Wikipedia) there is a sentence like "an svm is a large-margin classifier).

Are there other large margin classifiers than SVMs?


texts: How to collect info about unseen bugs given user's comments/feedbacks?

texts:
I have a dataframe which looks like:
user_id, comment
0, 'Functional but Horrible UI'
1, 'Great everything works well'
2, 'I struggled finding plus button because of theme colors in dark mode'
3, 'Keeps stopping on Android 10'
4, 'I like the functionaity but color theme could be better'
5, 'Consistently crashing. Uninstalled'
6, 'Good overall'
7, 'sfdfsdlfksd'
8, 'I lost in complex settings'
9, 'Configuring app is really a headache'
10, 'aaaaaaaaaaaaa'

And I want to figure out some data science approach to pluck out information about what users are struggling with and which issues appeared how much and stuff like this. Even some simple output would be good for me so that we know which parts of app to focus on more. Like for sample above I am aiming for an output as simple as:
problems = {
'color_theme': 3,
'app_settings': 2,
'crashing' : 2}

So I kinda wants labeling and how much time a label is occured based on to which label a review belongs. But the problem is I cannot train a model with predefined labels because:

I do not have labels for reviews. If we have to go through each review to know what problem is it talking about (i.e. to label it), we would just have filed it as well and would know what we have to work on.

I do not know in advance what problems are gonna come in future so even if we somehow label all at some point in time, it wouldn't be enough as some unseen problem may come and we have to do again.

Even if we have a system of labeling somehow, how would we update model, like do we define a new model with a different architecture for ever changing labels?

So under these circumstances, I was trying to figure out an AI approach to ease in my situation. I am pretty good at python and do have working knowledge of keras/tensorflow and other libraries but none of them seem to have such flexible model approach. I was going through Google Cloud Platform's AI platform as well but it could do sentiment analysis to an extent but not understand in an app context that e.g. button is a part of UI and color as well. So how could I approach this problem in a more elegant way?


texts: Sound Classification for Multiple Classes for English Letters

texts: I have recorded audio files for the English letters, each file includes 26 letters. I have split each letter into a separate audio file. Now I want to put similar audio letters into one folder. I can do it manually but it will take time. Is there a classifier method to this?


texts: Dealing with correlated features when calculating permutation importance

texts:
I have implemented the permutation importance calculation as found here in my attempt to identify features that contribute little to my model's (Gradient Boosted Tree model) predictive power.

The issue I have encountered is that some of my features are highly correlated, potentially masking the true importance of features when being evaluated by permutation importance. Typically the solution to this would be to perform something like Recursive Feature Elimination instead. Unfortunately, I cannot do this as the cost of retraining the model is prohibitive. The model takes ~3 hours to train with a feature set of 39 features.

My question is whether it is possible to use permutation importance while dealing with correlated features? My initial thought was to invert the process and shuffle all features other than the one I want to investigate although I do not know if this would have the same level of explanation.


texts: Visualize frequency of 5 Boolean variables together

texts:
I have a data set with 5 variables,
a b c d e
1 0 0 1 0
0 1 0 1 1
0 1 1 0 0
0 0 0 1 0
1 1 1 0 0
0 1 1 0 1
1 0 1 0 0
1 0 0 1 1
0 1 0 1 1
0 0 1 1 0
I am only interested in the percentages of occurrence,
occurrence,
| a | b | c | d | e |
.4 | .5 | .5 | .6 | .4
BUT, I would like to visualize in such a way that I can see the overlap, or not, among all the different groups.
Any idea?


texts: What is the definition of precision for no positive classifications?

texts:
The precision is defined as

$$\text{precision} = \frac{\text{true positive}}{\text{true positive} + \text{false positive}}$$

Is there any definition what this value should be if there is no positive classification (but of course positive elements)?


texts: Baby cry detection model using binary classification through logistic regression

texts:
I need some help regarding my final year project.
I am fairly new to machine learning and I have tried a lot to understand how to train a model using logistic regression.
I have two datasets of audio clips of 5 seconds each, one of the babies crying voices and one of baby 'not crying'.
I want to train the model using logistic regression on google colab. Now I have been successful converting the audio clips to spectrogram images. But I am stuck in training the model because I am unable to understand how to train the model (the code is very complex where ever I have seen it up till now).
 What I want is to train the model using logistic regression, and get 100 weights at the output. Then what will happen is that I will record baby crying audio using Arduino. Then I will get 100 data points from newly recorded audio.
Then the previous and new 100 weights will be multiplied one to one. and then added. and then a sigmoid function will be run on it(all of this calculation after getting 100 weights, will not happen in google colab but in Arduino).
And then if the output value is greater than 0.5, it means that baby is crying (which means that the newly recorded audio is of the baby crying) and if the value is less than 0.5, it means the baby is not crying. Now the 100 data points from the Arduino recording, have a range of 0 to 1. It will be good if the 100 weights of the trained model also lie between 0 to 1.
Any help regarding the code or links to a good tutorial video on logistic regression will be much appreciated.


texts: Neural network with multiple layer: learning function

texts:
Here is my code to implement the learning of my neural network using the backpropagation learning. The algorithms is stable but I don't learn correctly the output. Do you see anything wrong in my learning process?

//### Parameter ###
#define Nb_entry 2
#define coeff_app 0.01
#define par_momentum 0.8
#define par_nb_test 0.2
#define para_tolerence 0.05
#define para_stop_learning 1000

Here are my learning functions

void fonction_neuron(double * input, Neuron * neuron_info)
{
    int i=0; // loop variable
    double net=0;

    // Computation of the net value type: net = w[0]*bias + sum(w[i]*imp[i])
    net=(*neuron_info).weight[0]; //bias = 1
    for(i=1;i<(*neuron_info).nb_input;i++) net+=(*neuron_info).weight[i]*input[i];

    (*neuron_info).output=net;
    //print_stat_neuron(neuron_info);
}

void fonction_network(Neuron N_network[][10], int nb_layer, int *nb_neuron_per_layer,double *input)
{
    int i,j; //loop variable
    double previous_layer_output[10]={0};

    //Propagation of the signal into the neural network
    for(i=0;i<nb_layer;i++)
    {
        if(i!=0) for(j=0;j<nb_neuron_per_layer[i-1];j++)
        {
            previous_layer_output[j]=N_network[i-1][j].output;//save previous layer output
            //printf("previous_layer_output[%d]=%f\n",j,previous_layer_output[j]);
        }

        for(j=0;j<nb_neuron_per_layer[i];j++)
        {
            //printf(" i=%d j=%d \n", i,j);
            N_network[i][j].old_output=N_network[i][j].output; //save previous value
            if(i==0)fonction_neuron(input, &(N_network[i][j])); //first layer
            else fonction_neuron(previous_layer_output, &(N_network[i][j])); //other layer using the previous layer output
        }
    }
}


texts: How to select good inputs and fitness function to achive good results with NEAT for Icy Tower bot

texts:
I'm trying to make a bot to the famous "Icy Tower" game.
I rebuilt the game using pygame and I'm trying to build the bot using Python-NEAT.

Every generation a population of 70 characters tries to jump to the next platform and increase their fitness. right now the fitness is the number of platforms they jumped on, each platform gives +10.

The problem I'm facing is that the bot isn't learning good enough after 1000 generations the best score was around 200 (it can get to 200 even at the first few generations by mistake. 200 means 20 platforms which is not a lot).

when I look at the characters jumping it looks like they just always jump and go left or right and not deliberately aiming to the next platform.

I tried several input configurations to make the bot perform better. but nothing really helped.

these are the inputs I tried to mess around with:

pos.x, pos.y
velocity.x, velocity.y
isOnPlatform (bool)
[plat.x, plat.y, plat.width] (list of the 3-7 next platforms locations)
[prev.x, prev.y] (2-6 previous character positions)

I'm not so proficient with neuroevolution and I'm probably doing something wrong. glad if you could explain what's causing the bot to be so bad or what's not helping him to learn properly.

Although I think that the fitness function and the inputs should be the only problem I'm attaching the python-NEAT config file.

[NEAT]
fitness_criterion     = max
fitness_threshold     = 10000
pop_size              = 70
reset_on_extinction   = False

[DefaultGenome]
# node activation options
activation_default      = tanh
activation_mutate_rate  = 0.0
activation_options      = tanh

# node aggregation options
aggregation_default     = sum
aggregation_mutate_rate = 0.0
aggregation_options     = sum

# node bias options
bias_init_mean          = 0.0
bias_init_stdev         = 1.0
bias_max_value          = 30.0
bias_min_value          = -30.0
bias_mutate_power       = 0.5
bias_mutate_rate        = 0.7
bias_replace_rate       = 0.1

# genome compatibility options
compatibility_disjoint_coefficient = 1.0
compatibility_weight_coefficient   = 0.5

# connection add/remove rates
conn_add_prob           = 0.5
conn_delete_prob        = 0.5

# connection enable options
enabled_default         = True
enabled_mutate_rate     = 0.01

feed_forward            = True
initial_connection      = full

# node add/remove rates
node_add_prob           = 0.2
node_delete_prob        = 0.2

# network parameters
num_hidden              = 6
num_inputs              = 11
num_outputs             = 3

# node response options
response_init_mean      = 1.0
response_init_stdev     = 0.0
response_max_value      = 30.0
response_min_value      = -30.0
response_mutate_power   = 0.0
response_mutate_rate    = 0.0
response_replace_rate   = 0.0

# connection weight options
weight_init_mean        = 0.0
weight_init_stdev       = 1.0
weight_max_value        = 30
weight_min_value        = -30
weight_mutate_power     = 0.5
weight_mutate_rate      = 0.8
weight_replace_rate     = 0.1

[DefaultSpeciesSet]
compatibility_threshold = 3.0

[DefaultStagnation]
species_fitness_func = max
max_stagnation       = 3
species_elitism      = 2

[DefaultReproduction]
elitism            = 3
survival_threshold = 0.2

Note 1: the previous character positions are the position in the previous frame, and if the game runs at 60 fps the previous position is not that different from the current one...
..Note 2: the game score is a bit more complex than just jumping on platforms, the bot should also be rewarded for combos that can make him jump higher. the combo system is already implemented but I first want to see the bot aiming to the next platform before he learns to jump combos.


texts:
I am new to Orange and it seems very useful for conducting exploratory data analysis.
One question I have is - can we use a widget to modify some values in the dataset?
For example, several numerical columns have some values of 0 which does not make sense and should be a "missing data". I do not want to ignore the line as some other column data are important. Also not all 0 should be replaced with "missing data" as for some column, a 0 is a valid data.
Can we do that with a widget? I could not figure it.
Thank you for the help!


texts: What is a manifold for Unsupervised Learning?

texts:
I've been watching Dr. G. Hinton lectures on Neural Networks in Machine Learning, and in one of the lectures he explains what the goals of Unsupervised Learning are.

I am having trouble understanding the part where high-dimensional inputs such as images live on or near a low-dimensional manifold (or several such manifolds). What is a manifold exactly, and why is this the case?

Thanks!


texts: How can I provide an answer to Neural Network skeptics?

texts:
After given several talks on NN's, I always have a skeptic that wants a real measure of how well the model is. How do you know the model is truly accurate?

I explain the use of test data etc. to evaluate the total error, however, there is always someone who wants to know about the error associated with each weight?

Can anyone enlighten me as to how I can satisfy these types of questions?

It has become a real issue.


texts: Setting up RNN in TensorFlow for time series forecast with variable input series lengths

texts:
I am building a model with keras for time series prediction. The structure of the problem is as follows:

The input is a time series of 5 numeric features
The output is a binary class (0 or 1)
The model needs to predict on variable lengths of input data. For example, in some cases I might only have the first 2 days available for a sample, and in others I may have 7 days.

Here is how I am setting up the layers in the model:

import tensorflow as tf

model = tf.keras.Sequential()
model.add(tf.keras.layers.LSTM(64, input_shape=(None, 5))
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))

I am seeing good performance when training on sequences of a constant size and predicting on that same size. However, I am not sure how to best configure and train the model to work on variable-length inputs.

I'm wondering if there might be zero-padding going on under the hood when the sequences aren't all the same length. Values of 0 are common and meaningful for the features I am using and so filling in missing data with 0's wouldn't make sense to do in this case.

I'm also wondering how people usually structure their training set in this case. I could do any of the following:

Include each sample only once with a random input length
Include each sample once per possible input length, with a different input length each time
Include each sample only once, with the maximum input length

The third doesn't seem right as the model probably needs to see training examples with shorter lengths in order to be able to predict on them.


texts: Learning image embddings for clustering based on custom distance metric

texts:
I have a large dataset of images, and i can calculate their distance to oneanother. I will at a later point recieve new images where i can not determine their distances to the ones in my training set(or anyone else for that matter). My idea was to cluster these images, and then infer similar features for the new images, as those in the same cluster. I tried using the vgg16, vgg19, and resnet50 to encode the images and then use DBSCAN to cluster. It worked decently, however from my experience, DBSCAN does not work optimally in such a high dimensional space.

I found this on here

  Another way is to learn an embedding that optimizes your similarity metric using a neural network and just cluster that

But i am not sure what is meant here, how exactly should i train the network, in my mind i would have to define a loss function that punishes bad encodings. I though of simply taking the square distance between the calculated distance, and the distance in the encoded space, however i feel like there are multiple issues with that.

I am most comfortable with keras, so if anybody know a repo or something along those lines, i would be grateful aswell!


texts: KerasClassifier.fit VS model.fit giving different results


texts: How to Manipulate data for multiple visits per person?

texts:
I have a query to solve. I have data regarding customers and number of visits done to them. These are in two tables. So I want to join two table and create different features so that I can find better/lagging customers from the data based on scoring.

customer_dataset:

id(unique id for customer)
name
assigned_user_id(user assigned)
company_id(company_which user works for)
expected value(revenue expected)
status(whether a lead/customer([0,1]categorical))

visit_dataset

customer_id(unique_id)

user_id
company_id
date(visit date done)
schedule_date(actually scheduled)
next_action_date(if any next action)
status(visit done or not)

So I need to score lead/customer on the basis of these data and find a list of top user. Also from the number of visits done, I need to figure out how many average/sum of visits done per customer so that I can find next forecast of visit(In case he doesnt go or forget about lead).

I'm new in manipulating data with pandas. How can I;

find total visit done per customer and other features from dates?
any resource that I can follow for how to work with dates data in pandas?
I'm trying to use featuretools to get the features.(any other advise)?
also I'm trying to automate this feature. So I'll feed in last 30 days visits as data and then I have to figure out the next visit date(prediction)Important (I so don't know how will i do that but trying ;P)

Sorry If i missed any info, please give me feedback if any issue.

Thanks a lot.


texts: How to apply a groupby rolling function to create multiple columns in the dataframe

texts:
I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data is here:

Timestamp          Close    High    Low     Open    Volume
2017-09-27 06:30:00 927.74  927.74  927.74  927.74  14929
2017-09-27 07:00:00 933.0   935.53  928.47  928.745 136776
2017-09-27 07:30:00 932.365 934.0   930.065 932.98  73146
2017-09-27 08:00:00 932.39  933.14  930.3   932.45  56624
2017-09-27 08:30:00 934.8675 935.62 931.1495 932.15 74960
2017-09-27 09:00:00 934.325 935.96  933.6 935.2308  62358
2017-09-27 09:30:00 935.48  935.7   933.56  934.36  45043
2017-09-27 10:00:00 935.5   935.56  933.71  935.385 32434
2017-09-27 10:30:00 935.96  936.86  935.21  935.5   58203

Here the author gets the output by applying the function over a slice of rows the index and apply the main function by applying this code:

mp = MarketProfile(df, tick_size=1)
mp_slice = mp[df.index.max() - pd.Timedelta(13, 'H'):df.index.max()]

Here the function takes a slice of indexes from whole dataframe and apply its function in that slice and the author get his results through this code:

mp_slice.as_dict()

What I want is to make rolling(w) of indexes and apply that function to the whole Data frame in pandas of index and make new columns in the data frame from the starting date. i.e df['poc_price'], df['value_area'], df[initail_balane'].etc. (all that includes in the as_dict() function output).

i tried many codes but although nothing get the final output that i want. My least code is:

def make_marketprofile(df, window=50, mode='vol', tick_size=1):
for i in range (0, df.shape[0] - window - 1):
    try:
        mpt = MarketProfile(df[i:i+window], mode=mode, tick_size=0.5)
        mpt_slice = mpt[df[i:i+window].index.min() :
                    df[i:i+window].index.max()].as_dict()
    except KeyError:
        print('error: {}'.format(df_1['Date'][i+window]))
#        for key, value in mpt_slice():
    df.iloc[i+window] = mpt_slice #value
    try:
         df.drop(['Date'], axis = 1, inplace = True)
    except AttributeError:
         pass
for column in df:
    df[column][1:] = pd.to_numeric(df[column][1:])
    df = df.iloc[window:]
    df.fillna(method = 'ffill', inplace = True)
return df

but it throws value error:

ValueError: Must have equal len keys and value when setting with an iterable


texts: what happens to the sample points on pruned leaves

texts:
I think I understand what pruning is (in concept) in a decision tree. What is not so clear to me (in reading) is what happens to the pruned observations. What do you do with them? Are they just simply not used and forgotten?
What bothers me is the "hole" left behind by the pruned node. What if a future $Y$ needs to be predicted at that node? Do you go to the parent node and treat that parent node as an end node and predict $Y$ by the data at that parent node (even though some data at the parent node continue to be divided in sub-nodes, other than the pruned node)?


texts: How to handle features which are not always available?

texts: I have a feature in my feature vector that is not always available respectively sometimes (for some samples) it makes no sense to use it. I feed a sklearn MLPClassifier with this feature vector. Does the neural network learn by itself when the feature make sense to use for its decision or do I have to add a flag in the feature vector which says for example “1” if it makes sense or “0” if it does not make sense.


texts: Specific data formatting techniques for discontiguous time series?

texts:
I'm facing a predicting problem for food alerts. The goal is to predict the variables of the most probable alert in the next x days (also any information I could get about future alerts is really useful for me).

Here is a sample of the dataset that I'm using, where alerts are recorded over time (so it's a time series problem):

The problem is that observations are not uniform over time (not separated by equal time lapses), i.e: since alerts are only recorded when they happen, there can be one day without alerts and another with 50 alerts.

The entry for the possible model could be the alerts (each alert correctly coded as they are categorical variables) of the last x days, but this entry must have a fixed size/format. Since the time windows don't have the same number of alerts, I don't  know what is the correct way to deal with this problem.

Any data formatting suggestion to make the observations uniform over time?

Or should I just face the problem in a different way (different inputs)?

Thanks.


texts:
My question is related with model-weights optimization during back propagation.
In this image I'm trying to represent an auto-encoder having 7 layers where 4th one is center layer.

If my understanding is correct, during back propagation outcome of jth+1 layer, will help in optimizing the weights of jth layer. If it is correct, I want to restrict the weights training of Encoder layer and want to perform my customized training.

Customized training should be like this:

Till layer 4 normal back propagation.
Layer 3 weights will get trained(optimized) using the output of 4th layer as mentioned in PINK arrow.
Layer 2 weights will get trained with the outcome of layer 5 as mentioned in YELLOW color arrow.
Layer 1 weights will get trained with the outcome of layer 6 as mentioned in BLUE color.

I want to know how I can  implement such customized trainable architecture?I'm familiar with Tensorflow, but I'm not able to figure out any solution and kind of stuck.

NOTE: Bear with my drawing.


texts: There are lots of noise removal algorithms for offline dataset. I would like to ask if there are some suitable online implementations to remove noise of the data in real-time (similar to Kalman Filtering- though I could not come up with suitable parameters to successfully use it.)


texts: Feature selection with L1 regularization on sklearn's LogisticRegression

texts: I'm using sklearn's LogisticRegression with penaly=l1 (lasso regularization, as opposed to ridge regularization l2). Lasso is causing the optimization function to do implicit feature selection by setting some of the feature weights to zero (as opposed to ridge regularization, which will preserve all features with some non zero weight). Is it possible to extract this weight information (feature selection) from sklearn somehow?


texts: What is the difference between SVM classification error, SVM margin error, and SVM total error ? Is there any clear definition for them ? And what is C parameter in SVM ? Its totally confusing me !!!


texts:
I have to impute missing values in the column named <Age> with the mean of the nearest available values above and below in the column <Age>.

If the Age column had values in order

NA,7,6,NA,7,8,NA,NA,NA,10,5,NA,NA,5,9,9,12,8,6,NA,NA

After imputation, column should look like
 7,7,6,6.5,7,8,9,9,9,10,5,5,5,5,9,9,12,8,6,6,6

I know about filling value with mean, mode, or median forward filling, backward filling but not this one. I am new to this field and a student. Any help would be appreciated.

Thanks


texts: Dying gradient issue in Graph Neural Networks

texts:
I am using Pytorch-Geometric library to implement a Graph Convolutional Layer(GCN) followed by few linear layers for a prediction task. But after training on graphs with np. of nodes being 10K and no. of edges being ~ 20K-40K for around 50 epochs in a batch-gradient descent manner, the gradient values hit close to zero values and show a terribly bad gradient flow during backpropagation. How can this be handled to ensure a proper gradient flow for weight training?
For reference, I've pasted the model code below:
import torch
from torch.nn.parameter import Parameter
import torch.optim as optim
import torch.nn.functional as F
import torch.nn as nn
from torch_geometric.nn import GCNConv

class Model(nn.Module):
    def __init__(self, nin=1, nhid1=128, nout=128, hid_l=64, out_l=1):
        super(Model, self).__init__()

        self.gc1 = GCNConv(in_channels= nin, out_channels= nhid1)
        self.gc2 = GCNConv(in_channels= nhid1, out_channels= nout)
        self.lay1 = nn.Linear(nout ,hid_l)
        self.l0 = nn.Linear(hid_l,hid_l)
        self.l1 = nn.Linear(hid_l,hid_l)
        self.lay2 = nn.Linear(hid_l ,out_l)
        self.active = nn.LeakyReLU(0.1)

        with torch.no_grad():
            self.gc1.weight = Parameter(nn.init.uniform_(torch.empty(nin,nhid1),a=0.0,b=1.0))
            self.gc1.bias = Parameter(nn.init.uniform_(torch.empty(nhid1),a=0.0,b=1.0))
            self.gc2.weight = Parameter(nn.init.uniform_(torch.empty(nhid1,nout),a=0.0,b=1.0))
            self.gc2.bias = Parameter(nn.init.uniform_(torch.empty(nout),a=0.0,b=1.0))
            self.lay1.weight = Parameter(nn.init.uniform_(torch.empty(hid_l, nout ),a=0.0,b=1.0))
            self.l0.weight = Parameter(nn.init.uniform_(torch.empty(hid_l, hid_l),a=0.0,b=1.0))
            self.l1.weight = Parameter(nn.init.uniform_(torch.empty(hid_l, hid_l),a=0.0,b=1.0))
            self.lay2.weight = Parameter(nn.init.uniform_(torch.empty(out_l,hid_l),a=0.0,b=1.0))

    def forward(self, data):
        x, adj = data.x, data.edge_index
        x = self.active(self.gc1(x, adj))
        x = self.active(self.gc2(x, adj))
        x = self.active(self.lay1(x))
        x = self.active(self.l0(x))
        x = self.active(self.l1(x))
        x = self.active(self.lay2(x))

        return x

Below is the gradient flow across the layers:


texts: Parameter optimization and selection in dynamic neural networks

texts:
I have used a Bayesian optimization to tune machine learning parameters. The optimized parameters are "Hidden layer size" and "learning rate". Now I have 2 questions while dealing with Dynamic Neural Networks:

I have 4 datasets i.e (House 1, house 2, house 3, house 4) as shown in below table. The program is executed 4 times in a loop for each dataset with the same dynamic algorithm. Will the optimized parameters remain the same for each dataset. For instance, can we use the optimized parameters of House 1 for House 4 also? Conventionally, I have seen in academic papers authors setting only 1 set of optimized parameters for various datasets in static neural networks.

In dynamic networks, where the network architectures conditionally change with every input sample.How can we claim that this is the optimal parameter for the network with minimum loss? Although the parameters vary in a small amounts on every new run of the script.


texts: Why does a class weight fraction improve precision compared to under-sampling approach where precision drops?

texts:
I have an imbalanced data where the ratio between positive to negative samples is 1:3 (positive samples are 3 times higher than negative). For my case it is is important to have a higher precision (and lower FPR) even if it comes at the cost of low recall (higher FNs). I intend to reach this goal by training a random forest with a class weight of negative class many times more than the positive class and observe that precision increases (lesser FPs) as I increase the weight of negative class and recall drops (more FNs). This was kind of what I expected. See a sample table below:

Next I also try to see if I can also train the model using an intentionally imbalanced data to introduce a bias towards classification of negative class. I try to achieve this by under-sampling my positive class with different fractions. So 1:10 means my negative samples are ten times more than positive in the training phase. What I observe now is that both the precision and recall go lower as I keep decreasing the number of positive samples in the training (hence making the negative class majority). Why is that Precision drops in this case although the FPR is decreasing? Should precision and FPR not be inversely proportional? Thanks


texts: Jupyter, Python: the kernel appears to have died while training a model on a big amount of data

texts: I am training my model on almost 200 000 images, i'm using Jupyter and now after 3 days of training ( i used 800 epochs and batch-size = 600) I have this " the kernel appears to have died. It will restart automaticaly" And this appears after 143 epochs only. Can anyone help me to solve this, and also can anyone advise me something in case of using big amount of data, because i am struggling with this dataset and I can't retrain the model each time the Jupyter blocks. Infact, I'm working on my internship project so I have to use all the data. I will be so grateful for your help.


texts: Classification with Orange error "test and train datasets have differents target variable"

texts:
m having a problem on the widget "Test and score" used for classification. Im doing a classification on images and i would like to use the widget "test and score" with the option "Test on test data".
I used select columns to select the feature "category" as target variable for the two datasets.
The training dataset is ordered in folders (so in class) with 1 image per folder. The feature "category is created by orange using the folder architecture.
For the test dataset the widget "create feature" create the feature "class_name" using a substring of each image and then I create the target variable "category" using the widget "create class". Finaly the feature category is setted as target variable with "select columns" widget.
The two datasets are passed to the "test and score" widget with the option "test on test data"
However I get the error "Test and train dataset have different target variables".
I checked my datasets and i have the same target variable: category.
Error message Test and train dataset have different target variables
<img src="https://i.stack.imgur.com/x3XJF.png" alt="
" />

Here is the widget "select columns" showing that my two target variable are setted. The variable class_name is a column to drop


texts: SimCLR: What is the purpose of masking correlated samples in this code?

texts:
import torch
import torch.nn as nn

class NT_Xent(nn.Module):
    def __init__(self, batch_size, temperature, device):
        super(NT_Xent, self).__init__()
        self.batch_size = batch_size
        self.temperature = temperature
        self.mask = self.mask_correlated_samples(batch_size)
        self.device = device

        self.criterion = nn.CrossEntropyLoss(reduction="sum")
        self.similarity_f = nn.CosineSimilarity(dim=2)

    def mask_correlated_samples(self, batch_size):
        mask = torch.ones((batch_size * 2, batch_size * 2), dtype=bool)
        mask = mask.fill_diagonal_(0)
        for i in range(batch_size):
            mask[i, batch_size + i] = 0
            mask[batch_size + i, i] = 0
        return mask

    def forward(self, z_i, z_j):
        """
        We do not sample negative examples explicitly.
        Instead, given a positive pair, similar to (Chen et al., 2017), we treat the other 2(N − 1) augmented examples within a minibatch as negative examples.
        """

        p1 = torch.cat((z_i, z_j), dim=0)
        sim = self.similarity_f(p1.unsqueeze(1), p1.unsqueeze(0)) / self.temperature

        sim_i_j = torch.diag(sim, self.batch_size)
        sim_j_i = torch.diag(sim, -self.batch_size)

        positive_samples = torch.cat((sim_i_j, sim_j_i), dim=0).reshape(
            self.batch_size * 2, 1
        )
        negative_samples = sim[self.mask].reshape(self.batch_size * 2, -1)

        labels = torch.zeros(self.batch_size * 2).to(positive_samples.device).long()
        logits = torch.cat((positive_samples, negative_samples), dim=1)
        loss = self.criterion(logits, labels)
        loss /= 2 * self.batch_size
        return loss

I am struggling to understand the essence of masking the correlated samples. From the SimCLR paper, I understand 2 augmented views are created for each sample in the mini-batch.
Is this the same as masking the correlated samples? If yes, how?
Here's the paper.


texts: How to visualize two small samples with different sizes?

texts:
As part of a psychology experiment, I surveyed 15 users. I separated them in two groups, those who have become smoke free, and those who are still smoking. These groups have different sizes (9 and 6).

I want to show graphically that the smoke free group reported higher values of the survey index. This is my current best attempt, with Google Sheets:

However, I think that a correct comparison should have the same number of data points. I guess I need some form of interpolation, but not sure how to proceed.

An alternative approach is using a bubble chart:

My questions:
1. Would you recommend a different chart?
2. Should I have the same number of data points? If yes, how to achieve that?


texts: Good dataset for sentiment Analysis in Tickets for IT Support

texts:
I am working on Sentiment Analysis in Tickets for IT Support, But I have only seen the movie reviews and Twitter sentiment analysis which are not related to my work. I found a research paper related to my research work on this link  https://ieeexplore.ieee.org/document/7832903/  but they have denied giving the datasets. For my model, It gives 89% accuracy for the IMDB datasets but for IT purpose It doesn't.

Can anybody suggest some other datasets which can help me to increase the result?


texts: Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

texts:
As a part of my university project, I am researching/developing a sentiment analysis model wherein I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning trained classifier for prediction of Sentiments on Twitter data.

Detailed description -

To explain what I am trying to do is - Combine Machine Learning classifier and NLTK Vader sentiment analysis to get the better classification of tweets as Positive, Negative or Neutral.

What I have done is -

Cleaned the data (Niek Sanders twitter corpus) and pre-processed the tweets which include stop words removal, URL removal, User mention removal, remove the # symbol from Hashtags, lower case conversion, stemmer processing, etc.
Split into 80:20 training: test ratio
Used TfidfVectorizer to create a sparse matrix of features with TFIDF of words. The number of columns is equal to the number of words in clean data.
Use this vector matrix to train and test the classifiers using scikit-learn.

Classifiers used - This same vector matrix is being used to train - KNN, Random forest, Naive Bayes, SVM, Artificial Neural Network and Convolutional Neural Network.

Now the main doubt arises when trying to combine NLTK Vader (SentimentIntensityAnalyzer results).

What I am doing is - From step three above, in the tfidf vector-matrix I am adding 2 columns and I am adding the Positive and Negative polarity result for the tweet given by NLTK, so now the vector-matrix has 2 new columns n+1 (total positive polarity of tweet by NLTK), n+2 (total negative polarity of tweet by NLTK) and it looks like -

    0      1     2     3     4     5     6     7  ......... |  n+1    |   n+2
------------------------------------------------------------|---------|----------
    0.4   0.3   0.4   0.1   0.5   0.3   0.2   0.4 ......... |  0.345  |  0.345
    0.5   0.3   0.2   0.8   0.3   0.6   0.4   0.5 ......... |  0.765  |  0.523
    0.6   0.4   0.1   0.7   0.8   0.8   0.2   0.2 ......... |  0.392  |  0.664
    0.2   0.9   0.7   0.4   0.9   0.9   0.8   0.5 ......... |  0.832  |  0.658
    0.9   0.5   0.9   0.7   0.3   0.2   0.2   0.5 ......... |  0.273  |  0.283
    0.5   0.2   0.2   0.7   0.2   0.1   0.6   0.6 ......... |  0.505  |  0.194
    0.4   0.3   0.2   0.3   0.3   0.9   0.5   0.5 ......... |  0.102  |  0.927
    0.1   0.8   0.1   0.2   0.1   0.5   0.2   0.7 ......... |  0.735  |  0.455

Question - So is it correct to add it this way?

I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

Though it has increased the overall accuracy of the classifiers by small margin of 1-2%.

But am I doing it right?
If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.


texts: Filenotfounderror: [errno 2] no such file or directory: 'nsr001.ecg'

texts:
I am trying to save contents of physiobank Normal Sinus Rhythm RR Interval Database into a numpy array but I keep getting an error:
Traceback (most recent call last):
  File "AverageRRI.py", line 20, in <module>
    averageArray = np.fromfile(file,dtype=float)
FileNotFoundError: [Errno 2] No such file or directory: 'nsr001.ecg'

but the file does exist in the directory.
import os
import numpy as np

for root, dirs, files in os.walk('normal-sinus-rhythm-rr-interval-database-1.0.0'):
    for file in files:
        if file.endswith(".ecg"):
            print(file)
            averageArray = np.fromfile(file,dtype=float)
            print(averageArray)

When I add the pathname like:
averageArray = np.fromfile('normal-sinus-rhythm-rr-interval-database-1.0.0/nsr001.ecg',dtype=float)
            print(averageArray)

It works.
Thanks so much!


texts: Why Markov Chain Monte Carlo allows sampling from a large class of distributions and scales well with the dimensionality of the sample space?

texts: How can I understand "Markov chain Monte Carlo allows sampling from a large class of distributions and which scales well with the dimensionality of the sample space"


texts: Curve Fitting By Predict polynomial degree with ANNs

texts: My Science Fair Project is on Curve fitting with neural nets as an alternative to polynomials. I know this is a widely done procedure in datascience but I wanted to do my own research on it. Currently the biggest problem is training time, and to solve that I had an idea. What if I I trained an ANN to predict the degree of the polynomial that best fits the model then use standard polynomial regression to curve fit. This would make the training only happen once so I could train for as long as I wanted, and thus it would increase speed by a lot. What are your thought in this? Has anyone done this before? Is it plausible?


texts:
In the paper Batch Normalization: Accelerating Deep Network Training b
y Reducing Internal Covariate Shift (here) Before explaining the process of batch normalization the paper tries to explain the issues related with (I am not getting what the exact issue addressed here is).

excerpt from section 2, para 2:

  We could consider whitening activations at every training
  step or at some interval, either by modifying the
  network directly or by changing the parameters of the
  optimization algorithm to depend on the network activation
  values (Wiesler et al., 2014; Raiko et al., 2012;
  Povey et al., 2014; Desjardins & Kavukcuoglu). However,
  if these modifications are interspersed with the optimization
  steps, then the gradient descent step may attempt
  to update the parameters in a way that requires
  the normalization to be updated, which reduces the effect
  of the gradient step. For example, consider a layer
  with the input u that adds the learned bias $b$, and normalizes
  the result by subtracting the mean of the activation
  computed over the training data: $\hat x= x − E[x]$ where
  $x = u + b, X = {x_{1...N}}$ is the set of values of $x$ over
  the training set, and $E[x] = \frac 1 N(\sum_{i=1}^nx_i)$
  .

  If a gradient
  descent step ignores the dependence of E[x] on b, then it
  will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then
  $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$.

  Thus, the combination of the update to b and subsequent
  change in normalization led to no change in the output
  of the layer nor, consequently, the loss. As the training
  continues, b will grow indefinitely while the loss remains
  fixed. This problem can get worse if the normalization not
  only centers but also scales the activations.

here is my understanding of the literature:

We have a batch of size N (One training batch)
Let there be two arbitrary hidden layer connected to each other (L1 and L2) connected by parameters $W$ and $b$
output coming out of L1 is x1
$u = x1W$ (this is where the literature above starts. dimension of u is MxN) (M is the number of units in L2)
$x = u+b$ (dimension b = dimension x = dimension u = MxN)
Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x= x − E[x]$)
We compute the loss and backpropogate the gradient and update just this layer's $b$ in order to give it a sanity test. New $b$ = $b + \Delta b$
We run it again on the same batch with updated $b$
repeat 3 and 4
$x_{new} = u+b + \Delta b$ (dimension b, $\Delta b$ = dimension x = dimension u = MxN)
Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x = x + \Delta b − E[x + \Delta b] = x - E[x]$).
which is the same as what was calculated before updating b and hence updating b had to effect on the training

My question is with this part of the excerpt:

"If a gradient
descent step ignores the dependence of E[x] on b, then it
will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then
$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$."

Why is

"
$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$." dependent upon what comes before it? What is even the point of that bit? Please also note the useage of word "Then" (made bold) implying the statement necessarily draws causality from what comes before


texts: Is this loss function useful for quantized training data

texts:
For training a classification/regression neural network (with TensorFlow), I implemented the following loss function with the aim to prevent overfitting.
Any Ideas if this could be useful?
The idea is a dead zone for the loss (error smaller than 1 percent).
In my case, my training data is quantized. However, the trained model should not return quantized values. So I think dropout or similar will not work here.
def myloss(y_true, y_pred):
  sq = tf.square(y_true - y_pred)
  sq = tf.nn.relu( sq - tf.square(y_true) * 0.01)
  return tf.reduce_mean(sq,axis=-1)

Any ideas?
Best regards.


texts: How general are the possible computations of a tensorflow Graph?

texts:
Tensorflow works through the creation of computational graphs. I know about tensorflow in the context of machine learning, specifically deep learning. But I know that tensorflow can also be used for wildly different data processing tasks, such as the solving of partial differential equations.

Can the computational Graphs in tensorflow in principle be used to define any arbitrary computational model? (e.g. as an extreme example, could it in principle be used to design the circuitry of a microprocessor?)


texts: I have tried 5 different types of model but all returns really low training accuracy (~64%) and low testing accuracy (~14%). What should I do?

texts:
I am working with a typical regressor problem. There are $6$ features in the dataset that I am concerned with. There are about $800$ data points in my dataset. The features and the predicted values have high non-linear correlation so the features are not useless (as far as I understand). The predicted values have a bimodal distribution so I disregard linear model pretty quickly.

So I have tried 5 different models: random forest, extra trees, AdaBoost, gradient boosting and xgb regressor. The training dataset returns $%64$ accuracy and the test data returns $15%$. Both numbers scare me haha. I try tuning the parameters for the random forest but seems like nothing particularly make a drastic difference.

What should I do next if you guys don't mind giving me a recommendation?

Function to tune the hyperparameters

def hyperparatuning(model, train_features, train_labels, param_grid = {}):
    grid_search = GridSearchCV(estimator = model, param_grid = param_grid, cv = 3, n_jobs = -1, verbose =2)
    grid_search.fit(train_features, train_labels)
    print(grid_search.best_params_)
    return grid_search.best_estimator_`

Function to evaluate the model

def evaluate(model, test_features, test_labels):
    predictions = model.predict(test_features)
    errors = abs(predictions - test_labels)
    mape = 100*np.mean(errors/test_labels)
    accuracy = 100 - mape
    print('Model Perfomance')
    print('Average Error: {:0.4f} degress. '.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%. '.format(accuracy))


texts: Constituency vs Dependency Parsing: What is more effective for Sentiment Analysis?

texts:
Parsing is often used to understand the sentiment of complex sentences filled with double negations or very articulated.
There are two main ways of parsing a sentence: Constituency and Dependency Parsing. What is the most successful application for Sentiment Analysis?


texts: How does Markov Chain Monte Carlo work in a neural network from beginning to end?

texts:
I am trying to reason through how this works from the beginning of the neural network model to the end from a high level and I want to make sure my understanding is correct, I have probably misunderstood something somewhere....

Start with random weight initialization in the network, at this point, bayes theorem looks something like this, since we have to start somewhere, and the random weights should have a mean of zero...

$$
p(\theta | X) \propto p(X|\theta)p(\theta) \text{ where } p(\theta) \sim \mathcal{N}(0, 1)
$$

Minimize the error for a given number of iterations, add some gaussian noise to all the weights and then minimize them again.
Do step 2 repeatedly for a fixed number of iterations.
End up with a discrete set of model weights that approximate the distribution of possible weights
Make predictions with all of the weights, which are weighted according to their distance from the mean
Take the mean and variance of the predictions as the final prediction which has the mean as the prediction and the variance as the uncertainty.


texts: How to sample with guaranteed minimum spacing?

texts:
I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of sampling data points randomly, but with at least 12 hours between them, and then treating them as IID data.
How do I do this algorithmically, in an efficient way?

ps if I want at most one data point per 12 hour window, I could do like below:
import pandas as pd
import sklearn
import numpy as np

ndata = 2*24*60*60
s = pd.Series(data=np.random.random(ndata),index=pd.date_range(start=pd.Timestamp("2000-01-01"), periods=ndata, freq='S'),name='data')
window = pd.Series((s.index - s.index[0]).total_seconds()//(12*60*60),index=s.index,name='window_id')
df = pd.concat([s,window],axis=1)

random_datapoints = sklearn.utils.resample(df,stratify=df.window_id,n_samples=df.window_id.unique().size,replace=False)

However, most probably (this happens almost surely as $n\to\infty$, Ill get two samples that are less than 12 hours apart, even though they will be 12 hours apart on average....


texts: I have a dataset with categorical features. I want to segment the data using clustering techniques. What could be the possible choices for this scenariogiven the fact that data has categorical features. Is there any variation of k-means which can be used here.


texts: Can I conclude my finding with just one linear regression result?

texts:
I'm trying to learn Linear Regression and I'd like someone to look at my notebook to see if I miss out anything at all. How how else I can tune my model better.

Here's my notebook.

I get the dataset from here which is really simple.

http://college.cengage.com/mathematics/brase/understandable_statistics/7e/students/datasets/slr/frames/frame.html

%matplotlib inline
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

df = pd.read_csv('slr12.csv', names=['annual', 'cost'], header=0)
df.describe()

df.plot(kind='scatter', x='X', y='Y');

slope, intercept, r_value, p_value, std_err = stats.linregress(df['X'], df['Y'])

plt.plot(df['X'], df['Y'], 'o', label='Original data', markersize=2)
plt.plot(df['X'], slope*df['X'] + intercept, 'r', label='Fitted line')
plt.legend()
plt.show()

So from this trend we can predict that if you annual fee is high then you need your startup cost will be high as well.


texts: Problems in building model predicting probablity of student's admission

texts:
I'm trying to build a model predicting the probability of a student's admission in Russian educational system. There are exams for every school subject. The student can take several of them. Based on the results of the exams, the student could be admitted to some faculty (for example, Chemistry, Computer Science, Economics, Politics).

I use scikit-learn to build the model. I tried Naive Bayes, Random Forest and another classification estimators, but there was no much difference. I used method predict_proba to get probabilities, and a dataset with scores for every taken exam as features and faculty as target (faculty is treated as class)

The main problem is that when adding more taken exams, the probability of faculty decreases, while it should increase or not change.

My guess is that the model gives the most popular faculties among students with such scores, not probability of admission. Is there something I can do to improve the model?


texts:
I have a large set of data (about 8GB). I would like to use machine learning to analyze it. So, I think that I should use SVD then PCA to reduce the data dimensionality for efficiency. However, MATLAB and Octave cannot load such a large dataset.

What tools I can use to do SVD with such a large amount of data?


texts: Illustrating the dimensionality reduction done by a classification or regression model

texts:
Tl;DR: You can predict something, but how do you explain the prediction?

Your usual classification/regression setup
Lets say the data is a classic regression/classification problem: several numerical columns, several nominal columns, and an event which we are trying to predict:

user1, age:18, wealth:20000, likes:tomatoes, isInBigCity:yes, hasClicked:yes
user2, age:25, wealth:24000, likes:carrots , isInBigCity:no , hasClicked:no
...

With the help of Random Forests, SVM, Logistic Regression, Deep Neural Network, or some other method we export a model that can output a probability of the event hasClicked:yes for a new user faced with the choice of clicking.

Extracting the inner topology surfaced by a model
Now, those algorithm do some dimensionality reduction, reducing those inputs to a single probability. My question is: how would you extract what those models are doing and show the dimensionality reduction steps to a human? How would you illustrate the inner topology of the dataset with regards to the predicted class?
I am looking for either:

Visualizations of what a model produced by Random Forests, SVM, Logistic Regression, Deep Neural Network is doing.

Clusterers being extracted from regression/classification models (Surely a single decision tree can be viewed as a hierarchical clusterer)

A model-specific way to project the input data in a space where the Euclidian distance of T-SNE makes sense.

A way to learn a T-SNE-compatible distance out of a regression/classification model.

Clustering methods that optimise the separation of one variable while not using it to cluster.

Clusterers built out of regression/classification models

The goal is to extract some sort market segmentation based on the behaviour of users. And give a high level visualization of it. Something that would expose clearly the reasons why some users transform better than others.
EDIT: Let's reduce the question to one estimator: Random Forests; what would be your answer?


texts:
I've tried a number of experiments with machine learning. From trying to use GANs to upscale images to playing with auto-encoders.
There is one problem that haunts me and always ends up ruining my experiments.
My networks always seem to "learn" to ignore the input and always produce the same image output. This output is often an interesting blend of the input images.
It seems that it's learning to cheat the test and produce the same image regardless of the input.
Here are some examples:

This has happened in both my GAN and Auto-encoder projects.
How do I avoid this sort of thing happening?


texts:
What does the "randomly shuffle training samples" in stochastic gradient descent attain?

I interpreted that since the training samples are used to compute

$$\hat{y}=f(w^t x)$$

so if the order of $x$s changes, then the weights will be assigned "based on different order"?

Although, since $w^tx$ is linear (order doesn't matter), then where is the effect of this seen?

Or maybe it's not seen in $\hat{y}$ but in the LMS update rule:

$$\Delta w_{ij}^k=\lambda(\hat{y}_i^k-y_i^k)\color{red}{x^î_j} $$


texts: How to learn noSQL databases and how to know when SQL or noSQL is better

texts: I want learn about NoSQL and when is better to use SQL or NoSQL. I know that this question depends on the case, but I'm asking for a good documentation on NoSQL, and some explanation of when is better to use SQL or NoSQL (use cases, etc). Also, your opinions on NoSQL databases, and any recommendations for learning about this topic are welcome.


texts: In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models?

texts:
How many definitions of attention are commonly employed for Deep Learning tasks?

That's what I've encountered up to now:

Self-attention
Bahdanau
Luong
Multi-Head (used in Transformers)

Could you provide formal explanations of each of these (and others in case the list is incomplete) and tips on when to prefer one to the other?

And what is the history of Attention models? How did they develop through time, and how did they improve previous formulations?


texts:
I am using the fashion MNIST dataset to try to work this out:

I am using the data from the links:

Training :
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz

training set labels:
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz

test set images
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz

test set labels
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz

I use the code to open the dataset:

def load_mnist(path, kind='train'):
    import os
    import gzip
    import numpy as np

    """Load MNIST data from `path`"""
    labels_path = os.path.join(path,
                               '%s-labels-idx1-ubyte.gz'
                               % kind)
    images_path = os.path.join(path,
                               '%s-images-idx3-ubyte.gz'
                               % kind)

    with gzip.open(labels_path, 'rb') as lbpath:
        labels = np.frombuffer(lbpath.read(), dtype=np.uint8,
                               offset=8)

    with gzip.open(images_path, 'rb') as imgpath:
        images = np.frombuffer(imgpath.read(), dtype=np.uint8,
                               offset=16).reshape(len(labels), 784)

    return images, labels

label = ['T-shirt/top',  'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt',
         'Sneaker', 'Bag', 'Ankle boot']

data_dir = './'
X_train, y_train = load_mnist('D:\book', kind='train')
X_test, y_test = load_mnist('D:\book', kind='t10k')

X_train = X_train.astype(np.float32) / 256.0
X_test = X_test.astype(np.float32) / 256.0

I am trying to build a Convolutional Neural Network with the following architecture:
- Convolutional Layer with 32 filters with size of 3x3
- ReLU activation function
- 2x2 MaxPooling
- Convolutional Layer with 64 filters with size of 3x3
- ReLU activation function
- 2x2 MaxPooling
- Fully connected layer with 512 units and ReLU activation function
- Softmax activation layer for output layer
For 100 epochs using the SGD optimizer

My Code is:

X_train = X_train.reshape([60000, 28, 28, 1])
X_train = X_train.astype('float32') / 255.0
X_test = X_test.reshape([10000, 28, 28, 1])
X_test = X_test.astype('float32') / 255.0
model = Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=[28,28,1]))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))
model.summary()
y_train = keras.utils.np_utils.to_categorical(y_train)
y_test = keras.utils.np_utils.to_categorical(y_test)
model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=100)

But it is taking a lot of time for execution. It is like 30 minutes per epoch. I think I am doing something wrong in my code. Can someone help me figure that out?


texts: Entity Recognition to extract question,options and diagrams from a question paper?

texts:
Here is a sample question paper. I want to extract questions, their options and diagrams for the corresponding question(if any). Thinking of doing entity recogntion for the same. But I am not sure if that is possible for diagrams? Can someone help me out.
I tried OCR techniques previously but they are not doing well.


texts: Using a combination of gradient boosting with LSTM for classification?

texts:
I am presently using an LSTM model to classify high dimensional tabular data which is not text/images (dimensions 21392x1970). I also tried XGBoost (Gradient boosting) in Python separately for the same classification task (classify into one of 14 categories of different categorical values). I have come across the provision of using feature_selection_ method in XGBoost, which can provide me the F1 scores of the most relevant features in prediction.

I would like to create a hybrid model that combines the LSTM with the XGBoost but am confused as to how I can do something such as using the most important features for classification (probably getting these through XGBoost) and then feeding to an LSTM ?)by a combined approach. Any ideas, suggestions, and comments are appreciated!


texts:
According to several sources this formula, or the center originated version of it, is used to calculate an element of a convolution between an image $I$ and a kernel $K$ of size $k \times k$:
$$
(I*K)_{ij} = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} I_{(i-m),(j-n)}K_{mn}~~~~~~~~~~~~~~~~~~~~~(1)
$$
but it is also the cross-correlation of $I$ and the flipped version of $K$, so that:
$$
(I*K)_{ij} = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} I_{(i+m),(j+n)}K_{-m,-n}~~~~~~~~~~~~~~~(2)
$$
But calculating these two expressions do not yield the same result if the first expressions indices are interpreted as they normaly would.
For example
$$ (1) \Rightarrow (I*K)_{00} = I_{00}K_{00} + I_{0,-1}K_{01} + I_{-1,0}K_{10} + I_{-1,-1}K{11}$$
while
$$(2) \Rightarrow (I*K)_{00} = I_{00}K_{00} + I_{01}K_{0,-1} + I_{10}K_{-1,0} + I_{11}K_{-1,-1}$$
not only do these results differ, they also go out of bounds.
The only way I've been able to make the results equal is by interpreting the $I$ indices of equation (1) as such:
$$ I_{(i-m),(j-n)} = I[i-(k-1):i, j-(k-1):j]_{-m,-n} = I_{(i+(k-1)-m),(j+(k-1)-n)}$$
That is, by interpreting $ I_{(i-m),(j-n)}$ as the -m,-n element of the submatrix $I[i:i+(k-1), j:j+(k-1)]$. Where for $k = 2$ and $i,j = 0,0$:
$$
I[i:i+(k-1), j:j+(k-1)] = I[0:1, 0:1] =
    \begin{bmatrix}
    I_{00} & I_{01} \\
    I_{10} & I_{11} \\
    \end{bmatrix}
$$
and where $-m$ means row $(k-1)-m$ and $-n$ means column $(k-1)-n$.
This is my main source.
Can someone please explain to me how i should interpret the formulas so that (1) = (2). I'm very desperate for help.


texts: Word embeddings for Information Retrieval - Document search?

texts: What are good ways to find for single sentence (query) the most similiar document (text). I asked myself if word vectors (weighted average of the documents) are suitable to map a single sentence to a whole document?


texts:
I have a dataset containing 150k rows and 10 columns. After clustering, I would like to get clustering metrics. Below are lists of metrics that I would like to use>

 "C_index", "Calinski_Harabasz", "Davies_Bouldin", "S_Dbw", "Silhouette"

I have a problem. I've tried getting values using clusterCrit (from R) and sklearn (from python), but when I tried to get each value by using clusterCrit, it took a huge amount of time, so I couldn't get it.

I think this is because of the large data (my RAM is 8GB). Is there any way for me to get these values?


texts:
K-means may give different results, because the initial choice of centroids is random.

However, if I were to choose k=1, will the algorithm always provide the same answer equal to the "barycentre" of my data?


texts: Is there a model-agnostic way to determine feature importance?

texts:
Sklearn has a feature_importances_ attribute, but this is highly model-specific and I'm not sure how to interpret it as removing the most important feature does not necessarily decrease the models quality most.

Is there a model-agnostic way to tell which features are important for a prediction problem?

The only way I could see is the following:

Use an ensemble of different models
Either start with a big set of features and remove one at a time. To find the features "uplift", compare the ensembles quality with the full feature set against the ensembles quality against the removed feature set.

(What this can't do is to find connected features: Some features might be not exactly the same, but have a common underlying cause which is important for the prediction. Hence removing either of them doesn't change much, but removing both might change a lot. I ask another question for that.)


texts: Tensor Shape 'Reshape' during the dataflow

texts: I would like to combine an Autoencoder with a LSTM. However, the 'timestep' is a block for the implements and I would like to train them together. Is there a solution to the tensor 'reshape'? I mean, it's not the simple 'Reshape' but for using timestep data.


texts: How to keep the test data from leaking into the training process of a machine learning algorithm?

texts:
I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to keep the test data from leaking into the training process.

To avoid learning insignificant details of the data (which will increase the algorithm's generalization ability) I can further split my training data into the proper training set and a validation set multiple times, and choose the parameters of the algorithm that give the best average performance for all such splittings.

Finally I evaluate my algorithm on the test set, and get some numbers: MSE, RMSE, etc. But do these numbers really show, how good is my algorithm, and aren't they affected by the test set?

Of course, I didn't use my test data during training but the parameters of the algorithm that I got are valid only for this particular splitting into the training and test sets. If I split my data differently, I will get different MSE, RMSE, and different optimal parameters.

I can then do many different splittings into training/test, and calculate the error for each split. I that case how is it different from the cross-validation that I did during the first split? Do I need to further divide my training data into the proper data set and a validation set?

If I do many different splittings into the training an test sets, doesn't it mean that my test data leak into the training process because some data were in the test set for one splitting but in the training set for another splitting?


texts: How to train the models if the target is a 2d array?

texts: I have a dataset that contains 5 input parameters and the output target is a 2D array (time-series data saved in a separate file) for each set of 5 input parameters. How do I treat the output parameter for a regression problem?


texts: Correlating company entities between different data sources

texts:
I have two datasets with information about companies and my task is to correlate (match) companies from dataset A to companies in dataset B. Datasets are from different sources.

The columns in both datasets include fields such as company_name, country, state, city, address, zip.  All companies are in the US.

The problem is that even though we have the company_name on both sides - the names in A aren't equal to the names in B. So for example on A you might have Google and on B you might have Google Inc. Another example is Amazon and Amazon LLC. etc, there are many different variations to that.
These aren't typos, but just different representations of the same entity, one is more common and the other is more formal.
The addresses themselves aren't always the same as well. Probably b/c a company might have more than one address. (at least large companies do)

What is the best approach to correlate (match) these entities b/w these two data sources?
There are about 500k companies in each dataset.

A few ideas come to mind:

Soundex function on the company name (tried it, not great)
Levenshtein distance b/w names of each two potential matches (didn't try it yet, but it is O(n*m))
Levenshtein b/w the concatenated values of company names state, city, etc (also O(n*m))
Geocode the address and build a function that takes into account the Levenshtein distance as well as geographical distance.
Clean up the "INC", "LLC" and all other extensions and run any of 1-4.

What's your take? Any other suggestions?

Thanks!


texts: Can data science be applied to establishing the author of a text?

texts: There are many instances of quotes and texts where the true author may be disputed. Are there data science or machine learning techniques which may be useful to establish the authorship?


texts: Do I have to scale/normalize my training data for LSTM Classification, even if I only have one feature?

texts:
I have a time-series data as follows:
# Time, Bitrate, Class
  0.2,  312,     1
  0.3,  319      1
  0.5,  227      0
  0.6,  229      0
  0.7,  219      0
  0.8,  341      1
  1.0,  401      2

I am using only the "Bitrate" column as a feature, and "Class" for the labels for an LSTM classification model. In case of multiple features, I need to scale my data of course, to prevent domination from one feature to another. However, in my case, do I still need to scale/normalize my data, considering there is only one feature?
Thanks!


texts: Solving classification task with deep network

texts:
I am currently working on a classification problem:

The dataset (2000 features, 25 labels) I am using is seperable by using a 2-layered Multi-Layer-Perceptron (1 Input + 1 OutputLayer = 1 Weight Matrix) with achieving great accuracy.

Note: The Data I am using are spectrograms.

Now what I am curious about is the following: Can I use a deep network (more than 2 layers) to even increase the accuracy?

What I think about: Increasing the network structure would make the network interpret the data in a more complex way and therefore should be able to even correctly classify little "outbrakers" and "noisy elements" of the dataset.


texts:
When I build models and before I train them, I have the habit of estimating the training time to make sure it is reasonable. This usually comes in the form of my naive estimate,

training hours = floating point ops / 10^9 secs

Where I assume my processor runs at 1 GHz as a reasonable estimate.

This gets kind of annoying for more complicated models, but I can usually guess. Is there a way to get Tensorflow to estimate the training time before running?


texts: What is the best way to optimize the parameters in a Sklearn classifier, when I have little data?

texts:
What is the best way to optimize the parameters in a Sklearn classifier when I only have a data set with 684 rows and 177 columns, and the column I want to predict has 3 labels?

I know I should split my data into training, validation, and test set, and then find the parameters to train the training set that maximizes a metric in the validation set and use this optimized classifier in the test set. However, when I did this using a decision tree classifier, the parameters that worked best for the validation set showed a worse result in the test set than the default parameters. So what is the best way to find the best parameters in this data set? I do not know if it helps, but my data set is very sparse.


texts: How to derive gradients for softmax function

texts:
We have the following feedforward equations:

$z_1 = W_1x + b_1$

$a_1 = f(z_1)$

$z_2 = W_2a_1 + b_2$

$a_2 = y^* = softmax(z_2)$

$L(y, y^*) = -\frac{1}{N}\sum_{n \in N} \sum_{i \in C} y_{n,i} \log{y^*_{n,i}}$

Now, I'm trying to compute the following partial derivates $\frac{dL}{dW_2}$,  $\frac{dL}{db_2}$, $\frac{dL}{dW_1}$, $\frac{dL}{db_1}$. I'm familiar with how to compute these gradients in a simple network, but am running into difficulties with the softmax function here. This is what I have so far:

$\frac{dL}{dW_2} = -\frac{1}{N}\sum_{n \in N}\sum_{i \in C}(\frac{d}{dW_2} y_{n,i}\log{y^*_{n,i}}) = -\frac{1}{N}\sum_{n \in N}\sum_{i \in C}( \frac{y_{n,i}}{y^*_{n,i}}\frac{dy^*_{n,i}}{dW_2})$. Then, $\frac{dy^*_{n,i}}{dW_2} = \frac{dy^*_{n,i}}{dz_2}\frac{dz_2}{dW_2}$ and $\frac{dz_2}{dW_2} = a_1$. I'm just not quite sure how to compute $\frac{dy^*_{n,i}}{dz_2}$. Would anyone be able to help with this or suggest a different way of approaching the computation for $\frac{dL}{dW_2}$?


texts:
Imagine you have two arrays.
arrayA = [1,2,4,1,3,9,8]
arrayB = [4,3,2,4,1,3,9,8,9]

and there's a sequence of numbers inside of them, if you look at arrayA then you can see that it has "2,4,1" and arrayB has "2,4,1" as well (it also has "3, 9, 8"). I need to write a small script in Python that can find the starting position of the sequence and the length of the matching sequence. Does anyone have any idea how I can do that? The result should look like this:
[{start_position: 1, size: 3}, {start_position: 4, size: 3}]


texts: Forecasting monthly visitor count from daily values

texts:
I have a dataset of the daily visitor count of a website. Given this information, I want to forecast what the monthly visitor count will be.
Depending on the visitor count on a day of the month, I would want to update the forecast for the monthly count.
Which method can I use to forecast the monthly number of visitors on a website using the same information on a daily level?


texts: I scraped my Facebook chat history and wanted to try out some basic machine learning stuff with word2vec. However, the data has all sorts of stuff - links, emoji, Cyrillic alphabet, etc. Even if I manage to clear some of those, would it be possible to process the Cyrillic alphabet with word2vec?


texts: The question is simple: is there any advantage in using sigmoid function in a convolutional neural network? Because every website that talks about CNN uses Relu function.


texts: I need help to write an essay about: probabilistic method || Occam's razor || mathematics in the 21st

texts:
I am interested in one of the master's programs in Data Science. In the application process I need to submit an essay of 1,000 words about one of the following topics:

Drawbacks of the probabilistic method
A mathematical theory for Occam's razor
What drives mathematics in the 21st century?

I have a background in numerical analysis and I do not have any idea about how I can tackle these topics. I would like if someone familiar with one of these topics to give me pieces of advice, resources, or how the essay must be structured so can start writing the essay.
I appreciate your help, thank you in advance.


texts:
from keras.models import Sequential
  from keras.layers import LSTM, Dense, Dropout
  from sklearn.metrics import precision_recall_fscore_support, roc_auc_score
  from keras.utils import np_utils
  from keras.callbacks import EarlyStopping
  data_dim = 41
  timesteps = 20
  num_classes = 10

  expected input data shape: (batch_size, timesteps, data_dim)

model = Sequential()

returns a sequence of vectors of dimension 256

model.add(LSTM(256, return_sequences=True, input_shape=(timesteps, data_dim)))
model.add(Dropout(0.2))

return a single vector of dimension 128

model.add(LSTM(128))
model.add(Dropout(0.2))

apply softmax to output

model.add(Dense(num_classes, activation='softmax'))

compile the model for multi-class classification

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

load fold1 for testing

train_x, train_y = load_folds([1,2,3,4,5,6])

load fold2 for validation

valid_x, valid_y = load_folds([9])

load fold3 for testing

test_x, test_y = load_folds([10])

a stopping function to stop training before we excessively overfit to the training set

earlystop = EarlyStopping(monitor='val_loss', patience=0, verbose=1, mode='auto')
model.fit(train_x, train_y, batch_size=128, nb_epoch=10, callbacks=[earlystop], validation_data=(valid_x, valid_y))

this is my code when i run it i got error "ValueError: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (0,)"

can someone help me to understand this and how can i solve it ?


texts:
Before anybody asks please let me confirm this question is related to trading technical analysis.

The problem is quite simple, but I lack the scientific background required to solve it.

Data comes as an array of float values equally spaced temporally. Some local minimums (same apply for maximums) tend to repeat over time (although not at the very same value, but in a rather tight range), because of human psychology: if some price level was important in the past, stopping the price to go below, it will tend to resurface a few times, blocking again the price from going below.

What I'm interested in is to find the most significant levels over a given past period (say 5000 time intervals). Basically the "importance" of a level is determined by the number of times it was tested (meaning the number of times the price tried go past that level and failed); however, a level tested three times at relatively large time intervals (like once each month for a daily chart) is more significant than a level tested three times in a short period (like within a week for a daily chart).

Again, data is as follows:

data comes as an array of float values
data points are equally spaced temporally (however, time is of little to no importance)
some local extremes are more important than others and tend to repeat over time

Can somebody help me with some Python code to extract important levels? Or at least point me to some theory/frameworks to learn? I know about pandas/numpy but I lack the math background required to a least identify what could be used of these...

Thank you!


texts: Why augmenting the training data with binary attributes works better for our dataset?

texts:
We have a dataset with multiple features ~400 where all of the features have a histogram as you can see in the following picture (sampled only a few)

Our assumption

We thought that this looked like some categorical data where few of the times had a special negative number and other times were zero.

So we treated each feature as category of whether the value is zero or not and we created a new column with binary values, 0 or 1. We did the same thing for whether the value is this special negative number or not. We ended up with one more column.

In other words we did the same thing as if we had three categories for each attribute A, B and C and we wanted to encode them with one-hot-encoding. Where here the A is 0, B is our special negative value and C is anything else. Recall that in one hot encoding you throw away one of the three columns because it is redundant.

Experiments to test our assumption

First experiment:
Linear Regression with only original data

Second test:
Linear Regression with the binarized features (2 columns for each feature, we had twice as much features)

Third test:
Linear Regression with the binarized features plus the original (2 columns for each feature plus the original results is three times as much features)

Results

The results was that the first experiment performed worse than the rest which was expected.

If our theory is correct that these can be represented as categories then the second experiment would perform better in terms of RMSE. And indeed the second experiment performs better.

The surprising result was that when combining the initial data with our binarized ones (augmented dataset with three times more features than the original one) the RMSE was even lower!
Why is that?


texts: Ratio of positive to negative sample in data set for best classification

texts: Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the number of positive data sample to get the negative data sample. Will such a data set be useful?


texts: Can one property name be used twice in the same branch of a DecisionTreeRegressor?

texts:
I am using this dataset for the analysis (Generated using make_regression of sklearn library)

I was trying to learn the DecisionTreeRegression algorithm of sklearn library. I used the following code to fit the regressor.

from sklearn.tree import DecisionTreeRegressor as DTR

regressor1 = DTR(max_depth=2)
regressor1.fit(X,y)

y_pred1 = regressor1.predict(X)

These are the leaf node values that I got,

It seems like, the decision tree first did a split on prop 2 at -1.0923644932716892 for the root node then on the right child of the root it again did another split on prop 2 at 0.0340153523120978.

But what I learnt about Decision Trees is that is a split is done at a property then in the same branch the property should not be used again. Then why the sklearn library is doing this thing?


texts: Train a cascaded network with a classifier in between

texts:
I am attempting to do a two-fold task. The input is an image and based on the input I want to pick another image from a set of images (classification task) and then use both the images to obtain an output tensor. Clearly, I can train both the models separately if I know the ground truth of which image I should pick from that set. But, I only have the output tensor ground truth.
The problem, as it appears to me, is that if we employ a classification layer, the gradients will not be differentiable anymore. How do I deal with this problem? Is there literature which uses this kind of architecture for any application?
TIA


texts: I'm studying K-means, and one important drawback of K-means is the lack of robustness to outliers. My question is: are there any cases when the lack of robustness to outliers may be considered not as a defect of K-means but as a virtue instead?


texts: training gradient boosting algorithm in python testing in Golang

texts: What are the best strategy to train and save a gradient boosting algorithm, e.g. LightGBM or XGboost or Catboost in Python but load the model in GoLang and make prediction with Golang ?


texts: What does SpatialDropout1D() do to output of Embedding() in Keras?

texts:
Keras model looks like this

inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = SpatialDropout1D(dropout)(x)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool])
outp = Dense(6, activation="sigmoid")(x)

According to the documentation the output shape = input shape i.e. (samples, timesteps, channels)

Questions

What does SpatialDropout1D() really do to the output of Embedding()? I know the output of LSTM Embedding is of dimension (batch_size, steps, features).
Does SpatialDropout1D() just randomly replace some values of word embedding of each word by 0?
How is SpatialDropout1D() different from Dropout() in Keras?


texts:
I have composed a customized loss function (kl_loss):

def tensor_pValue(pnls,pnl):
    vec=tf.contrib.framework.sort(pnls,axis=-1,direction='ASCENDING')
    rank_p=tf.divide(tf.range(0,264.5,1),264.0)
    return tf.gather(rank_p, tf.searchsorted(vec,pnl,side='left'))

def kl_divergence(p, q):
    epsilon = 0.00001
    p=p+epsilon
    q=q+epsilon
    return tf.reduce_sum(p * tf.log(p/q))

def kl_loss(predicted_pnL,actual_pnl_tensor):
    p_dist=tf.squeeze(tf.map_fn(lambda inp:tensor_pValue(inp[0],inp[1]),(predicted_pnL,actual_pnl_tensor),dtype=tf.float32))
    u_dist=tf.random.uniform([264],0,1,dtype=tf.float32)
    return kl_divergence(p_dist,u_dist)

And then i constructed a simple net work using Keras:

optimizer = tf.train.AdamOptimizer(0.001)
input_dim = X_train.shape[1]
model = keras.Sequential([
keras.layers.Dense(UNITS, activation=tf.nn.relu,
             input_dim=input_dim),
keras.layers.Dense(UNITS, activation=tf.nn.relu),
keras.layers.Dense(264)
])
model.compile(loss=lambda y, f: kl_loss(f,y), optimizer=optimizer)
model.fit(X_train, train_y, epochs=EPOCHS, batch_size=BATCH_SIZE,verbose=0)

And got following errors:

ValueError: No gradients provided for any variable: ["", "", "", "", "", ""].

Can anyone help to take a look on where might be wrong on this? Thank you very much!


texts:
I want to use t-SNE in WEKA just for visualization purposes. I tried to look at the package manager but could not find any implementation of it.

Is there anything that I can do to achieve it?


texts:
I was going through an IEEE research paper which has used Fuzzy ARTMAP for predicting the price of electricity given some highly correlated data.
As per my basic understanding about Fuzzy ARTMAP, it is a classification algorithm, so how will it be able to predict continuous data?
The text from research paper is:

In the architecture of the FA network, the preprocessing stages take
the input vector and contribute to produce complement coding, which
avoids category proliferation, i.e., the creation of a relatively
large number of categories to represent the training data. A sequence
of input vectors (price and demand) and their respective target
vectors are introduced to the FA network in order to classify the
input pattern correctly. The classiﬁed input patterns are then grouped
into labels using membership functions.

I was using MATLAB to implement the same. So is there a library in MATLAB to approach towards the solution?


texts:
Tensorflow: convert PrefetchDataset to BatchDataset
With latest Tensorflow version 2.3.1I am trying to follow basic text classification example at: https://www.tensorflow.org/tutorials/keras/text_classification. Instead of creating dataset from directory as example does, I am using a csv file:
SELECT_COLUMNS = ['SentimentText','Sentiment']
LABEL_COLUMN = 'Sentiment'
LABELS = [0, 1]

def get_dataset(file_path, **kwargs):
    dataset = tf.data.experimental.make_csv_dataset(
      file_path,
      batch_size=3, # Artificially small to make examples easier to show.
      label_name=LABEL_COLUMN,
      na_value="?",
      num_epochs=1,
      ignore_errors=True,
      **kwargs)
    return dataset

all_data = get_dataset(data_path, select_columns=SELECT_COLUMNS)

As a result I get:
type(all_data)
tensorflow.python.data.ops.dataset_ops.PrefetchDataset

Example loads data from directory with:
batch_size = 32
seed = 42

raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    'aclImdb/train',
    batch_size=batch_size,
    validation_split=0.2,
    subset='training',
    seed=seed)

And gets dataset of another type:
type(raw_train_ds)
tensorflow.python.data.ops.dataset_ops.BatchDataset

Now when I try to standardise and vectorise data with functions from example:
def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    stripped_html = tf.strings.regex_replace(lowercase, '<br />', ' ')
    return tf.strings.regex_replace(stripped_html,
                                  '[%s]' % re.escape(string.punctuation),
                                  '')

max_features = 10000
sequence_length = 250

vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode='int',
    output_sequence_length=sequence_length)

And apply them to my dataset I get error:
# Make a text-only dataset (without labels), then call adapt
train_text = all_data.map(lambda x, y: x)
vectorize_layer.adapt(train_text)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-20-1f1fc445912d> in <module>
      1 # Make a text-only dataset (without labels), then call adapt
      2 train_text = all_data.map(lambda x, y: x)
----> 3 vectorize_layer.adapt(train_text)

/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/layers/preprocessing/text_vectorization.py in adapt(self, data, reset_state)
    378       shape = dataset_ops.get_legacy_output_shapes(data)
    379       if not isinstance(shape, tensor_shape.TensorShape):
--> 380         raise ValueError("The dataset passed to 'adapt' must contain a single "
    381                          "tensor value.")
    382       if shape.rank == 0:

ValueError: The dataset passed to 'adapt' must contain a single tensor value.

How to convert PrefetchDataset to BatchDataset ?


texts: searching for vocabulary for machine translation

texts:
Searching for DE-EN vocabulary, that I can use in my machine translation.
It should consist of DE-EN phrases, words, sentences etc.

Got one from here http://www.manythings.org/anki/ , but searching for another one

Any thoughts please?