texts: tf.keras.experimental.export_saved_model error: __init__() missing 1 required positional argument: 'feature_columns

texts:
Objective:

I am trying to save my model in a .pb or .pbtxt format using tf.keras.experimental.export_saved_model(model, saved_model_dir) as described in the following tutorial:

https://github.com/GoogleCloudPlatform/data-science-on-gcp/blob/master/updates/cloudml/flights_model_tf2.ipynb

I get the error:

init() missing 1 required positional argument: 'feature_columns'

Thank you in advance for your help.

The code is below:

training_history = model.fit(train_ds,
                    validation_data=val_ds,
                    epochs=epochs,
                    callbacks=[EarlyStopping(patience=patience), tensorboard_callback ])

now=str(datetime.datetime.now()).replace(':','_').replace(' ','_').replace('.','_')

saved_model_dir="saved_models/"+now

tf.keras.experimental.export_saved_model(model, saved_model_dir)

Output:

TypeError: __init__() missing 1 required positional argument: 'feature_columns'


texts: Wich GPU-card to make deep learning with a MacBook Pro?

texts:
I'm a owner for a MacBook Pro. I want to accelerate my deep learning models, because with the CPU, it's very slow !
Could you tell me which GPU card I can buy for it ?
Thanks.


texts: 3d density of population visualisation on map Python

texts: I am looking for Python module, in which I could create 3d visualization of population density on map. To create 3d plot like this:


texts: What metrics to use in regression if variance in output label is very low?

texts: What metrics/method should one adopt in judging the error when variance is low in output variable. To give you an example : the output variable can be stock prices over a month, the variance generally would be low, (like 100.2, 100.23, 100.45, 101.02 etc.). A model made on this predicting 100 for each instance may give a great score (on a metric like R2 or MAE), but that won't be correct if you look from perspective of the error we wanted to see. How to handle this ? Do we normalize the output variable ? Or do we use a metric that increases the error exponentially on small deviation from true value, if so how ?


texts: How to add more theta parameters into my logistic regression?

texts:
I am a complete beginner in machine learning and coding in python. I have been tasked with coding logistic regression from scratch in comparison with using sklearn. My question is, with my code below I believe I have set the number of thetas I want with:

"X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]"

but I am unsure how to prove this is true with my code below and its definition of theta, if I added more parameters (e.g. all 31 variables of this dataset [https://www.kaggle.com/uciml/breast-cancer-wisconsin-data] which is for classifying tumours) would I just need to add them into this list above? Any help pointing me towards the right direction just to understand this better would be appreciated.

X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)

X = data["diagnosis"].map(lambda x: float(x))

X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]
X = np.array(X)
X = min_max_scaler.fit_transform(X)
Y = data["diagnosis"].map(lambda x: float(x))
Y = np.array(Y)

def Sigmoid(z):
    if z < 0:
        return 1 - 1/(1 + math.exp(z))
    else:
        return 1/(1 + math.exp(-z))
def Hypothesis(theta, x):
    z = 0
    for i in range(len(theta)):
    z += x[i]*theta[i]
return Sigmoid(z)enter preformatted text here

def Cost_Function(X,Y,theta,m):
    sumOfErrors = 0
    for i in range(m):
        xi = X[i]
        hi = Hypothesis(theta,xi)
        error = Y[i] * math.log(hi if  hi >0 else 1)
        if Y[i] == 1:
            error = Y[i] * math.log(hi if  hi >0 else 1)
        elif Y[i] == 0:
            error = (1-Y[i]) * math.log(1-hi  if  1-hi >0 else 1)
        sumOfErrors += error

    const = -1/m
    J = const * sumOfErrors
    print ('cost is: ', J )
    return J

def Cost_Function_Derivative(X,Y,theta,j,m,alpha):
    sumErrors = 0
    for i in range(m):
        xi = X[i]
        xij = xi[j]
        hi = Hypothesis(theta,X[i])
        error = (hi - Y[i])*xij
        sumErrors += error
    m = len(Y)
    constant = float(alpha)/float(m)
    J = constant * sumErrors
    return J

def Gradient_Descent(X,Y,theta,m,alpha):
    new_theta = []
    constant = alpha/m
    for j in range(len(theta)):
        CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)
        new_theta_value = theta[j] - CFDerivative
        new_theta.append(new_theta_value)
    return new_theta

def Logistic_Regression(X,Y,alpha,theta,num_iters):
    m = len(Y)
    for x in range(num_iters):
        new_theta = Gradient_Descent(X,Y,theta,m,alpha)
        theta = new_theta
        if x % 100 == 0:
            Cost_Function(X,Y,theta,m)
            print ('theta: ', theta)
            print ('cost is: ', Cost_Function(X,Y,theta,m))

initial_theta = [0,1]
alpha = 0.01
iterations = 1000
Logistic_Regression(X,Y,alpha,initial_theta,iterations)


texts: Counting the number of trainable parameters in a gradient boosted tree

texts:
I recently ran the gradient boosted tree regressor using scikit-learn via:

GradientBoostingRegressor()

This model depends on the following hyperparameters:

Estimators ($N_1$)
Min Samples Leaf ($N_2$)
Max Depth ($N_3$)
which in-turn determine the number of trainable parameters in this model.  My question is, how can I count the number of parameters (trainable or otherwise randomly assigned) which determined the final model as a function of the above?

My guess is $N_1 \times N_2 \times N_3$ but is this correct?


texts: Does it make sense to build a ROC for a decision tree where there are multiple threshold you can adjust?

texts:
I understand building a ROC curve when the output is a probability, say, from a logistic regression model. You can build a ROC curve by varying the cutoff threshold.
But what about decision trees of the form:
if attribute_1 > x:
  decision = positive
else:
  if attribute_2 < y:
     decision = position
  else:
     decision = negative

You can adjust the cutoff for both attributes and all will affect your confusion matrix. Does it make sense to build a ROC curves when there are multiple thresholds?
Thanks


texts: Is epsilon error a standard known error or custom created by this paper?

texts:
I'm reading this computer vision paper, research paper link, about creating a model to estimate the real age and perceived age of the person in the image (or at least that's what I think it's about). The perceived age is decided by this method: each image is looked at by 10 independent individuals and they estimate the age of the person. The mean and standard deviation are taken from the age guesses of the 10 individuals.
The paper then goes on to use an epsilon error by this equation as part of the model evaluation for perceived age and state the following the evaluation employs fitting a normal distribution with the mean µ and standard deviation σ of the votes for each image. The paper also says that this epsilon error covers the aspect of the uncertainty of the ground truth age.

The results are then posted in a table showing different epsilon error values based on different models.

My questions are the following:

What is that equation and is it standardly used? It says

What is x in this equation? I thought it was some type of feature scaling of the image but that makes no sense because the results are all different in the table and completely out of context anyway. Is x supposed to be the predicted age?


texts:
What kind of Machine Learning/Data Mining packages are available that can easily be scaled on a cluster. For instance H2O is one of them because it's running in Java so it can be easily extended to other machines providing parallelism. What other are popular? or even for general Data munging purposes.

Language wise preferably Python or R. (if other please don't be discouraged to include)


texts: Best tool for analyzing inventory that has binary sales outcome of yes or no

texts:
I am trying to determine a good tool that will help me generate a probability of a sale for a list of of 300,000 products. I have a table of historical sales data (with about 300,000 records) that contains around 10 continuous variables along with a dependent variable that has a yes/no (i.e., binary outcome) value indicating whether product in the list has had a sale in the past 12 months.

The historical data essentially looks like this.

Product1,2,3 etc
Variable 1
Variable 2
Variable 3
Variable 4
Variable 5
Variable 6
Variable 7
Variable 8
Variable 9
Variable 10
Sold in past 12 months (Yes or No)

The last variable in the list is of course the dependent variable.

All I want to do is to find a tool that is going to be the best or easiest to use, so that I can assign a probability to each product in the list, essentially giving me the chance to condense my list to the products that are the highest likelihood to generate a sale, so that I can list those products instead of the others that have lower probability of generating a sale.

Ideally, the tool could do a quick logistic regression, or some other probability calculation based on the available variables, and thereby give me a (RVU-like) number (perhaps a probability ranging from 0 to 1) for each product, allowing me to quickly select the top 50,000 products to list on a website, since they have the higher probability of generating a sale according to the available variables.

I am of course assuming that the variables are somehow correlated to the outcome, but perhaps the tool will help me determine that.

Does anyone have any suggestions of a good tool to accomplish this? I would presume that there is a simple way to set this up in Microsoft Excel, but if not, then a piece of software that does this would of course be great too.
I am also open to suggestions as to which type of regression analysis (or other analysis) is the best to accomplish this.

Thanks for any suggestions.


texts: How to fix IndexError: Too many levels: Index has only 1 level, not 2 in Python

texts:
I have written the following code that has a function model_data to perform a particular set of tasks. I have to pass the list of Badges and the type of category 1 or 2 along with an empty dataframe data.
But while running the code I am getting an error. I searched SO for answers but this type of Question was not found.
CODE
#Model Function

def model_data(badge_list, data):

  for key, value in badge_list.items():

    #Check for Post Type
    if (value == 1):
      badge_type = posts.loc[posts.PostTypeId == '1']

    elif (value == 2):
      badge_type = posts.loc[posts.PostTypeId == '2']

    #Obtain required fields from Badge Data
    badge_type = badge_type[['OwnerUserId', 'Id','Score', 'CreationDate']]
    badge_type.columns = ['UserId', 'Id', 'Score','CreationDate']

    Badge = key

    #Obtain time when user first obtained Badge
    badge_data = user_badge_dt(Badge)

    #Find the number of posts made before and after 1 week of Badge Attainment
    post_data = post_details(df1 = badge_data, df2 = badge_type)
    post_data.date = pd.to_datetime(post_data.date)

    #Calculate APR
    post_data = APR(post_data)

    #Calculate Score
    post_data = score(df = post_data, post_type = badge_type)

    #Generate Final Dataframe with Badge Count
    data1 = badge_number(post_data)
    data1 = data1[['1','2','3','date','Score','APR']]

    #Append Dataframe
    data = data.append(data1)

  return data

#Function Call
questionBadge_list = {'Good Question':1, 'Explainer':2}

data = pd.DataFrame()
badge1_data = model_data(badge_list = questionBadge_list, data = data)

ERROR
IndexError: Too many levels: Index has only 1 level, not 2

ERROR LINE
The code line badge_data = user_badge_dt(Badge) gives this error so I am adding the complete function.
#Function to obtain UserId with the date-time of obtaining given badge for the first time
def user_badge_dt(badge):

  #Creating DataFrame to obtain all UserId and date-Time of given badge
  df = badges[['UserId','Date']].loc[badges.Name == badge]

  #Obtaining the first date-time of badge attainment
  v = df.groupby("UserId", group_keys=False)['Date'].nsmallest(1)
  v.index = v.index.droplevel(1)

  df['date'] = df['UserId'].map(v)
  df.drop(columns='Date',inplace=True)

  #Removing all duplicate values of Users
  df.drop_duplicates(subset='UserId',  inplace=True )

  return df

Badges DataFrame Sample
   Id UserId    Name            Date
0   2   23  Nice Answer     2016-01-12T18:44:49.267
1   3   22  Autobiographer  2016-01-12T18:44:49.267
2   4   21  Curious         2016-01-12T18:44:49.267
3   5   20  Autobiographer  2016-01-12T18:44:49.267
4   6   19  Autobiographer  2016-01-12T18:44:49.267


texts:
I have the following keywords retrieved from a text document.

natural language processing
sequential labeling
programmable
spell checking
techniques
forensics
important issue
categorial grammar
girls
applications

Now I want to remove unwanted words such as programmable, techniques, important issue, girls, applications from this keyword list. Is there a way to automate this? Can I consider some pos-tag pattern to do this?

Please help me!


texts:
I have a dataset with roughly 800 images that are classified in 18 classes.
The classes are spread unevenly, with some classes having 30 images and others having 5.

In order to increase my dataset,I've decided to use image augmentation modifying each image a little,making 20 new images for every image.

I then decided to use my created images as my training set and my original images as my validation set.

Due to unavailability of a GPU,I couldn't train it a lot,but I ended up with around 50% success rate on a training set and 30% on the validation set.

Was the decision of only using my original dataset as validation a good one? If not,why?


texts: Handling a Pandas Data Frame containing multiple non-ordinal categorical features

texts:
I'm currently trying to analyse a dataset containing multiple non-ordinal categorical features and a binary target variable. The table looks something like this:

+------------+---------+------------+--------+
|    Col1    | ....    |    Col14   | Target |
+------------+---------+------------+--------+
| cat 1      |  cat 1  | cat 1      |    0   |
| ...        |  ...    | ...        |  ...   |
| cat 9      |  cat 50 | cat 450    |    1   |
+------------+---------+------------+--------+

The entire table is 400.000 rows x 15 columns, from which the last column is the target variable. Each feature has multiple non-ordinal categories ranging from 9 categories to multiple hundreds of categories.

My first instinct would be to one hot encode all the categorical variables. However, I'm scared that doing so will make any model prone to overfitting.

How could I handle/encode the features variables to analyse their effect on the target variable, using Python?


texts:
I'm fine tunning a gpt-2 model following this tutorial:

https://medium.com/@ngwaifoong92/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f

With is associated GitHub repository:

https://github.com/nshepperd/gpt-2

I have been able to replicate the examples, my issue is that I'm not fiding a parameter to set the number of iterations. Basically the training script shows a sample every 100 iterations and save a model version every 1000 iterations. But I'm not finding a parameter to train it for say, 5000 iterations and then close it.

The script for training is here: https://github.com/nshepperd/gpt-2/blob/finetuning/train.py

I suspect it should be inside this loop, but really don't get to understand the code:

    try:
        while True:
            if counter % args.save_every == 0:
                save()
            if counter % args.sample_every == 0:
                generate_samples()
            if args.val_every > 0 and (counter % args.val_every == 0 or counter == 1):
                validation()

            if args.accumulate_gradients > 1:
                sess.run(opt_reset)
                for _ in range(args.accumulate_gradients):
                    sess.run(
                        opt_compute, feed_dict={context: sample_batch()})
                (v_loss, v_summary) = sess.run((opt_apply, summaries))
            else:
                (_, v_loss, v_summary) = sess.run(
                    (opt_apply, loss, summaries),
                    feed_dict={context: sample_batch()})

            summary_log.add_summary(v_summary, counter)

            avg_loss = (avg_loss[0] * 0.99 + v_loss,
                        avg_loss[1] * 0.99 + 1.0)

            print(
                '[{counter} | {time:2.2f}] loss={loss:2.2f} avg={avg:2.2f}'
                .format(
                    counter=counter,
                    time=time.time() - start_time,
                    loss=v_loss,
                    avg=avg_loss[0] / avg_loss[1]))

            counter += 1


texts: How to download a Jupyter Notebook from GitHub?

texts:
This is a fairly basic question.

I am working on a data science project inside of a Pandas tutorial. I can access my Jupyter notebooks through my Anaconda installation. The only problem is that the tutorial notebooks (exercise files) are on GitHub.

My question: how do I download the exercise files from GitHub and then have them display in the Jupyter notebook section on my computer so that I can use them interactively?

I am very new to Jupyter Notebooks. So far, I have been using PyCharm to execute all of my code.


texts: How does the time taken for backpropagation change with the increase in neurons?

texts:
This is a question I come across a lot in interviews but I don't understand how and what the answer is:-
If we have a neural net, and we double the neurons in it, how does the time taken for backpropagation change? does it increase by

2 times
4 times
8 times
16 times


texts: What will happen if the number of units and the time-steps are different numbers in LSTM?

texts:
How will data feed the LSTM in following scenarios?
I have a data array with the shape of (100,10,3) and a LSTM like one of these:

  First scenario:

  LSTM(units = 5 , input_shape(time-steps = 5, No_of_features = 3))

  Second scenario:

  LSTM(units = 15 , input_shape(time-steps = 5, No_of_features = 3))

  Third scenario:

  LSTM(units = 5 , input_shape(time-steps = 20, No_of_features = 3))

  Fourth scenario:

  LSTM(units = 15 , input_shape(time-steps = 20, No_of_features = 3))


texts: Is datawarehouse considered as datalake in big data environnment?

texts: Suppose I have a datawarehouse (DWH) and now I would like to add many other bigdata sources of information most of them are not structured. I still keep the DWH with no architectural change. The only thing I do is to enrich the bigdata with the data resides in the DWH using connectivity. Can the DWH defined as datalake in this case?


texts: Adding multilabel classifier to TensorFlow example

texts:
I am starting with the generic TensorFlow example.

To classify my data I need to use multiple labels (ideally multiple softmax classifiers) on the final layer, because my data carries multiple independent labels (sum of probabilities is not 1).

Specifically in the retrain.py these lines in add_final_training_ops() add the final tensor

final_tensor = tf.nn.softmax(logits, name=final_tensor_name)

and here

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
      logits, ground_truth_input)

Is there already a generic classifier in TensorFlow? If not, how to achieve multilevel classification?

add_final_training_ops() from tensorflow/examples/image_retraining/retrain.py:

def add_final_training_ops(class_count, final_tensor_name, bottleneck_tensor):

  with tf.name_scope('input'):
    bottleneck_input = tf.placeholder_with_default(
        bottleneck_tensor, shape=[None, BOTTLENECK_TENSOR_SIZE],
        name='BottleneckInputPlaceholder')

    ground_truth_input = tf.placeholder(tf.float32,
                                        [None, class_count],
                                        name='GroundTruthInput')

  layer_name = 'final_training_ops'
  with tf.name_scope(layer_name):
    with tf.name_scope('weights'):
      layer_weights = tf.Variable(tf.truncated_normal([BOTTLENECK_TENSOR_SIZE, class_count], stddev=0.001), name='final_weights')
      variable_summaries(layer_weights)
    with tf.name_scope('biases'):
      layer_biases = tf.Variable(tf.zeros([class_count]), name='final_biases')
      variable_summaries(layer_biases)
    with tf.name_scope('Wx_plus_b'):
      logits = tf.matmul(bottleneck_input, layer_weights) + layer_biases
      tf.summary.histogram('pre_activations', logits)

  final_tensor = tf.nn.softmax(logits, name=final_tensor_name)
  tf.summary.histogram('activations', final_tensor)

  with tf.name_scope('cross_entropy'):
    cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
      logits, ground_truth_input)
    with tf.name_scope('total'):
      cross_entropy_mean = tf.reduce_mean(cross_entropy)
  tf.summary.scalar('cross_entropy', cross_entropy_mean)

  with tf.name_scope('train'):
    train_step = tf.train.GradientDescentOptimizer(FLAGS.learning_rate).minimize(
        cross_entropy_mean)

  return (train_step, cross_entropy_mean, bottleneck_input, ground_truth_input,
          final_tensor)

Even after adding the sigmoid classifier and retraining, Tensorboard still shows softmax:


texts: One hot encoding with too many features (~ 10,000)

texts:
I am building a model to predict time off and sick leave for a specific employee.

Each of the employees has one row per day from 01/01/2013 to 31/12/2018 in the dataset flagged with 0 or 1 (if that day was taken as a time off or sick day). I am using information like location, age, job position, etc.

I was thinking to use employee id as a feature to train the model. Some users have obvious patterns, e.g, taking Monday off for some consecutive weeks. On the other hand, I feel like I do not have enough information as to have one model per employee, specially if they have being hired recently.

After researching a bit, seems that the only encoding that would make sense for employee id would be 'One Hot Encoding'. However, this would generate up to 10,000 features which doesn't seem very optimal.

Do you have any ideas that can help me build this model in a better way?

Thanks in advance


texts:
I have a dataset of 1000 labels. I would like to build a custom model from this dataset with three labels, say Dog, Cat, and Others. Obviously, Others will be large in number for this approach.  I understand that this could be an issue since the classifier has seen more Others examples.

Pipeline

Large dataset --> Seperate to Dogs, Cats and others --> Classifier --> Predicts Dogs, Cats or Others.

What are the other approaches which can be taken in this situation?


texts: Does Orange scale the data automatically for the linear regression with Ridge regularization

texts:
I'm using the linear regression tool with the Ridge regularization. To use the Ridge regularization, I have to scale the data first. Does Orange scale the data automatically? I can't find any information about this mentioned in Orange's documentation for Ridge regularization.

In python's scikit-learn, I have to scale the data manually before using Ridge Regression. In MATLAB, the scaling in the Ridge function included. So, do I have to scale the data manually before I'm using the Ridge Regression in orange?

Thanks for your help.


texts: Will I be qualified for data science roles with a PhD in Engineering and Machine Learning?

texts:
Currently doing a funded PhD in Civil Engineering using Machine learning applications. In essence data driven modelling of a niche area of engineering (to do with some material deformation predictions etc.)

I have a years experience working as a Civil Engineer on site, and have a first class MEng in Civil & Structural Engineering.

I think I would really enjoy data science, and I am wondering if my PhD will provide a good stepping stone for this? And if so, what are some other things I can do to really make myself stand out as a data scientist?


texts:
is the best way to create the most accurate classifier to train a bunch of classifying algorithms like ANN, SVM, KNN, etc, and test different parameters to get optimal parameters for each classifier, and see which classifier has the least testing error?

Or is it better to use ensemble method and choose the "majority" decision of different kinds of trained classifiers?


texts: Is there a way in pandas to import NA fields as a string rather than NaN?

texts:
I'm doing a Kaggle challenge, and a lot of entries in the data are NA. However, according to the data description, this doesn't actually mean "missing data", it means something like "Not applicable", in the sense of it just not having that quality (for example, not having a basement).

However, when I use pandas to import the data using read_csv(), and then use head() to look at it, it shows NaN for all those things that should be NA (comparing with the spreadsheet in LibreOffice).

I know how to just replace one value with another for a given column, but there's still a problem. NaN is usually what it should enter if there's just a missing value. So, if it's importing both "NA" and a blank cell as NaN, I won't know which it is.

It's probably not a huge issue, because I'll probably want to replace actual missing values with something common anyway, but it's good to know. I know I could also go into the csv with regular python and rename those actual "NA"s, but that's cumbersome. Is there a way I can just import certain columns as strings?


texts: Difference between paragraph2vec and doc2vec

texts: Is paragraph2vec the same as Doc2vec or is every approach different?


texts: how to implement a hierarchical clustering technique using parallel execution in R

texts:
In R, currently to implement wards method for hierarchical clustering, I use the following code - results <- hclust(data, "ward.D2").
However as my data size has increased exponentially, I wanted to implement parallel execution.

Since I found no package, I tried to re-write the hclust method, however most of its code is written in fortan.

Does there exist any way to implement parallel execution here?


texts: How to calculate a weighted Hierarchical clustering in Orange

texts:
I am doing my first cluster analysis with Orange (which I recently discovered and looks promising for this iterative and interactive process).

Apparently, there are several methods of creating clusters based on distance algorithm:

Single linkage (which computes the distance between the closest elements of the two clusters)
Average linkage (which computes the average distance between elements of the two clusters)
Complete linkage (which computes the distance between the clusters’ most distant elements)
Weighted linkage
Ward

Since I have several columns, and some of them are more important than others in terms of defining clusters, it seems to me that using weighted linkage method may be what I am looking for. Unfortunately, I don't know how to do that, since I didn't find a way to assign a weight to each column.

To make things worse, I have only found an explanation about the first three on this Orange's blog post but nothing about Weighted linkage  (nor Ward, which may be a recent addition, since it is not even mentioned on widget's help).

Am I on the right path to achieve what I am looking for? Is there any way to make some columns more or less important/definitory when calculating the distances?


texts:
In the loss function between generated image and content image, we calculate the error taking the only activation of the corresponding channel but for calculation of loss function between style and generated image we calculate the gram matrix.
But we should do the same for the loss function between content and generated image to find the correlation between different channels.

So why we do not use gram matrix in loss function calculation between content and generated?
Computational efficiency is not an option :)


texts: Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

texts:
I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or negative (bearish). While the model is accurate when plugging in my own test data (F1 Score ~86%), the classification itself is not accurate. Tweets that are undoubtedly positive/bullish, and not classified as so. Perhaps this is because the language in the investment world is different than a movie review - which uses universally recognized positive/negative words and/or sentences.

The same is true when I take my tweet dataset and use Vader SentimentIntensityAnalyser to parse pos/neg tweets into separate folders.

So my question is... since the language that is used for telling whether a stock is bullish/bearish is uniquely different from that of an Amazon review, or movie review, would it be optimal for me to manually classify my dataset into positive (bullish) and negative (bearish) datasets?


texts: Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than dropout?


texts:
I am trying to implement demo of Image Captioning system from Keras documentation. From the documentation I could understand training part.

max_caption_len = 16
vocab_size = 10000

# first, let's define an image model that
# will encode pictures into 128-dimensional vectors.
# it should be initialized with pre-trained weights.
image_model = VGG-16 CNN definition
image_model.load_weights('weight_file.h5')

# next, let's define a RNN model that encodes sequences of words
# into sequences of 128-dimensional word vectors.
language_model = Sequential()
language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
language_model.add(GRU(output_dim=128, return_sequences=True))
language_model.add(TimeDistributedDense(128))

# let's repeat the image vector to turn it into a sequence.
image_model.add(RepeatVector(max_caption_len))

# the output of both models will be tensors of shape (samples, max_caption_len, 128).
# let's concatenate these 2 vector sequences.
model = Merge([image_model, language_model], mode='concat', concat_axis=-1)
# let's encode this vector sequence into a single vector
model.add(GRU(256, 256, return_sequences=False))
# which will be used to compute a probability
# distribution over what the next word in the caption should be!
model.add(Dense(vocab_size))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)

But now I am confused in how to generate caption for test image. Input here is [image, partial_caption] pair, now for test image how to input partial caption?


texts: How do you set sigma for the Gaussian similarity kernel?

texts:
Let's say we have $n$ two-dimensional vectors: $$\mathbf{x}_1,\dots,\mathbf{x}_i,\dots,\mathbf{x}_n=(x_{1_1},x_{1_2})^T,\dots,(x_{i_1},x_{i_2})^T,\dots,(x_{n_1},x_{n_2})^T$$ How do you set $\sigma$ for the Gaussian similarity kernel:
$$s(\mathbf{x}_i,\mathbf{x}_j)=\exp\left(-\frac{||\mathbf{x}_i-\mathbf{x}_j||^2}{2\sigma^2}\right)$$


texts: How to calculate precision and recall in an unsupervised ranking method in order to be compared with a supervised one?

texts:
I am working on an ML supervised learning project for Key-phrase extraction. The evaluation dataset (different from training-validation datasets) contains around 200 phrases labeled as key or not key.

The problem is that I want to compare the results obtained (in terms of precision, recall and f1) via different classifier's algorithms with existing unsupervised methods. These methods are based on ranking and extract the top-k key phrases.

Using the dataset that I mention, what is the proper way to calculate precision and recall for these unsupervised ranking methods?


texts: how to use multiple generator in keras fit_generator()

texts:
I want to train the multi-input model on a set of images. I use ImageDataGenerator.flow_from_directory() and fit_generator in keras.

The problem is I don't know how to use multiple generators.

This is my inputs & output generator and seeds are same:

input1 = imageDataGenerator.flow_from_directory(directory=base_data_directory + 'img/' + mode+'/', **img_generator_config)
input2 = imageDataGenerator.flow_from_directory(directory=base_data_directory + 'edge/' + mode,**edge_generator_config)
output = imageDataGenerator.flow_from_directory(directory=base_data_directory + 'label/' + mode, **label_generator_config)

How should I use fit_generator with this data?


texts: ML technique to predict next year output based on text quantities

texts:
I have a random data that I would like to predict how much a quantity will be in 2020.
The data looks like this:

 year        components           total_components
 2019        [Pen, Pencil, Books       4
             Paper]
 2018        [Pen, Pencil, Books       5
             Paper, Eraser]
 2017        [Pen, Pencil, Books       6
             Paper, Eraser, Napkin]
 2016        [Pen, Pencil, Books       7
             Paper, Eraser, Napkin,
             laptop]

In my head, I thought of time series forecasting or RNNs but the data is a bit strange to apply the techniques.

Which ML technique would you suggest here? Thanks :-)


texts: What does the pair of vertical lines in empirical entropy formula mean?

texts:
I am learning from this post.

$\alpha$ is the ratio of the first subset,

$$\alpha=\frac{\left|D_{1}\right|}{\left|D\right|}$$

according to the context and code of the post, $\left|D\right|$ means the number of samples?

What are the pair of vertical lines called?

Is it the L1 norm symbol? It does not seem to be the absolute value symbol.


texts:
What metrics can be used for evaluating text clustering models? I used tf-idf + k-means, tf-idf + hierarchical clustering, doc2vec + k-means (metric is cosine similarity), doc2vec + hierarchical clustering (metric is cosine similarity).
How to decide which model is the best?


texts: BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

texts:
For example I have an original sentence. The word barking corresponds to the word that is missing.

Original Sentence : The dog is barking.
Incomplete Sentence : The dog is ___________.

For example, using the BERT model, it predicts the word crying instead of
the word barking. How will I measure the accuracy of the BERT Model in terms of how syntactically correct and semantically coherent the predicted word is?

(For an instance, there are a lot of incomplete sentences, and the task is to evaluate BERT accuracy based on these incomplete sentences.)

In other words, how will I measure the distance in terms of semantics in terms of model between the two words barking and crying.

Please help.


texts: Strange Pearson Correlation Coefficient Given DataFrame

texts:
I have a Python DataFrame that looks something like:

pageviews, type
100         0
50          1
...

When I run:

df.corr()

            pageviews   type
pageviews   1.000000    -0.009611
type       -0.009611    1.000000

So there's less than a 0.01 correlation between type and pageviews.  However, when I do:

df.groupby('type')['pageviews'].mean()

I get back:

type
0    1421.406621
1     885.092874
Name: pageviews, dtype: float64

So type 0 has almost twice the number of average pageviews but there's very little correlation.

How is this possible?

Thanks!


texts: How to forecast time series analysis for more then one dependent variables?

texts:
I have three datasets:

Dataset_1= column name as ID, Date, counter_id

Dataset_2= column name as ID, no. of tickets sold

Dataset_3= column names as ID, country_code.

Here, country_code ranges from 1 to 30 as codes for different countries and counter_id as counter codes ranging from 1 to 100. I merged the datasets based on common column ID.

I have to estimate how many tickets will be sold during the next 20 days in a given country and through a specific counter.

What I have done is predict the number of tickets sold for next 10 days using time series forecasting. But, I'm not able to decide how I can predict the counter_id and country_code along with tickets sold? Please give me some useful feedback. I am a beginner in data science. Thanks!


texts: Linear regression model with (categorical) predictor variables

texts:
I used LM model with (categorical) predictor variables on my data in r like this (I have count variable as dependent/target variable):

# Fit linear model to total viewing:
fit <- lm(
  total_viewing_minutes ~ panelist + sex + internet + education,
  data = cln
)

If I understood correctly in r the first category is always the reference one so has by definition a coefficient of 0.

So by looking at the model result, is 68.3 minutes (intercept)  average time viewing of

Panelist0 + sex1 + internet1 + education1

and by taking into account the term Panelist1
the 68.4 + 17.6 is avg time viewing of

 Panelist1 + sex1 + internet1 + education1

is that correct interpretation?


texts: What is the 1 Unit in the contraint of SVM: $y_i(wx_i+b) \geq1$

texts:
I am following this note on SVM.

The constraint, $y_i(wx_i+b) \geq 1$, basically said all inputs, $x_i$, lie at least 1 unit away from the hyperplane on the correct side.

What does it mean by 1 unit? In 2D space, does it mean moving the hyperplane, $w$, up by 1?


texts: Understanding Layers in Recurrent Neural Networks for NLP

texts:
In convolution neural networks, we have a concept that inner layers learn fine features like lines and edges, while outer layers learn more complex shapes.

Do we have any such understanding for layers in RNNs (like LSTMs), something like inner layers understand grammar while outer layers understand more complete meanings of sentences assuming that we are using the LSTM for some natural language task like text summarization?


texts:
I have a dataset with 4700 records and it's a classification problem. Proportion of classes is 33 and 67%

few questions

1) does this proportion qualify dataset as imbalanced ?

2) should I do cross validation and then apply (over/under or SMOTE sampling)  or I should first balance my sample through these sampling techniques and then do cross validation?

3) Why is propensity score matching used only in healthcare related studies and not much in other applications?

4) How is Propensity score matching different from other ML classification algorithms?


texts:
I have a collection of articles and for each articles I have the number of clicks and views, so I can measure the Click-Through Rate (CTR) as the number of clicks over the number of views.

However, the CTR is not a very informative quantity because, for instance, two pages with CTR 10% can have very different values for the clicks and the views. For example, page A can have 10 views and 1 clicks and page B 10 million views and 1 million clicks.

So I was wondering if there was an algorithm that can rank pages based on their CTR that also account for the absolute value of clicks and views because in the previous example page B may have a better ranking than A.


texts: How to transform an imbalanced attribute to make it more suitable for linear regression?

texts:
I'm new to data science but trying to get better

Here I have an attribute and plotting its histogram

From what I know so far such a distribution is imbalanced and my goal is to equalize things a little bit right?
Again from what I know, I have to transform this attribute to be more suitable for linear regression?

Is it obvious (to someone more experienced than me) which kind of transformation is applicable in this case?

Note that this is an attribute and not my target, this is not what I am trying to predict. This is one of the attributes to be used for predicting


texts: Why are random forest permutation importances almost all zero?

texts:
I have built a random forest binary classification model with 18 input variables. Gini importance (MDI) shows that all variables have positive importance, although some are very small. According to permutation importance, all but 3 variables have zero importance. Does that result make sense?
When I set the maximum depth of the forest to 3 or 4, more variables become important for permutation importance.
What should I do to get the "real" importances of the variables?


texts: Spacy dependencymatcher pattern not returning matches

texts:
I am trying to create, add and get results from a pattern using spacy DependencyMatcher.
I created a pattern for the sentence: "From Monday to Friday"
The full pattern:
pattern = [
    {
        "RIGHT_ID": "node0",
        "RIGHT_ATTRS": {'DEP': 'ROOT', 'POS': 'ADP', 'TAG': 'IN'}
    },
    {
        "LEFT_ID": "node0",
        "REL_OP": ">",
        "RIGHT_ID": "node1",
        "RIGHT_ATTRS": {'DEP': 'pobj', 'POS': 'PROPN', 'TAG': 'NNP'},
    },
    {
        "LEFT_ID": "node1",
        "REL_OP": "$--",
        "RIGHT_ID": "node2",
        "RIGHT_ATTRS": {'DEP': 'prep', 'POS': 'ADP', 'TAG': 'IN'},
    },
       {
        "LEFT_ID": "node2",
        "REL_OP": ">",
        "RIGHT_ID": "node3",
        "RIGHT_ATTRS":{'DEP': 'pobj', 'POS': 'PROPN', 'TAG': 'NNP'},
    },

]

The simpler pattern is :
pattern = [
    {
        "RIGHT_ID": "node0",
        "RIGHT_ATTRS": {"POS": "ADP"}
    },
    {
        "LEFT_ID": "node0",
        "REL_OP": ">",
        "RIGHT_ID": "node1",
        "RIGHT_ATTRS": {"POS": "PROPN"},
    },
    {
        "LEFT_ID": "node1",
        "REL_OP": "$--",
        "RIGHT_ID": "node2",
        "RIGHT_ATTRS": {"POS": "ADP"},
    },
       {
        "LEFT_ID": "node2",
        "REL_OP": ">",
        "RIGHT_ID": "node3",
        "RIGHT_ATTRS":{'POS': 'PROPN'},
    },

]

My question is, why is this pattern not giving any matches, not on the full or simpler pattern?
import spacy
from spacy.matcher import DependencyMatcher

nlp = spacy.load("en_core_web_sm")
matcher = DependencyMatcher(nlp.vocab)

text="From monday to friday"
doc = nlp(text)
matcher.add("pattern1", [pattern])

matches = matcher(doc)

# Each token_id corresponds to one pattern dict
match_id, token_ids = matches[0]

spacy versions:
spaCy v3.0.6
NAME             SPACY            VERSION
en_core_web_sm   >=3.0.0,<3.1.0   3.0.0   ✔


texts: Fitting probabilities in scikit-learn RandomForestClassifier

texts:
This is a newbie questions, so please bear with me. Given this random forest model:
from sklearn.ensemble import RandomForestClassifier

X = [ [2,1,1,1], [2,0,2,1], [3,1,1,1] ]
y = [ 0, 1, 2 ]

regr = RandomForestClassifier(n_estimators=200, max_depth=5)
regr.fit(X, y)

X_test = [ [3, 1, 1, 1] ]
y_result = regr.predict_proba(X_test)
print('y_result:' , y_result )

The result is:
y_result: [[0.26 0.05 0.69]]

I understand that these are the probabilities of the first, second and third value, or 0 = 26%, 1 = 5% and 2 = 69%.
Question: if the test set is [3, 1, 1, 1] and it fits to the value 2, why do I get 69% probability of 2 instead of 100%?


texts: Using transformers for information extraction

texts:
Task
I am trying to do some information extraction on earnings reports. I am trying to extract certain metrics, e.g. net sales for quarters. The earnings reports differ quite a lot in how they are structured but they use similar language. A "simple" example follows

Nine-month period in figures
Order bookings increased 6.6% to SEK 1,095.5 million (1,027.9).
Net sales rose 22.8% to SEK 1,153.5 million (939.4). Adjusted for currency fluctuations, sales increased 19.6%.
Operating profit rose 31.3% to SEK 171.8 million (130.8), corresponding to an operating margin of 14.9% (13.9). Adjusted for currency fluctuations, operating profit increased 24.5%.
Profit before tax amounted to SEK 176.3 million (137.9).
This result includes nonrecurring items (refer to page 10 of the attached interim report). These items had a positive net effect of SEK 1.4 million on operating profit.
Cash flow after changes in working capital amounted to SEK 215.2 million (180.4).
Third quarter in figures
Order bookings increased 10.3% to SEK 431.1 million (390.7). Of the order bookings during the quarter, 25% were recognized during the third quarter and 32% to 42% pertain to revenue within 12 months after the end of the quarter.
Net sales increased 38.8% to SEK 457.4 million, while it was SEK 329.5 million in the previous year. Adjusted for currency fluctuations, sales increased 34.9%.
Operating profit rose 98.6% to SEK 99.7 million (50.2), corresponding to an operating margin of 21.8% (15.2). Adjusted for currency fluctuations, operating profit increased 87.6%.
Profit before tax amounted to SEK 100.7 million (51.4).
This result includes nonrecurring items (refer to page 10 of the attached interim report). These items had a positive net effect of SEK 1.4 million on operating profit.
Cash flow after changes in working capital amounted to SEK 134.1 million (80.0).

where I have highlighted the metric I want to extract.
Note that Third quarter in figures is important, i.e. information from previous phrases are needed.
Initial approach
My first strategy is to extract all candidate spans (SEK 1,095.5 million, SEK 1,153.5 million etc.) and then classify each one of them (EXTRACT or DO_NOT_EXTRACT), but I haven't got any great results with that approach. I was thinking that maybe I could exploit transformer models like BERT etc. to capture context from previous phrases?
I hope someone have a good idea on how to tackle this task.


texts:
I am studying for an exam and encountered this problem from past worksheets:

This is the data frame called 'contest' with granularity as each submission of question from each contestant in the math contest. The question is

and the answer is in red. I get why that works, but why is the 4th choice wrong? I really can't figure it out - please help.
*please let me know if this is not allowed as a post in this community.
for full description of the problem:

In this question, we will be looking at the contest dataframe which contains data from a
math contest in 2019. In the contest, each participant had a total of five questions. The participants submit each question separately and each row of the DataFrame records a particular submission of one of the contestants by some participant. The Timestamp column specifies the
time a given problem is submitted by a participant; each timestamp is discretized to the minute
and has been properly converted to a Pandas datetime object with pd.to datetime.
The Contestant column contains the id-name pair of each participant. The Question
column contains the question that was submitted. The Correct column tells us if the answer
given in the submission is correct (1) or not (0). Assume each participant can have several
submissions for the same problem, but they can only submit one question per minute.


texts: Classification targets with heterogenous meanings

texts:
I am training a classification model on a dataset of users on a website and each has 100 different measurements of their behaviour on the platform.

Most of these users are dormant but about 10% will reawken. We are interested in purchasing activity once awake.

We have separated the customers into two classes:

Those who reawaken and purchase products 5+ times (positive class)
Those who remain dormant and those who rewaken but purchase less than (or including) 4 times.

We also have a separate model that sorts customers into groups

Will reawaken.
Remain dormant

My question is:

If we output the probability of being in the positive class for both of these classifiers, what will the product of these two probabilities mean semantically?

Will it be P(5+ and reawaken), or will it be something else?


texts: Cosine similarity with arrays contaning NaN

texts:
I am trying to calculate a cosine similarity using Python in order to find similar users basing on ratings they have given to movies. As it can be expected there are a lot of NaN values.
I am using movie dataset from Kaggle.

When I use np.dot() on two nd.arrays the outcome is:'nan'.
I have checked with np.nansum() that there are some other than 'nan' values.

I do not want to change all 'nan' values to '0' as it would mean that users have given 0 rating to the movies which lead to 'false' similarity between users.

Please, give me some advice regarding how to proceed with this problem.

Thanks in advance.


texts: What is the BLEU score used in Google Brain's "Attention Is All You Need" paper?

texts:
Google Brain's Attention Is All You Need paper on sequence-to-sequence translation reports:

  Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German
  translation task, improving over the existing best results, including
  ensembles, by over 2 BLEU

but the Wikipedia entry on BLEU says:

  BLEU’s output is always a number between 0 and 1

What definition of BLEU is the Google Brain paper using? I could not find a separate definition in the paper itself.


texts: What is a suitable Tensorflow model to classify images into foggy/not foggy?

texts:
I want to classify photos taken by multiple webcams that are operating in mountainous regions into foggy / not foggy. The photos are in various sizes and were taken under very different light conditions and in different areas.

I read about Tensorflow and its ready-to-use image recognition models (which of course would have to be re-trained for the foggy/non-foggy categories).

However, these models are trained to classify images into categories according to objects within these images. As I want to classify my images based on their overall appearance (blurry, greyish, far away objects hardly detectable, ...) I was wondering if these models are really suitable or if there is a better approach for this task. Any help is highly appreciated!


texts: Deriving answers to specific queries from a text

texts:
Introduction

I am looking to extract out sentence(s) from a news article for questions like 'who', 'when', 'what', 'why' and 'how'. Now I did some research and found bert model which can be utilised to make query based summarizer. But it was not satisfactory as it sentences extracted were small and wrong by huge margins. It makes sense as it was designed to answer full questions and not something like just 'when'.

Spacy

I knew about spacy and from that I was able to guess answers for 'who' and 'when'. Also 'what' isn't difficult as it can be obtained from description or first few sentences.

Problem

Now I have no idea for 'why' and 'how'. Only thing I can think of is finding words that are used to describe when we give reason to something (why) and  words that describe a  process(how). But again many things become subjective here as it depends on the writer. What is best way possible right now for achieving this, or anything closest to it?

Examples

https://drive.google.com/file/d/1ejHc3sFZNYKPnmQZhX34R0lY93NAj6Vh/view?usp=sharing


texts: What are practical benefits of the various distance metrics in scipy?

texts:
I’m looking for a information about distance metrics.
Python’s scipy gives many different metrics, and I’d like to know more about their practical use.
In particular, I’m trying to understand the differences & benefits of Correlation vs Euclidean
Is there an article describing all of them somewhere?
https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html


texts: Should we use discrete or continuous input for decision trees

texts:
I have 2 datasets, a continuous dataset(75 datapoints and 14 variables) and a discretized dataset which was made by placing the continuous datasets into buckets. I have built a decision tree classifier (using the python sklearn package) and the classifier works much better for the discrete dataset rather than the continuous dataset.

I have also read in a few papers that sometimes it is preferable to use discrete datasets. But I don't know why. I Would appreciate any input or an explanation.


texts: The Israeli MOH databases on Covid patient numbers have a lower cutoff value on published numbers of patients. How does this protect privacy?

texts:
The Israeli ministry of health reports many statistics on Covid patients (Numbers of confirmed cases, hospitalizations, people in quarantine - Divided by age and other demographics) available here (Albeit only in Hebrew)
In any category, numbers below 15 are reported as <15 (see e.g. this table, with statistics on medical personnel cases). Someone told me this was for privacy protection reasons, but I don't understand why.
Does this make sense? How is privacy compromised if exact low numbers are published?


texts:
In Keras I have to predict 47 columns and my query is below:

from keras.models import Sequential
from keras.layers import Dense, Activation

model_1 = Sequential([ Dense(32, input_shape=(192,)), Activation('relu'), Dense(47), Activation('softmax'), ])

model_1.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

model_1.fit(standardized_X_train, train_y, batch_size=32, epochs=15, verbose=1, validation_data=(standardized_X_test, test_y))

classes = model.predict(x_test, batch_size=128)

But, when I try to predict all my columns are showing NaN as predictions. Any reason as to why this is happening?


texts: What does attention weights output from Transformer network do?

texts: I'm trying to understand transformer networks. I want to know that are the attention weights, which are the outputs from forward/predict method where we get final output and attention weights as return, important after we done predicting?


texts: Handling unbalanced datasets with XG boosting

texts: Suppose you want to model (predict) a rare disease, and you use the parameter "pos scale weight" as a hyperparameter in XG boost . For example I have 20 times more positive cases, can I then use pos scale weight = 0.05, even though for example the ratio in het the real world would be not 1/20 but 1/2000?


texts: Predicting class of email based on its associated parameters

texts:
I am solving a challenge where the data represented as above is given. In this challenge I have to associate each email with one of the the 2 criteria (automation, Or SEO). Example if email falls under SEO then I can reach out that person regarding SEO related queries.

I have these parameters associated with each email:

Position of the person
Organisation name
Company domain

My task is to associate each email with one of the the 2 criteria (automation, Or SEO)

Some entries for Position (job title) are missing from my dataset. But I always have Organisation name and Company Domain and associated email.

To solve this I referred to a well known problem of predicting tags of a stack overflow post: https://medium.com/datadriveninvestor/predicting-tags-for-the-questions-in-stack-overflow-29438367261e

But in my case is this approach suitable? How should I approach this problem with a better method?


texts: Categorize text as Body, Heading in a loosely formatted document

texts:
Given a semi-structured document with only texts and images, and some style properties present on the text, What is
  the best possible way to classify the text present in the document to
  any formal HTML standard class such as Heading, Body, Caption, List
  etc.

PS: Please forgive me if you think the question is highly abstract, but the task I am attempting can be broadly categorized in this way only. I need to extract structure from a loosely structured document. Any suggestions to this incorporating ML/Heuristics or links to any such existing problems/existing solutions will be deeply appreciated as well.


texts: Find correlation between grades from two raters

texts:
The question is whether we can find a correlation between two sets of grades (categorical data).
Let’s say we have a dog competition and there are 1000 dogs participating.
There are two rounds of assessment
first round
dog owners give their assessment on the scale from A to C. Where A is excellent and C is bad. There are four criteria for assessment during both tours (behaviour etc).
second round
one judge gives his assessment of one dog based on the same assessment criteria as in round 1. however, grades vary from M - meeting expectation, E - exceeding expectation, B - Bellow expectation.
We understand that M is B, E is A and B is C.
After two rounds our table would look like:

| dog             | round one | round two |
| --------------- | --------- | --------- |
| Dog1_criteria1  | A         | B         |
| Dog1_criteria2  | A         | E         |
| Dog1_criteria3  | A         | E         |
| Dog1_criteria4  | B         | M         |
| Dog2_criteria1  | A         | E         |
| Dog2_criteria2  | B         | M         |
| Dog2_criteria3  | A         | E         |
| Dog2_criteria4  | C         | B         |
....

How do we find a correlation between the two sets of answers? Thank you!


texts: What is the efficient way to generate a similarity score when comparing two face images?

texts: I am working on a face recognition application using deep learning. To plot the ROC curves and do performance evaluation, I extracted the features from the last layer of the deep neural network and I have tried Euclidian and Manhattan distances between high dimensional feature representation, but this won't give me accurate measure in terms of comparison as faces have intra-personal and interpersonal differences and other challenges such as pose, lighting condition, and expression. I came across techniques such as joint bayesian, sparse projection, KL-divergence based regularizer/prior(EM algorithm) etc. Are there any other approaches that I'm missing here? What would be the efficient approach to generate similarity scores?


texts: How does my score ranges change when I use class weight in Keras vs when I don't?

texts: Following are images of Unweighted Score Range and Weighted Score Range


texts:
I work on NLP binary classification task (but actually the question can be applied to any ML task using augmentation) and use Augmentation technique for creating additional data.
I already have trained model and new small dataset (model was trained on another one!).
So I create additional dataset using augmentation.
I have several functions/libraries/approaches to augment data.
My question is how to understand are new data (augmented) good or not at stage of creating new data (i.e. without retraining the model)?
At the current moment I have next idea:
Get from my small dataset examples where model falls (False negative/false positive) ->
Augment data on them -> Feed to model -> Look at the scores.
If model still fails on them - data is good enough.
But at all I'm not sure if this is a reliable approach.
Are there are some acknowledged metrics for that?
Or maybe someone can suggest ways to do this correctly?


texts: Suggestion of a model for these type of data?

texts:
I've got a data set that looks like this

physical_data1   physical_data2  switch1   switch2   state
400              500             1         0         Normal
400              500             1         1         Normal
500              650             0         0         Normal
600              700             1         0         Normal
1000             300             1         1         Anomaly!

where physical_data are data that are from 0 to 1000 and switch are switches that are binary (1 means on 0 means off).

I am still relatively new to machine-learning and so I am wondering what kind of machine-learning algorithm is best suited for these kinds of data to detect anomalies, where my data has mixed features of physical and binary quantities.

Things that i have done are, normalizing to [0,1], however i am not quite sure if applying PCA to this kind of data will result in a loss of detection rate because it requires all features to determine if a particular reading is normal or an anomaly.

One other question I have is that what if in my data set i do not have the Anomaly data, instead, only have normal data. In that case what kind of model can i use?


texts: How do I convert image bounding boxes to SSD multi-resolution bounding box labels for training?

texts:
In SSD bounding boxes are predicted for feature maps at different resolutions. I have trouble understanding how to relate the four predicted quantities of each box detector, at different resolutions, to the bounding box coordinates in the actual image (expressed as pixels or percentages relative to the image size).
Concretely, if I have an image with a bounding box for an object, how do I convert that bounding box to labels for the SSD model to train on?


texts: how to increase the coverage of wordnet using wikipedia

texts:
I want to increase the coverage of wordnet by increasing the synset count.

Is there a way in which I can proceed and increase sysnets. I am trying to work on Hindi wordnet with Princeton wordnet  as the  standard


texts: Difference in threshold values for DBSCAN and Threshold clustering

texts: I am trying to cluster similar faces using Facenet embedding approach. I am extracting a 256 feature vector using Facenet model on a standardized labelled Celeb-faces dataset, and trying cluster using two methods - one, threshold clustering and two, DBSCAN. The idea of threshold clustering is, we manually put two images in the same cluster if the L2-norm between those two images are less than the threshold value = 1, else place it in different clusters. The reason I am taking threshold to be 1, because Facenet is trained with the margin loss=1.   If we use the same threshold value epsilon=1, I am getting abysmal performance in DBSCAN. But once I take epsilon=0.504 I am getting really good accuracy. I want to understand what is really going on ?


texts: What is a 'hidden state' in BERT output?

texts:
I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'.

I understand what RNN's have a 'hidden state' that gets passed to each time step, which is a representation of previous inputs. But I've read that BERT isn't a RNN - it's a CNN with attention.

But you can output the hidden state for each layer of a BERT model.  How is it that BERT has hidden states if it's not a RNN?


texts: What is the shape of Decision Boundary for binary classifiers?

texts: There are different types of Binary Classifiers each will produce a different type of decision boundary some produce decision boundary as straight-line others as circle or square, some of the classifiers also produce decision boundaries as polynomial functions. Which are the Binary Classifiers which produce straight line as decision boundary? What are the other shapes of decision boundaries associated with each algorithm?


texts:
I am new to deep learning and I am trying to train a NN to recognize house numbers gathered from street view. I have already managed to recognized MNIST sequence of hand written digits by means of a CNN. In that case I have trained the CNN with a thousands of examples consisting of 1 from up to 5 digits sequences created by me.

I have thus thought to do the same thing for SVHN, I have cropped the numbers and I now have a large dataset of single digits from street view. However the MNIST data set is very homogenous (i.e. images have same size, same angle, same color, etc.) while SVHN is obviously not. Trying to create sequences of digits from different SV photos led to weird examples that are actually not representative of the real word objects I want to model.

So I am wondering if it was better to train a NN on the single digits and then use it to recognize sequences. Would it be possible? How?


texts: Feature Selection: Having a large number of features select those features that classify the target class best

texts:
I have time series data and I'm using the cesium package to extract several features.

Now the thing is, the number of extracted features is pretty significant. I have tried various visualizations to make sense of what features classify my labels the best but it is too cumbersome for so many features.

Are there any algorithms or packages available that I can use for feature selection?


texts: Need to calculate derived metrics in a classifier

texts:
I have a classifier that predicts a class given roughly 100 datapoints. The classifier that tends to perform best on my data is RandomForest.

I have a lot of metrics such as revenue for each of the past months, number of web traffic coming from different sources etc.

Here is my question, if I should calculate derived metrics as input for the model (e.g. year over year growth %, % traffic from each source etc.) or just put the features into the model as it is?


texts:
I will try to implement a k-means algorithm over this dataset:

Team_Categorical  CreditAmount_Numeric   Retired?_Binary
       A                 15.3                   1
       B                 12                     0
       C                 6.2                    1

In order to apply k-means algorithm I need to transform the categorical variables into numeric. So I will apply binning transformaton over the Team_Categorical field. So I will have this dataset:

A    B   C  CreditAmount_Numeric   Retired?_Binary
1    0   0          15.3                 1
0    1   0          12                   0
0    0   1          6.2                  1

My question is: Should I transform CreditAmount_Numeric to binary too?


texts: How can i get all the values related to the first column in a direct or indirect relationship?

texts:
I wanna write a function using python that allows me to compare each value from the second column of my DataFrame with the first column and if they don't matche compare the second column in the same iteration with the first column in the next iteration.
WHat I have :

What I want :


texts: How does epochs related with converging the model?

texts: I have read on Internet that epochs is used to give the time for the model to converge but I don't know how ? . I was thinking that epochs is used because to train the model sufficient times . How does model convergence relates with epochs . Also tell me that why epochs is useful ?


texts: How to read the output of Binary cross entropy?

texts: Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How to calculate its binary cross entropy? And how output value decides whether a sample belongs to class a or b? I also want to know that that in the formula of binary cross entropy what is 1-p?


texts: Association rule learning for multi-classification suggestions

texts:
The task is the following: given a training set of medical symptoms and an associated diagnosis, output a list of the most likely diagnosises for a combination of symptoms.
As of now, a solution exists which makes use of association rule learning methods: we find rules on the attributes of the training dataset and infer the likely diagnosises and their probability from the confidence we have in said rules to hold true.
However, this method doesn't seem to be able to scale for large datasets because of the sheer number of different attributes ($10^4$) and possible classes ($10^3$). Hence my question: are association rules a viable solution for this problem? Are there any alternatives?


texts: Hyperparameter Tuning in Random Forest Model

texts:
I'm new to the machine learning field, and I'm learning ML models by practice, and I'm facing an issue while using the machine learning model.
While I'm implementing the RandomForestClassifier model with hyper tunning it's taking too much time to predict output. And I'm also using GridSearchCV on it. so it's take much time.
Is there any way how can I solve this problem.
OR, Is Google Colab or Kaggle Notebook editor can perform better than Jupiter Notebook ?


texts: Nesterov Accelerated gradient is in correct order?

texts:
I've implemented Nesterov Accelerated Gradient (NAG) (link, section: "Nesterov Accelerated Gradient"), however not sure it's viable.

$$v_t = \gamma v_{t-1} +\eta\nabla_\theta J(\theta - \gamma v_{t-1})$$
$$\theta = \theta - v_t$$

I attempted the following (consider a minibatch consisting of merely 1 training case):

run a forward pass, and get the error
Apply the scaled previous momentum, (momentum_tMinOne * coeff) to the weights
run a backward pass using the error of each pass.
set momentum_tMinOne = currentGradient, but don't apply it to the weights yet
repeat forward pass, from 1.

Do I require to remember something like momentum_tMinTwo,  or just keeping hold of the previous momentum will suffice?

Am I correct that this way we don't need to keep two matrices for momentums, but only need momentum_tMinOne matrix?


texts: Best practices for memory management in image processing

texts:
I'm relatively new to data science, and I'm trying to create a CNN model for the Kaggle melanoma competition. I've created the following two functions to transform a folder of JPEG images to a Numpy array:
def keras_pipeline(file):
    TARGET_SIZE = (100,150)
    img = load_img(file, target_size=TARGET_SIZE)
    img_array = img_to_array(img)
    return img_array

def files_to_array(path, files_list):
    files = [path+file for file in files_list]
    with concurrent.futures.ProcessPoolExecutor() as executor:
        img_map = executor.map(keras_pipeline, files)
    return img_map

I would then use that array inside a convolutional network.
I'm running on an ASUS TUF A15, with eight cores, 16GB of RAM, and an NVIDIA GeForce GTX 1660Ti GPU. I've tried running this code with multiple resizing TARGET_SIZEs, and each time I overload the RAM. I can try to increase the swap, but I assume it will crash no matter how much I use.
Is there any literature or documentation that discusses RAM usage in image processing, or a way to calculate how much memory will be used by certain sizes of Numpy arrays? What is the best practice for dealing with memory limitations in a situation like this?


texts: Implementing back translation as a data augmentation for text classification

texts:
Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very different language families (but very well supported for economic reasons such as Chinese, Russian, Spanish, Korean, Arabic...) could make for a diverse set of effects occurring in the back translation.

Commercial translation APIs would be a straightforward way of doing this, but without free API key or budget from my organization (would not qualify as academic) that's quickly quite expensive for a private thing.

Pretrained translation models would seem like an obvious alternative (I have a GPU for inference, but clearly that's not enough to train all the models from scratch), but I could e.g. not find those for any OpenNMT variant. Are there any recommendations from others that have used this approach?


texts: Spark: how to process tree aggregation and statistic

texts:
I have a description of file system in form of a csv:

path;size;inode;type
/folder1/folder1.1;5;1;d
/folder1/folder1.1/file1;10;1;f
/folder1/folder1.1/file2;30;2;f
/folder1/folder1.1/folder1.1.1;5;4;d
/folder1/folder1.1/folder1.1.1/file3;300;5;f
/folder1/folder1.1/folder1.1.1/file4;20;6;f
/folder1/folder1.2;5;7;d
/folder1/file5;30;8;f
/folder1/file6;70;9;f
....

If I put this information in a RRD I will have 4 columns and I could make easy aggregation such as the sum of the size.

However here my goal would be to have a aggregation by folder of different stat such as:

Recursive file count per folder
Recursive total size of each folder
Recursive mean size of file for each folder

If I stop to those 3 values my result could look like this :

path;file count;total size of folder;mean file size in folder
/folder1;6;475;52
/folder1/folder1.1;4;365;73
/folder1/folder1.1/folder1.1.1;2;320;160
/folder1/folder1.2;0;0;0

I have several questions about that kind a treatment :

Is it relevant to use Spark for such calculation ( I have almost 500 Go of CSV to handle) ?
If relevant, what would be the best way to approach the problem, should I split my path column and do sub aggregation or should I use method such as TreeReduce and TreeAggregate or should I use an other way ?

Thanks for any advise on how to handle such problem and feel free to move this question to any Stack site if it belongs elsewhere.


texts: Constrained k-means algorithms in R (must-link constraints)

texts:
I currently face an unsupervised learning task that is to be approaches using clustering. More specifically, it is a segementation task and hence there is some prior knowledge about a) the number of clusters and b) the rough content of each segment.
From literature this seems like a prime example, where "constrained k-means" would come into play and I'm really eager to try it out. Both of the above mentionend pieces of prior knowledge can be incorporated in "constrained k-means" as a) is represented by k and b) can be expressed in "must-link constraints". (For a good overview over constrained k-means see Wagstaff, Cardie, Rogers, Schrödl, & others (2001).)

My problem now is that I can't find a suitable implementation in R on CRAN.
Programming it myself in R would be ok, based on the pseudocode, but my programming is most certainly not efficient enough, since we are talking about a somewhat large dataset (75.000 x 30).

So, I turn to this community hoping that someone has found an efficient implementation of "constrained k-means" in R or is willing to provide code that is somewhat efficient.

Thank you very much,

Clemens

Update 23.06.2016:
Thanks for your replies so far. In order to make the problem a little more succint please see a sample data file here. (Note: this is not an excerpt of my original data as it is highly sensitive, but an artificial dataset with the same characteristics (less rows though))

Two examples of simple constraints:

all observations that have either a 1 in v305 or a 1 in v306 must be linked (must-link-constraint)
all observations that have a 1 in v258 and v262 must be linked (must-link-constraint)

Currently looking at your suggestions, Rahul. Thanks very much.


texts: How to verify a CNN encoder works as expected?

texts:
I am using CNN as a part of kernel warping. The purpose here is to reduce input dimension (from N*M to K *1). The input data is not image data. I suspected that the CNN network might not work as I expected. How can I verify my assumption? I am using pytorch, by the way.

Thank you in advance.


texts:
I am new in this world, I am going to start working on my final degree project about data mining. I need to look for an ontology related with water supplier companies.

I mean, I have words like: SuppliedWater, RegisteredWater, LengthSupplyingNetwork...

I am looking at lod-cloud, but I don't know where to start looking for this.

Any help would be appreciate.

Thanks for your help.


texts: How to model machine learning problem for cache replacement policy?

texts:
I am trying to implement machine learning on Cache Replacement Policy. I want to train a ML model on labelled data acquire from Belady's Optimal Algorithm for Cache replacement policy.

For example,
If
X = [a,b,c,d,a,d,h,f,t,u,b,d,,t,u,i,k,h,w,r,t,b,v,c,d,,h,j,j,u,u,d,s,e,w,g,v,b,u,i,k,m,h,b]
is sequential block request and my cache frame size is 3, then at given time my cache would have 3 blocks saved.
Belady algorithm suggest which block from cache to remove for accomodating new block from the sequence.

So my labeled data would be current cache situation and label would be block which I want to remove for next new block.

For example,

| Cache   | Evict |
|---------|-------|
| [a,b,c] | c     |
| [e,g,v] | g     |
| [a,g,d] | a     |
| [w,t,p] | p     |
| [a,g,h] | h     |
| [q,f,v] | q     |
| [z,d,t] | z     |
| [t,g,m] | m     |
| [a,l,k] | l     |
| [p,d,q] | d     |

I want to train such data to predict the evict block given the cache configuration. (order of block in cache does not matter).

Please provide any material/resources that provide this type of ML models. I suspect it is a kind of scheduling or addignment problem but I could not find a good material that I can replicate.

Thank you in advance.


texts:
I often use Nose, Tox or Unittest when testing my python code, specially when it has to be integrated with other modules or other pieces of code. However, now that I've found myself using R more than python for ML modelling and development. I realized that I don't really test my R code (And more importantly I really don't know how to do it well). So my question is, what are good packages that allow you to test R code in a similar manner as Nose, Tox or Unittest do in Python. Additional references such as tutorials will be greatly appreciated as well.

Bonus points for packages in R similar to

Hypothesis

or
Feature Forge

Related Talk:

Trey Causey: Testing for Data Scientists


texts: Any pythonic way to auto determine imbalance class problem, specially in multiclass scenario?

texts:
A data is imbalanced if a target class proportions are unequal and typically, heavily biased. But, what is the exact measurement of this heavy bias?
Before applying imbalance techniques (SMOTE, ADASYN, Tree classifiers,  etc.), I want to auto detect if problem belongs to imbalance class problem.


texts:
why if I run my app as a super user(root) it gives me error,"ModuleNotFoundError: No module named 'tensorflow' " , while if I run it normally it works?
from keras.models import Sequential
from keras.layers.core import Dense, Activation

the error :
Traceback (most recent call last):
File "test.py", line 4, in <module>
from tdfmaze import *
File "/home/maicmania/PycharmProjects/smartWaste/tdfmaze.py", line 3,                      <module>
from keras.models import Sequential
File "/usr/local/lib/python3.8/dist-packages/keras/__init__.py", line
3 , in <module>
from . import utils
File "/usr/local/lib/python3.8/dist-packages/keras/utils/__init__.py",
line 6, in <module>
from . import conv_utils
File "/usr/local/lib/python3.8/dist-packages/keras/utils
/conv_utils.py", line 9, in <module>
from .. import backend as K
File "/usr/local/lib/python3.8/dist-packages/keras/backend
/__init__.py", line 1, in <module>
from .load_backend import epsilon
File "/usr/local/lib/python3.8/dist-packages/keras/backend
/load_backend.py", line 89, in <module>
from .tensorflow_backend import *
File "/usr/local/lib/python3.8/dist-packages/keras/backend
/tensorflow_backend.py", line 5, in <module>
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'


texts: how to predict the time series data with two target outputs

texts:
I am a newbie to time series problems, I am doing one of the time series problems, I a have 317 columns and, I need to predict the stock Revenue and Price, I am trying to do it, but the many columns contain more 0's and some of the info values,
please help me how to solve it.


texts: How to cluster sentences based on company names from a post(s) containing several company names using similarity metric.

texts:
My corpus contains several posts having text for several companies i.e. each post contains information about several companies.

I want to cluster the information based on few company names that I can specify. Clustering should be based on some similarity matrix such as euclidean or cosine similarity.

Which algorithm to use based on company name that I can specify and which similarity method to use?


texts:
I have a question regarding the pre-training section (in particular, the Masked Language Model).

In the example Let's stick to improvisation in this skit, by masking the word improvisation, after applying BERT (followed by the FFNN and Softmax), by looking at the probabilities of each of the possible English words,  we are able to correctly predict that the masked word was improvisation.

Is it possible to actually play with this by using my own examples? I was wondering if I can input a sentence in a different language (from the multilingual model) and have a sorted list of the most probable words that were masked in the original sentence. If it's possible, what needs to be tweaked?

Any help would be greatly appreciated.


texts: I have recently stepped into impurity based criteria for decision trees and I was just wondering why do we really need an impurity based criteria model such as the Gini index? What if we could simply label the entity with the majority class? What would be the effect on results?


texts: File format where column names are repeated on each row

texts:
I have received a dataset in text file with the following format

col1=datac1r1,col2=datac2r1,col3=datac3r1
col1=datac1r2,col2=datac2r2,col3=datac3r2
col1=datac1r3,col2=datac2r3,col3=datac3r3
col1=datac1r4,col2=datac2r4,col3=datac3r4

Each row is a unique entry, with columns separated by comma, just that the column name is repeated in each element.

I need to parse this in R and analyze it. I have worked with csv files extensively, but I have never seen this format before.

Is it a std. format I can import it in? Or do I need to write a script to convert it into a csv format?


texts: How to feed training set labels into Keras LSTM

texts:
I’m implementing an LSTM with Keras and I know that I have to reshape the training dataset in a 3D object. Basically I have a dataset of shape (300000, 839)and I reshape it as (100000, 3, 839) because I want a timestep of 3. The question is: how could I treat the training set labels? Have I to reshape also them? if I reshape the labels to 100000, I don't truncate 200000 labels, since the starting number of labels is 300000?

Thanks in advance.


texts: Does the bias of an artificial neuron adjust or remain constant during training?

texts:
Forum contributor David Waterworth wrote: "we train $w_{0j}$", here $w_{0j}$ is the bias of an artificial neuron.

However, this Wikipedia article

https://en.wikipedia.org/wiki/Artificial_neural_network

(in section "Optimization", under subtitle "Algorithm")

seems to indicate that the bias does not adjust during training:

(I only quoted relevant sentences.)

"Let $N$ be a neural network with $e$ connections, $m$ inputs, and $n$ outputs.

$w_0, w_1, w_2,$ ... denote vectors in $R^e$. These are called weights.

The output of the algorithm is then $w_p$."

In other words, the training/optimization algorithm starts with $w_0$, then produces $w_1$, then produces $w_2$, etc.

Please note that $w_1, w_2, ...$, are $e$-dimensional vectors, and $e$ is the total number of connections.

So, does this Wikipedia article say that only the weights of the connections adjust, but the bias (weight of the neuron) does not adjust?

If the bias also adjusts, then the output vector of the training/optimization algorithm would have more dimensions - The number of dimensions would have to be:

[the total number of connections $e$] plus [the total number of neurons which have biases].

Perhaps the training/optimization algorithm has evolved since this Wikipedia article was written?


texts: Using an autoencoder for anomaly detection on categorical data

texts:
Say a dataset has 0.5% of its features continuous and 99.5% categorical (binary) with ~2400 features in total. In this dataset, each observation is 1 of 2 classes - Fraud (1) or Not Fraud (0). Furthermore, there is a large class imbalance with only 2.6% of examples being Fraud, and the other ~97% of examples being Not Fraud.

Say we want to to predict whether a given example is Fraud or Not Fraud, and we take an anomaly detection approach using autoencoders.

Given the mixed data types in the dataset, in general, will an autoencoder, trained on only the Non Fraud examples, perform well in predicting Fraud examples? Is there any literature to suggest what architectures work best / if some preprocessing should be performed beforehand (scaling and PCA)? I ask because I feel an autoencoder may be hard to train with binary features.


texts: Pandas change value of a column based another column condition

texts:
I have values in column1, I have columns in column2.
What I want to achieve: Condition: where column2 == 2 leave to be 2 if
 column1 < 30 elsif change to 3 if column1 > 90.

Here is what i did so far, the problem is 2 does not change to 3 where column1 > 90.

filter1 = data['column1']
for x in filter1:
    if x < 30:
        data['column2'] = data['column2'].replace([2], [2])
    else:
        data['column2'] = data['Output'].replace([2], [3])


texts: Which is the "BEST" deep learning model for "Custom" object detection for images & real time. YOLO v3, v4, v5, EfficientDet?

texts: Whenever I look for object detection model, I find YOLO v3 most of the times and that might be due to the fact that it is the last version created by original authors and also more stable. In 2020, a new author released unofficial version called YOLO v4 and just after 5 days, another author launched YOLO v5. I am confused that if I have to chose one of the models, which one should I chose? EfficientDet is better than YOLO v3 in my opinion but there is very less talk about that. Is there any other model which is better than thsese? I want to know which models should I chose if I'm detecting objects in images and videos respectively. When speed is not an issue, which one is the best model and when speed is an issue  like in videos, which one should I use?


texts: Is there any way to logically partition dataset while training model?

texts:
I have list of questions (~5k) that are open ended and I want to check answers with pre trained ML model.
Schema for train dataset looks like this:
QuestionId (string) | Answer (string) | Status (int)

Given QuestionId and answer, Status must be determined for the given question.
I have two options:

Train one big model and use it for all questions.
Train model per question.

I believe second option will have better results with small learning dataset, but it has it's downsides too.
I am interested in if there is any way to build model that will consider only values attached to the given question while making prediction? In other words, values of other questions must not be considered.


texts:
Is there an easy way to convert API data with python commands and output the results in CSV format?

Here's what I'm doing now:

import requests

params = {
  "api_key": "fec742c1-c846-4343-a9f1-91c729acd097",
  "format": "jsonp"
  }
r = requests.get('https://data.sa.gov.au/data/api/3/action/datastore_search')
print(r.text)

# This bit of code will write the result of the query to output.csv

with open('output.csv', 'w+') as f:
    f.write(r.text)

This code doesn't work. The API data is available here


texts: Help needed in interpreting the loss, val_loss vs epoch plots for an autoencoder training?

texts:
I am training a variational autoencoder and I am getting a loss-plot as follows:

Rigt after epoch 224, val-loss overtakes train-loss and sort of getting bigger but at an extremely slow pace as you can notice. I trained for 300 epochs.
Any opinion about the training. I don't think it is overfitting the data. But I want to be sure and hence seeking opinion from the data science community.
Thanks.


texts: What is the gradient descent rule using binary cross entropy (BCE) with tanh?

texts:
Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE).

So we have

$$
\Delta \omega = -\eta \frac{\delta E}{\delta \omega}
$$

Now we have BCE:

$$
E = −(ylog(\hat{y})+(1−y)log(1−\hat{y}))
$$

Considering my output is $\hat{y} = tanh(\omega .x)$, $x$ is my input vector and $y_i$ is the corresponding label here.
$$
\frac{\delta E}{\delta \omega} = \frac{\delta −(ylog(tanh(wx))+(1−y)log(1−tanh(wx)))}{\delta \omega}
$$

Now on this website they do something similar for the normal sigmoid and arrive at (eq 60):

$$
\frac{σ′(z)x}{ σ(z)(1−σ(z))}(σ(z)−y)
$$

Could we use that and continue there? We can get the derivative like this and get:

$$
\frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y)
\\= \frac{x-xtanh(wx)^2}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y)
\\= \frac{x-x\hat{y}^2}{\hat{y}(1−\hat{y})}(\hat{y}−y)
\\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}}
$$

Wherever I look, I don't find this :)

Update

Given the first answer that gives $(1 + \hat{y})(1 - \hat{y})$, we arrive at the same

$$
\frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y)
\\= \frac{x(1 + \hat{y})(1 - \hat{y})}{\hat{y}(1−\hat{y})}(\hat{y}−y)
\\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}}
$$


texts: Confusion in applying k-fold Cross Validation to dataset

texts:
I have a dataset which is already divided into 10 folds with each fold having training, validation and test sets. I'm not able to understand how to apply 10-fold cross validation on this dataset.
In general, if we want to apply k-fold cross validation on a dataset, the procedure is as follows:

In my case, the dataset is already divided into 10 folds and each fold contains validation and test sets in addition to training set. It would be helpful if someone can guide me, how to do 10-fold cross validation for this kind of dataset.


texts: What algorithmic solution would you use for this scenario?

texts:
The Project In a Nutshell
Use an algorithmic solution to predict with 70%+ accuracy in as close to real-time as possible the increase and decrease of at least three numeric incremental movements for a few selected primary value inputs (example: five values) by cross-referencing multiple numeric inputs (example: 100+ inputs) live-streamed at a refresh rate of 0.2 seconds.
The Primary and Supporting Grids
There are two main types of input data streams, “Primary” and Supporting” as explained below and which can be visualized by the two visual example reference grids. There will only ever be one primary grid but there could be multiple supporting grids, the picture reference example has only one supporting grid shown.
Both example grids show fifty individual input squares in each grid, but that amount will change according to needs. The input values in every grid square are to be understood as “fixed in place” which means whilst the numerical values within each grid square will change, the actual data stream input for each grid square is always the same one.
Note: The amount of pre-chosen grid square data streams shown in the primary grid example is five, but that may be changed for another amount depending on requirements
Example grid picture explanations
The left Grid is the “Primary Values” grid. There are five selected individual values represented by the differently coloured numbers (circled). All values in the grid are “live” for cross-referencing but only the selected grid data stream values would be for prediction. All the values will be changing regularly, but it is important to know they increase and decrease in increments of 0.01, so one increment movement increasing, for example, might be going from “5 to 5.01”. The overall goal is to predict the incremental movement mentioned in the opening title “Nutshell” of at least a trend movement of three (example 5 – 5.03) with a 70% accuracy.
The Right Grid is an example of only one (1) “Supporting Values” grid. Notice the same grid numbers as the primary grid are all black (circled) to show these are not for prediction output themselves, however, they are still linked directly to the corresponding Primary Grid interface values relative to their grid position. All the values in the grid are “live” and used for cross-referencing purposes. The data stream input values of supporting grids are from different sources and can either increase or decrease in increments or fluctuate in varying amounts of no specific order.
Summary:

There is a direct relationship for all the "pairing" of values (same grid square location) between the two main types of grids, but overall every grid square is being monitored and cross-referenced by the algo to find trends.

The algo will need to give a prediction accuracy percentage output for each selected primary value and continuously re-evaluate and update the percentage strength as needed.

The main goal is to predict the incremental movements of the selected primary values in real-time to an accuracy of %70 or above.

My project progression to date:
Thus far I’ve coded a working infrastructure in Python with a no-SQL database (MongoDB) and am now studying solutions based on Deep Learning and LSTM (using Keras/TensorFlow). My programming background has not been extensive with ML but I’m enthusiastic with the learning curve. Maybe a similar algorithmic solution might be those used in real-time image recognition or possibly algo's that are used for live financial market trading price prediction. Unfortunately, I have the constraint of an NDA and can’t fully divulge the end-use for this project, but hopefully, the details given are enough to understand the concepts and I will answer questions as best that I can. Any advice would be very appreciated on which direction/s to explore the algorithm/data-science side of this project, thank you for reading this and hopefully taking some time to reply.


texts: How to implement distributed deep learning on small master-slave architecture through data parallelism approach?

texts: I am a beginner and I would like to deploy the distributed deep learning model followed by Hadoop on a toy example.  Like I want to use three Personals computers (PC), one would be work as a parameter server and the other two would be work as worker machines. Here I initially want to configure the Hadoop over the three machines (do not know exactly how it would be done on the three machines). Then distribute the data into pieces over the two worker machines via the Parameter server machine for training. Suppose I have 10 GB of data, so 5 GB would be a shift to the first worker personal computer and the other 5 GB of data would be allocated to the second PC. Then I would like to apply the data parallelism model synchronously on the data set. What would be the steps to implement distributed deep learning system on these small network machines?


texts: Segmenting pandas dataframe with lists as elements

texts:
I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. df has 5000 rows and 30 columns.
df = pd.DataFrame({'col1':list('mno'),
                    'col2':[['a','b'],['d'],['a','b','c','d']],
                    'col3':[1,2,3],
                    'col4':[[1,3,4],[5,6,7],[8,3]]})
df

   col1     col2              col3        col4
0   m    ['a', 'b']            1        [1, 3, 4]
1   n    ['d']                 2        [5, 6, 7]
2   o    ['a', 'b', 'c', 'd']  3        [8, 3]

I am hoping to use k-prototypes to segment customers into distinct groupings since there is no ground truth on class labels and I have mixed data. However, I am unsure how to effectively analyze the columns that have lists of multiple values, e.g. col2 and col4. I looked into one-hot encoding to break up those columns into multiple dummy columns, e.g. pd.get_dummies, but I'm concerned that creating a sparse matrix may negatively affect my results. Can anyone suggest a better way of handling this problem?


texts: Weight update to fully convolutional network when supervision is only for a patch

texts:
I had a fundamental question, that is independent of any DL framework. In a fully convolutional network, if we forward an image of size 1000 x 1000, but only provide supervision signal for a 100 x 100 crop of the output, how are the weights of the convolution filters expected to be updated?

Because the same filters were applied to all pixels.

Should they -

update all the filters with the average of the update derived by backproping the 100x100 pixel crop?
Or should they take the average for all the pixels. This means the update will be scaled down by 1e-2, because the gradient from most other pixels is zero.

Finally, I do this in PyTorch, so does anyone know which approach they use?

Thanks

Edit 1 - More information -

When computing the scalar loss, I take the average over the patch that I chose. Under the assumption that the error is evenly distributed, the scalar loss will have the same value.

But, this loss will only give gradients to a 100 x 100 patch. Say the the receptive field is x, it'll impact a max of (100 + 2x) * (100 + 2x) pixels at the shallowest layer. Since the conv filter was applied across 1000 x 1000, many of indexes will receive no grad. Does PyTorch ignore these for update calculation or consider them 0? And which is the correct way?


texts: Does CRNN use sparse tensor value for its label?

texts: I just read paper about cnn + rnn for text recognition. The labels of dataset is tensor of char index (e.g [0, 1, 2 ] for image with label "abc").  Since the label of each input has different length do I need to convert the label to sparse tensor value ? since the paper does not mention about it.


texts: Scikit-learn Random Forest - model changes as result of input scaling

texts:
Everything that I've read about random forests has indicated that they do not require scaling of inputs and that scaling should not affect the construction of the model.  Here's a quote from another SE question (https://stats.stackexchange.com/questions/255765/does-random-forest-need-input-variables-to-be-scaled-or-centered):

  Random Forests are based on tree partitioning algorithms.

  As such, there's no analogue to a coefficient one obtain in general
  regression strategies, which would depend on the units of the
  independent variables. Instead, one obtain a collection of partition
  rules, basically a decision given a threshold, and this shouldn't
  change with scaling. In other words, the trees only see ranks in the
  features.

  Basically, any monotonic transformation of your data shouldn't change
  the forest at all (in the most common implementations).

Here's what I'm using currently.  If I remove the weights multiplier, I get a different model (i.e., different value from model.score and different tree depths) despite setting random_state=0 in either case.

model = RandomForestRegressor(n_estimators=10, criterion='mse', random_state=0)
weights = np.arange(1,self.x_train.shape[1]+1)[None,:]
# weights = [[ 1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15.]]
model.fit(self.x_train * weights, self.y_train)

By comparison, I noticed that if I use XGBRegressor rather than RandomForestRegressor, scaling does not change the model.

Is there an obvious mistake that I'm making or is the explanation above not correct?