texts: Does adding new features that contain information derivable from current features help performance?

texts:
So say you have some data that consists of some values:

1.3, 0.9, 1.1

You introduce a new feature which is the average of these values: 3.3

In this example lets say that you know the average of these features is a good indicator to classify data with. My question is that, you're not actually adding anything to the model by creating this 4th feature, all the data is already in the features so does having this average help the model training identify the relationship more easily?


texts: Non-linear data preprocessing before mini-batch gradient descent

texts:
I found some realisation of pretty interesting ML algorithm. https://github.com/EderSantana/DeepEEG We have dataset X, matrix of spacial filter W, matrix of time filter V, and matrices U and B - parameters of classyfier.
$Xfilt_i=ln(W*(X_i*v)^2))$

$Z=(Xfilt)^T * U+B$

$Y\_pred = argmax(softmax(Z))$

So, quite simple logistic regression classification. But most inreresting thing was the way authors tune W,V,U and B  matrices. W was initialized by CSP algorith. V was initialized by ones, U and B by zeros. And then cross-entropy cost function optimised (mini-batch gradient descent) respect to this parameters. And accuracy, achived by this way pretty high. My quiestion is, why we need non-linearity here (squaring and log)

$Xfilt_i=ln(W*(X_i*v)^2))$
?

PS In experiments optimisation algorithm did not converge without this nonlineraity.


texts: LASSO remaining features for different penalisation

texts:
I am using the sklearn LASSOCV function and I am changing the penalisation parameter in order to adjust the number of features killed off. For example for $\alpha = 0.01$ I have 55 features remaining and for $\alpha=0.5$ I have 6 remaining features.

I would expect that the 6 features I am getting for the second case are a subset of the 55 features I am getting in the first case. This is not what is happening however. 4 out of 6 features are not in the 55 of the first case.

Could someone explain to me the intuition behind why this is happening?

Thanks,
P


texts: How to split data into 3 parts in Python - training(70%), validation(15%) and test(15%) and each part have similar target rate?

texts:
I'm working on a company project which I will need to do data partition into 3 parts - Train, Validation, and Test(holdout).

Does anyone know how I can split the data into 3 parts above and each part will have similar response variable(target rate) - (similar accuracy for classification and similar mean of (response) for regression.

I know how to split data into 3 parts by using train_test_split function from SKLEARN

from sklearn.model_selection import train_test_split

x, x_test, y, y_test = train_test_split(xtrain,labels,test_size=0.2,train_size=0.8)
x_train, x_cv, y_train, y_cv = train_test_split(x,y,test_size = 0.25,train_size =0.75)

But this does not give a similar target rate, can someone help me?


texts: Open source solver for large mixed integer programming task?

texts:
I'm currently using General Algebraic Modeling System (GAMS), and more specifically CPLEX within GAMS, to solve a very large mixed integer programming problem. This allows me to parallelize the process over 4 cores (although I have more, CPLEX utilizes a maximum of 4 cores), and it finds an optimal solution in a relatively short amount of time.

Is there an open source mixed integer programming tool that I could use as an alternative to GAMS and CPLEX? It must be comparable in speed or faster for me to consider it. I have a preference for R based solutions, but I'm open to suggestions of all kinds, and other users may be interested in different solutions.


texts:
How do different models take in account independence/dependence of features?

I've only found very rough "pointers" as to what assumptions different models make about the independence/dependence of features, but I've not found nothing holistic yet.


texts: When to use bayesian linear regression instead of linear regression?

texts:
When does it make sense to use a bayesian approach, maybe in context to linear regression?
To be more concrete: Assume you measure a certain number of devices and you wanna' check the linear relationship between, e.g., voltage and current.
Why bayesian and/or why not? How would it be different?


texts: Good practices for manual modifications of data

texts:
More often than not, data I am working with is not 100% clean. Even if it is reasonably clean, still there are portions that need to be fixed.

When a fraction of data needs it, I write a script and incorporate it in data processing.

But what to do if only a few entries needs to be fixed (e.g. misspelled city or zip code)? Let's focus on "small data",  such as that in CSV files or a relational database.

The practical problems I encountered:

Writing a general script trying solve all similar errors may give unintended consequences (e.g. matching cities that are different but happen to have similar names).
Copying and modifying data may make a mess, as:

Generating it again will destroy all fixes.
When there are more errors of different kinds, too many copies of the same file result, and it is hard to keep track of them all.

Writing a script to modify particular entries seems the best, but there is overhead in comparison to opening CSV and fixing it (but still, seems to be the best); and either we need to create more copies of data (as in the previous point) or run the script every time we load data.

What are the best practices in such a case as this?

EDIT: The question is on the workflow, not whether to use it or not.

(In my particular case I don't want the end-user to see misspelled cities and, even worse, see two points of data, for the same city but with different spelling; the data is small, ~500 different cities, so manual corrections do make sense.)


texts:
I have a json array file that i need to create a smaller sample of for testing purposes. A sample of the file looks like:
sample_json = {
  "targets": [
    {
      "start": 40,
      "end": 73,
      "item": "trust:team_member",
      "lookup": "the cathedral"
    },
    {
      "start": 40,
      "end": 74,
      "item": "contact",
      "lookup": "some text"
    }
  ]},{
    "targets": [
    {
      "start": 40,
      "end": 73,
      "item": "trust:team_member",
      "lookup": "the cats"
    },
    {
      "start": 40,
      "end": 74,
      "item": "music",
      "lookup": "some other music note"
    }
  ]
    ,
  "Options": {
    "Tags": {
      "test:image": [
        "test 1",
        "test 3",
        "test 4",
      ],
        "test:text": [
        "test 1",
        "test 3",
        "test 4",

      ]
    }
  }
}

I need to carry out Multi-label stratification based on
sample_json['options']['Tags']

I haven't done this before so would appreciate any help or guidance?


texts: Training neural nets: is it important that the data is randomly sorted?

texts:
Let's say I have a neural network that classifies data into A, B or C.
I've heard that it's bad to train the network with data from one class at a time, e.g. A,A,A,...,B,B,B,...,C,C,C. Now, is there a problem with other patterns in the data? For example A,B,C,A,B,C,A,B,C,... and so forth.


texts:
I have a pd.DataFrame
pd.DataFrame({
    'person': ['a', 'b', 'c', 'aa', 'bb', 'cc', 'aaa', 'bbb', 'ccc'],
    'group': [10, 10, 10, 20, 20, 20, 30, 30, 30],
    'target': [1, 2, 2, 3, 2, 3, 1, 2, 3]
})

I want to split df into train and test by group several times (K-Fold), so train and test contains examples from mutually exclusive group subsets. For example, if test contains examples from groups [10, 20] test should contain examples only from group [30]. I also want to keep stratification by target. So target distribution is also considered for groups subsets creation. How is it possible? Didn't find anything in scikit-learn that may help.
In practice I have a df with about 500k rows, 2.5k different persons, 30 groups and following target distribution


texts:
So, I'm a rookie at web scrapping and Python in general, and I'm having quite a hard time learning it. I'm trying to get some data from a csv file, but so far I could only split lines and columns. I don't have the slightest idea of how I'm supposed to proceed from now on.

From a certain CSV file, I'm supposed to say which Scandinavian country (Sweden, Norway and Denmark) won the most gold medals in the following categories: curling, skating, skiing and ice hokey, starting from year 2001.

Here's what I've managed to do so far:

import requests
from bs4 import BeautifulSoup

url = "https://sites.google.com/site/dr2fundamentospython/arquivos/Winter_Olympics_Medals.csv"

# Usando requests
csv = requests.get(url).text

lines = csv.splitlines()

for l in range(1, len(lines)):
  columns = lines[l].split(',')
  print(columns)

Any comments will be greatly appreciated


texts:
I am working on ANN. I have 2497 training examples and each of them is a vector of 128. So the input size is 128. Number of neurons in hidden layer is 64 and number of output neurons is 6 (since classes are six). My Target vector looks something like this : [0 1 0 0 0 0]. This means that the example belongs to class 2. I have used sigmoid as an activation at all layers and sum of squared error is loss. SSE is computed over one epoch. Total epochs are 10k. my loss starts from around 700 and reduces to 450. Should I say that loss is 18% per example since 450 is the loss for all the 2497 examples. How do I interpret this. Is my model good enough? I know that I should test it on unseen data to be sure of its accuracy but still does this tell anything about the performance at all or not.

ps: I am implementing it in C


texts: Why vSGD-fd optimization algorithm isn't popular?

texts:
As the paper by Schaul & LeCun states

The findings are clear: in contrast to the other algorithms tested,
vSGD-fd does not require any hyper-parameter tuning to give reliably
good performance on the broad range of tests: the learning rates adapt
automatically to different curvatures and noise levels.

Auto-adapting learning rate sounds like a huge deal,
but why is everyone seem to be using Adam and other optimizers?


texts: Replace Values in Vector on Specific Place in R


texts: Fitting Gaussian to data: Density-Estimation vs Regression

texts:
Recently I discussed the following topic with a friend. The setting is that we have a one-dimensional set of data. (In the example it was points of students, we would be grading.) The goal was to make a density estimation, but not to use anything "fancy" like kernel density estimation, but just use a Gaussian as the estimation. (Sure that makes the big assumption that the data is Gaussian, but that is not the point here.) We discussed two ways:

Make a density estimation using an unsupervised learning method, e.g. using EM-algorithm. In this case the claim is that simply calculating the mean of the data and the standard derivation is already giving one the parameters to get the Gaussian parametrized the right way.
Add up the number of occurrences for each value and then use a supervised learning regression with the Gaussian as function, powered by an optimisation algorithm.

Through discussion we found out that the two clearly have different outcomes - though probably similar. For case 2 we optimise the parameters of the Gaussian such, that the sum of the distances from the Gaussian to the occurrences is minimized (along the y-axis if you will). For case 1 we optimise the parameters along the x-axis if you will.

Questions

Preliminary question: Is it correct that the EM-algorithm will have the same result as just calculating mean and standard deviation from the data?

Assuming the answer to the preliminary question is "yes, the results are the same":

What is the intuition interpretation of those two approaches?
None of them can be wrong in itself, but one of them could be wrong in the sense of wrong-usage. Meaning: I want to do something and I use the wrong method for it, because of misunderstood interpretation of what happened. So in that sense: Is one of them wrong in a way?

Example Code

I managed to express myself in R code. As one can see from the plots, the result is definitely not the same for any dataset. Only if the data is Gaussian and large the results get to be similar. But that means little to me ($2^x$ and $exp(x)$ converge both to $infinity$ for $x -> infinity$ and they have little in common otherwise).

library(ggplot2)
library(dplyr)
library(tidyr)

std <- 10

datasets <- list(
  data.frame( x = round(rnorm(1000, sd = std))),
  data.frame( x = round(rnorm(1000, sd = std))) %>% filter(x > 0)
)

for(data1 in datasets){

  data1$densest <- dnorm(data1$x, mean = mean(data1$x), sd = std, log = FALSE)
  data2 <- data1 %>%
    group_by(x) %>%
    summarise(count = n())

  f <- function(x, m, sd, k) {
    k * exp(-0.5 * ((x - m)/sd)^2) # 1/sqrt(2*pi*sd^2) *
  }

  cost <- function(par) {
    rhat <- f(data2$x, par[1], par[2], par[3])
    sum((data2$count - rhat)^2)
  }

  o <- optim(c(0, std, 10), cost, method="BFGS", control=list(reltol=1e-9))
  data1$regr <- f(data1$x, o$par[1], o$par[2], o$par[3])
  data1$regrNormalized <- dnorm(data1$x, mean = o$par[1], sd = abs(o$par[2]), log = FALSE)

  g1 <- ggplot(data=data1, aes(x=x)) +
    geom_point(data=data2, aes(x=x, y=count), alpha=0.2)+
    geom_line(aes(y=densest), color="green") +
    geom_point(data = data.frame(mean = mean(data1$x)), aes(x=mean, y=0), color="green") +
    geom_line(aes(y=regr), color="blue") +
    geom_point(data = data.frame(mean = o$par[1]), aes(x=mean, y=0), color="blue")

  g2 <- ggplot(data=data1, aes(x=x)) +
    geom_line(aes(y=densest), color="green") +
    geom_point(data = data.frame(mean = mean(data1$x)), aes(x=mean, y=0), color="green") +
    geom_line(aes(y=regrNormalized), color="blue") +
    geom_point(data = data.frame(mean = o$par[1]), aes(x=mean, y=0), color="blue")

  plot(g1)
  plot(g2)

}

Plots


texts: How to handle unseen labels in test data?

texts:
I get something like TF-IDF of training corpus in python with (something like) TfidfVectorizer. In test data some features (here are the words of test corpus, every word is a feature) are not seen in the training data and because of this, the shape of the test and the train matrix are not equal and the program gives an error (number of columns isn't same and some words in the test data are not seen in the train data).
How should I solve this problem? How should I handle unseen features in test set?


texts: Classification Model based on Ordered Features

texts:
I am trying to build a classifier for a specific card dataset let's say cards or no cards. I am using Mobilenet trained on Imagenet dataset as my classifier and further training it on my dataset. I am able to train it, and it's performance is quite good on the dataset. Let's say my card has for different regions of interest as shown below:-

It is able to perfectly recognize the above passed image as card.

But I am able to fool my classifier if I change the ordering of the regions of card and then giving it to classifier as shown below:-

I want the classifier to learn the exact occurrences of features of cards as in First Image so that if I alter them it should be able to reject that card (like in second image).

Is there any way to achieve this?


texts: How to populate pandas series w/ values from another df?

texts:
I need help figuring out how to populate a series of one dataframe w/ specific values from another dataframe.

Here's a sample of what I'm working with:

df1 = pd.DataFrame({'Year':[1910, 1911, 1912],
                    'CA':[2.406, 2.534, 2.668],
                    'HI':[0.804, 0.821, 0.832]})

df2 = pd.DataFrame({'State':['CA', 'CA', 'CA', 'HI', 'HI'],
                    'Year':[1910, 1910, 1911, 1911, 1911]})

df2['Population'] = pd.Series()

*I'm trying to populate df2['Population'] w/ the corresponding populations from df1 (i.e. the population of a specific state from a specific year)

How can I do this?


texts: Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

texts:
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
S1 = ['Walk','Eat','Going school','Eat','Watching movie','Walk'......,'Sleep']
S2 = ['Eat','Eat','Going school','Walk','Walk','Watching movie'.......,'Eat']
.........................................
.........................................
S50 = ['Walk','Going school','Eat','Eat','Watching movie','Sleep',.......,'Walk']

The number of unique actions in the dataset are fixed. That means some sentences may not contain all of the actions.
By using Doc2Vec (Gensim library particularly), I was able to extract embedding for each of the sequences and used that for later task (i.e., clustering or similarity measure)
As transformer is the state-of-the-art method for NLP task. I am thinking if Transformer-based model can be used for similar task. While searching for this technique I came across the sentence-Transformer. But it uses a pretrained BERT model (which is probably for language but my case is not related to language) to encode the sentences. Is there any way I can get embedding from my dataset using Transformer-based model?


texts: Organization of layers in Keras for a NLP problem

texts:
I have been trying out an NLP problem where I have to predict multi-label-sentiments for some text.

I have 8 labels and 170k training examples, and 140k for the test set.
My final dictionary size is around 190k.

I am using Keras for trying out an NN approach, although I am not sure if my architecture is right, Below is the model which gives me a 95 % on the test set, I am testing out accuracies on 0.7 - 0.3 split while training:

model = Sequential
model.add(Embedding(max_indexes + 1, 100, weights=[embeddings],
                  input_length=100))

model.add(Bidirectional(LSTM(256, return_sequences=True)))

model.add(Convolution1D(256, 5, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 5, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 5, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPool1D())

model.add(Convolution1D(256, 4, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 3, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 3, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPool1D())

model.add(Flatten())

model.add(Dense(64, activation='relu',
                kernel_regularizer=keras.regularizers.l2(0.02)))

model.add(Dense(target_classes_len, activation='sigmoid',
                kernel_regularizer=keras.regularizers.l2(0.02)))
model.add(Dropout(0.1))

adam_opt = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

early_stopping = EarlyStopping(monitor='val_loss', patience=5, mode='min')

save_best = ModelCheckpoint('model_x.hdf', save_best_only=True,
                               monitor='val_loss', mode='min')
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
                              patience=5, min_lr=0.0001)

model.compile(adam_opt, 'binary_crossentropy', metrics=['accuracy'])

Over the internet, and while watching a video of CSE231N and other StackOverflow questions, I found out I should follow:

[(CONV-RELU)*n - POOL? ]*m - (FC-RELU)*K, Softmax
 N~5, M~very large, K>=0 and K<=2

Although it was for ImageNet classification problem and I have an NLP problem at hand.

Also, I found out that should place DropOut after final FullyConnected and BatchNormalization in a sequence with CONV-RELU.

I justed wanted to know what would a good organization of layers for an NLP problem, or any way of evaluating different architectures?

Thanks


texts: Modified Kneser-Ney Zero Weighting Factor

texts:
I’m attempting to implement the modified Kneser-Ney smoothing algorithm on  a 4-gram model.  This smoothing algorithm discounts the higher order probabilities and redistributes the discount among the lower order probabilities by multiplying the lower order probabilities by a weighting factor lambda.

$$\lambda_{low}(w_{i-n+1}^{i-1}) = \frac{D_{n,1} N_1(w_{i-n+1}^{i-1}\bullet) + D_{n,2} N_2(w_{i-n+1}^{i-1}\bullet) + D_{n,3+} N_{3+}(w_{i-n+1}^{i-1}\bullet)}{N_{1+}(\bullet w_{i-n+1}^{i-1}\bullet)}$$

The challenging i'm facing is that the lambda weighting factor is indeed zero at the higher orders, thus producing a zero probability for the n-Gram. For instance, given the following n-Gram "only seeing last word" in which "only seeing last", "seeing last", and "last" are unseen, I get a zero probability regardless of the continuation count for "word".

I've read mounds of literature on the Kneser-Ney Smoothing algorithm, but I've not found anything that specifically addresses this conundrum. Any insights on the implementation of the Kneser-Ney algorithm in such situations would be appreciated.


texts: Recovering data from under sampled data streams?

texts:
I’m trying to analyze a repeated data stream from a sensor.  The data probably has events at the 100us scale, it occurs at a fairly random interval, and the signals are extremely similar.  The problem is that I don’t have any sensors that have sub-millisecond refresh rates.  The best I can do with what I have is about 1ms sensor reads.
So . . .
The idea is that if I sample a large number of event the chances are that the sampling ticks are falling on different sections of the signal.  If this is true then a series of 10-20 events should give me the data that I need to reconstruct the signal.  The signals repeat in a fairly random fashion and are very similar in each dataset so the chances are that each sampling run is gathering “pieces” of the total dataset.
Is there a name for this sort of procedure so that I can start to research how to do it?  Are there algorithms or toolsets in the open source world to accomplish this task?
Right now I’m literally graphing the data on graph paper and overlaying the image.  This is less than ideal . . .


texts:
I'm interested in a neural net that takes a complete set as an input. For example: the net takes as an input a set of m 1-dimensional points and predicts a histogram of this set (e.g. as a vector where each element specifies the number of counts in a given histogram bin.) The training data for this net would be:

Input: a set of n different sets each containing m points.
Output: a set of n histogram vectors corresponding to each of the n sets.

The networks should be invariant to permutations in the order that the members of a set are presented to the network and, ideally, the ability to work with sets with different number of elements.

Is it obvious how to design a neural network with those properties? Is there a name for it?


texts: Tensorflow deep Q neural network wont learn with more than 1 hidden layer

texts:
I've been using TensorFlow in Python with a very simple deep Q neural network (NN) to play a very simple game.
The game is in an array [0, 0, 0, 0] and by picking one of the positions 0, 1, 2, 3, the number picked swaps between a 0 and a 1 so if i picked 2 at the start the game would look like [0, 0, 1, 0]. The game ends when the array is [1, 1, 1, 1]. It is therefore a very easy game for both humans and NNs to play.
What I would like to be able to do is not have to copy and paste sections of my code every time I want to add a new layer. My attempt at this was iterative:
W = []
b = []
hidden_outputs = []
shape = [4, 8, 4]

input_ = tf.placeholder(tf.float32, [None, shape[0]], name='input_')

for i in range(len(shape) - 1):
    print("i: " + str(i))

    ## Create the W and b variables for this layer of my network and add them to the list

    W.append(tf.Variable(tf.random_normal([shape[i], shape[i + 1]], stddev=0.03)))
    b.append(tf.Variable(tf.random_normal([shape[i + 1]])))

    ## Multiply the previous output by the current W and then add b

    if i == len(shape) - 2:
        if len(shape) > 2:
            Qout_new = tf.add(tf.matmul(hidden_outputs[i - 1], W[i]), b[i])
            sig_out_new = tf.sigmoid(Qout_new)
        else:
            Qout_new = tf.add(tf.matmul(input_, W[i]), b[i])
            sig_out_new = tf.sigmoid(Qout_new)
    elif i == 0:
        hidden_outputs.append(tf.nn.relu(tf.add(tf.matmul(input_, W[i]), b[i])))
    else:
        hidden_outputs.append(tf.nn.relu(tf.add(tf.matmul(hidden_outputs[i - 1], W[i]), b[i])))

My logic was that this would allow me to change shape and the shape of the NN would change with it.
This does work and the NN does learn to play my very basic game with shape = [4, 4] or [4, 4, 4] or [4, 50, 4], where the shape[0] is the input layer dimensions, and shape[-1] is that of the output layer.
However, it will not learn to play properly using 2 hidden layers such as shape = [4, 50, 50, 4] and I know that you don't need 2 hidden layers for this game but I'm doing this as a stepping stone for a bigger project.
Looking back at the code you can see that this only happens when the else statement is used.
What I don't know is whether the NN is failing to learn because there is some fundamental disparity between the game and a two hidden layer NN or whether there is an issue with the code in my else statement or I haven't trained enough (it has 200 attempts at the game with 4 moves per attempt) or my learning rate is too high or too low (y = 0.05).
Basically, do you know what causes it to fail?
Full code at here


texts: Subtraction of Positive and Negative Frequencies in Sentiment Analysis

texts: In the Positive Negative Sentiment Analysis,  Would it make sense mathematically to instead of keeping a score of the positive frequencies and negative frequencies of a word, calculate the difference between them? That way each word would have a positivity 'heat' in which a very high value would indicate a very positive word and vice-versa. How this approach would change the model performance?


texts: Predicting most likely application to be opened

texts:
Background

I'm currently preparing for a paper in which I will discuss the ability to predict the most likely application to be opened by the user at the given time.

The application will collect information and learn to predict which application that is most likely to be run based on the learned patterns.

Proposed method

I'm collecting the following features for any given program run on the machine:

Time of day (To predict if the program is being run)
Day of week (To learn when it is used)
Most recently opened program (To see connections between programs)
CPU load (Read below)
GPU load (Read below)
Memory usage (To connect programs to heavy usage)
Screen use (To see if the program has connections with an active screen)
Last mouse movement (To see if the program is actively used when it is
run)
Last keyboard usage (Same as above)
App running (Whether or not the application is running)

The mentioned features should allow for some seasonality to be present and should therefore yield some interesting results.

When the data has been collected, An ANN will be trained with the normalized inputs and each application as a separate output representing the possibility of the program being run. (0-1)

Questions

Is the proposed method adequate for such a task? I've read about using Bayesian classifiers but it doesn't seem like it's what I'm after.

I've also read about this being a "time series" task - something I couldn't quite wrap my head around but I assume I am exploiting through the proposed method. Is this a time series? Does it have to be treated as such?

Should each application be present as a separate output or should the current application be fed in as an input, with the output presenting the possibility of the given app being run?


texts: Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?

texts:
In the category of Gradient Boosting, I find some terms confusing.
I'm aware that XGBoost includes some optimization in comparison to conventional Gradient Boosting.

But are Gradient Boosting Machines (GBM) and GBDT the same
thing? Are they just different names?

Apart from GBM/GBDT and XGBoost, are there any other models fall into
the category of Gradient Boosting?


texts: why we have run make.sh file initially in darknet and YOLO object detection?

texts:
In have seen a couple of text and objection detection algorithm wher the first step everyone dose is to install cython and run a make.sh file . why we have run make.sh file initially in darknet/YOLO object detection ?

Below are the 2 links im referring to build yolo algorithm using darknet and im not able to understand why cython is needed and why it is necessary to compile make.sh file for the first step to build yolo model  ?

How to compile on Linux (using cmake)

How to compile on Linux (using make)


texts: Loss function for Autoencoder of sparse 3D Image

texts:
I have 3D structure data of molecules. I represented the atoms as points in a 100*100*100 grid and applied a gaussian blur to counter the sparseness. (nearly all of the grid cells contain zeros) I am trying to build an autoencoder to get a meaningful "molecule structure to vector" encoder.

My current approach is to use convolutional and max-pooling layers, then flatten and a few dense layers to get a vector representation. Then I reshape and increase the dimension again until the model predicts the probability of there being an atom in a grid-pixel with a sigmoid (see code below).

I am worried that the model does not learn if I use binary cross-entropy, because the data is so sparse. I want a loss function that punishes "not even close" atom predictions more than predictions that were just off by a few grid cells.

latent_dim= 512
input_mol = Input(shape=(100, 100, 100, 8))  # 8 channels for the different atom types

x = DepthwiseConv3D(kernel_size=(9,9,9), depth_multiplier=1,groups=8, padding ="same", use_bias=False)(input_mol) #gaussian blur
x = Conv3D(64, (3, 3, 3), activation='relu')(x)
x = MaxPooling3D((5, 5, 5))(x)
x = Conv3D(32, (3, 3, 3), activation='relu')(x)
x = MaxPooling3D((2, 2, 2))(x)
x = Conv3D(16, (3, 3, 3), activation='relu')(x)
x = MaxPooling3D((2, 2, 2))(x)
x = Flatten()(x)
x = Dense(1000, activation = 'relu')(x)
x = Dropout(rate=0.4)(x)
encoded = Dense(latent_dim, activation = 'relu')(x)

# add noise (variational autoencoder)
z_mean = Dense(latent_dim)(encoded)
z_log_sigma = Dense(latent_dim)(encoded)
z = Lambda(sampling, output_shape=(512,))([z_mean, z_log_sigma])

x= Reshape((8, 8, 8, 1))(encoded)

x = Conv3D(32, (3,3, 3), activation='relu', padding='same')(x)
x = UpSampling3D((2, 2,2))(x)
x = Conv3D(32, (3,3, 3), activation='relu', padding='valid')(x)
x = UpSampling3D((2, 2,2))(x)
x = Conv3D(32, (3, 3,3), activation='relu', padding='valid')(x)
x = UpSampling3D((2, 2, 2))(x)
x = Conv3D(8, (3, 3,3), activation='relu', padding='valid')(x)
x = UpSampling3D((2, 2, 2))(x)
decoded = Conv3D(8, (10, 10, 10), activation='sigmoid', padding='same')(x)

autoencoder = Model(input_mol, decoded)


texts: How to obtain original coefficients after performing linear regression on normalized data?

texts:
I am reading data from a file using pandas which looks like this:

data.head()

   ldr1  ldr2  servo
0   971   956     -2
1   691   825   -105
2   841   963    -26
3   970   731     44
4   755   939    -69

I proceed to normalize this data to perform gradient descent:

my_data = (my_data - my_data.mean())/my_data.std()
my_data.head()

       ldr1      ldr2     servo
0  1.419949  1.289668  0.366482
1 -0.242834  0.591311 -1.580420
2  0.647943  1.326984 -0.087165
3  1.414011  0.090200  1.235972
4  0.137231  1.199041 -0.899949

I perform multivariate regression and end up with fitted parameters on the normalized data:

Thetas:  [[-3.86865143e-17,  8.47885685e-01, -5.39083511e-01]]

I would like to plot the plane of best fit on the original data and not the normalized data using the normalized thetas.

I used scipy.optimize.curve_fit to perform multivariate linear regression and come up with the optimal fitted parameters. I know that the original thetas should be close to the following:

[   0.26654135   -0.15218007 -107.79915373]

How can I get the 'original' thetas for the original data-set in order to plot, without using Scikit-Learn?

Any suggestions will be appreciated.


texts: How to retrain the neural network when new data comes in?

texts: I am new to deep learning. Can anybody help me with the online learning implimentation for deep learning models. As per my understanding, i can save a keras/tensorflow model after training and when new data comes in, i can reload the network back and retrain the network using the new data. I haven’t seen anywhere documenting this method. Is my understanding incorrect? If yes, let me know what can be done so that the model keeps on getting retrained when new data comes in?


texts:
I have read a couple of papers now use rules-based system to create weak labels and then train a BERT-based model only using these weak labels. Both studies have reported better performances on manually labelled gold-standard test data.
However, I just don't follow the logic here. I understand distant supervision and all that. It's been around for a while now. I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as "ground-truth", and more importantly, don't you already know how to create "ground-truth" (by the rules-based system) ??? What's the point of the 2nd step?
Even though the performances on test data are better. It doesn't convince me your ML model (from the 2nd step) have learnt something beyond the weak labels you fed to it.
The only argument I find has some merits is that you basically treat the BERT-based model already as a zero-shot classifier and you are fine-tuning it with weak labels. I am just confused. Can someone pls enlighten me? Am I missing something obvious here?


texts: Are there well-known automated methods for deriving a name for each topic obtained through topic modeling? for a specifically given problem at hand I will probably default to an algorithm on top an ontology, on top the topic modeling results. But I'm highly curious as to whether there's something that I am missing, in the general case.


texts: What are the drawbacks of traditional clustering algorithms?

texts: As a newbie, I am interested what the major drawbacks of traditional clustering algorithms are. The purpose is to understand for which class of data or in which scenarios I shouldn't use traditional clustering methods (e.g. K-means)


texts:
I have an image data set that I want to classify into 3 classes: non-disease, disease-type 1, and disease-type2.

I trained two separate CNN's models, one is classifying the 3 classes, and the second classifies only two classes: disease or non-disease.

Since the accuracy of the second model is way higher, I want to combine the output of both models, meaning, to classify as a disease type only if the first model output was type (1/2) and the second output was disease. hopefully,  this will reduce the false positive error of the first model.

Is there any common way of doing that?
is it O.K to do the combination only on the test set (because the training is separetly) or maybe I need to saparate a validation set for that?


texts: LOGICS: GAN with image as input instead of a noise vector

texts:
i am having an idea for a single-class classifier. I don't know if this is a logical "short circuit", though. The idea is the following:

Instead of a noise vector, i use a "noise-image" as input for training the GAN. Assuming i have a trained network, the classifier would work as follows. For example, the GAN should generate images of apples.

Class 1:
An image of an apple is used as the input for the discriminator. Discriminator says: It's an original image
The same image of an apple is used as the input for the generator.
--> the generated image is an apple again. --> the discriminator says again: It's an original image
--> The difference of both discriminator-outputs is low.

Some other class (here: dog, it represents the "not-apple" class)
An image of a dog is used as the input for the discriminator. Discriminator says it is a "fake" image.
An image of a dog is used as the input for the generator --> the generated image is an apple.
--> the discriminator says it is an original image.
--> The difference of both discriminator-outputs is high. --> This means the input image is NOT an apple.

Is this approach legit? Or does it have some logical flaws? I would like to hear what you think about it!


texts:
Background:
I run a product that compares sets of data (data matching and data reconciliation).
To get the result we need to compare each row in a data set with every N rows on the opposing data set
Now however we get sets of up to 300 000 rows of data in each set to compare and are getting 90 Billion computations to handle.

So my question is this:
Even though we dont have the data volumes to use Hadoop, we have the computational need for something distributed. Is Hadoop a good choice for us?


texts: Can I perform Verification and validation checks on datasets like AndroPRAguard, Drebin(contain malware and benign mobile apps)?

texts:
Verification and validation checks for data:

Verification of data:
- Visual Checks: It checks data visually.

Double Entry Check: It checks duplication of data in the database.

Validation of data:

Range Check: It checks whether data falls between specified lower and upper boundaries.
Format Check: It checks whether the data is in the right format.
Check digits: It checks valid digits in data.
Presence Check: It checks whether data has been entered into the correct field.


texts: Which one first? Imputing missing values or dummy creation?

texts:
While working on model building using Logistic Regression, we did two different ways.

Method 1: Created dummy variables before treating missing values. This resulted in 37 columns expanding to 192 columns.
Method 2: Treated missing values first followed by dummy variable creation. This resulted in 119 columns.

Which one is preferred and why?


texts: Why I am having odd results with this neural network model

texts:
I am kind of new to neural networks, so I was doing some practice with iris data set. But the results are weird. Because I am running the network with the same data (not randomly chosen) and huge number of epochs (500 times), but results are sometimes 0,25 sometimes 1,0 or 0,95. The only thing that random is biases and weights. I think training the network with 500 epochs should remove this randomness at the end. But it seems like it doesn't or there is some another problem that I am not noticed yet. If you help me about this I will be thankful.
(PS: If you give me an advice about making the question better I will edit it, I am new at forum)
import tensorflow as tf
import pandas as pd
import numpy as np
from tqdm import tqdm

def to_one_hot(label_data):
    one_hot = []
    for item in label_data:
        temp = []
        for i in range(3):
            if(i == item):
                temp.append(1)
            else:
                temp.append(0)
        one_hot.append(temp)
    return one_hot

def read_data(n):
    df_data = pd.read_csv('iris.csv')
    x = []
    y = []
    species_dict = {'Iris-setosa' : 0,
            'Iris-versicolor' : 1,
            'Iris-virginica' : 2}
    for i in range(len(df_data['Id'])):
        temp_item = []
        temp_item.append(df_data['SepalLengthCm'][i])
        temp_item.append(df_data['SepalWidthCm'][i])
        temp_item.append(df_data['PetalLengthCm'][i])
        temp_item.append(df_data['PetalWidthCm'][i])
        x.append(temp_item)
        y.append(species_dict[df_data['Species'][i]])
    y = to_one_hot(y)
    x_train = []
    y_train = []
    x_test = []
    y_test = []
    for i in range(50):
        if (i < n):
            for t in range(3):
                x_train.append(x[i+50*t])
                y_train.append(y[i+50*t])
        else:
            for t in range(3):
                x_test.append(x[i+50*t])
                y_test.append(y[i+50*t])
    return x_train,x_test,y_train,y_test

x_train,x_test,y_train,y_test = read_data(40)

n_layers = 30
n_nodes = 7
n_classes = 3

x = tf.placeholder('float', [None,4])
y = tf.placeholder('float')

def neural_network_model(data):
    layers = []
    layers.append({'weights':tf.Variable(tf.random_normal([4,n_nodes])),
        'biases':tf.Variable(tf.random_normal([n_nodes]))})
    for i in range(n_layers-1):
        layers.append({'weights':tf.Variable(tf.random_normal([n_nodes,n_nodes])),
            'biases':tf.Variable(tf.random_normal([n_nodes]))})

    output_layer = {'weights':tf.Variable(tf.random_normal([n_nodes,n_classes])),
            'biases':tf.Variable(tf.random_normal([n_classes]))}

    for i in range(n_layers):
        if (i == 0):
            nl = tf.add(tf.matmul(data,layers[i]['weights']),layers[i]['biases'])
            nl = tf.nn.relu(nl)
        else:
            nl = tf.add(tf.matmul(nl,layers[i]['weights']),layers[i]['biases'])
            nl = tf.nn.relu(nl)

    output = tf.add(tf.matmul(nl,output_layer['weights']),output_layer['biases'])

    return output

def train_neural_network(x):
    prediction = neural_network_model(x)
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits = prediction, labels = y) )
    optimizer = tf.train.AdamOptimizer().minimize(cost)

    hm_epochs = 500

    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        for epoch in tqdm(range(hm_epochs)):
            _ ,c = sess.run([optimizer,cost], feed_dict = {x:x_train, y:y_train})
        correct = tf.equal(tf.argmax(prediction,1),tf.argmax(y,1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
        print('Accuracy : ',accuracy.eval({x:x_test,y:y_test}))

train_neural_network(x)


texts:
My question is similar to this post: What does the long curve-shape t-SNE mean?

but the problem is that my data is not time series, but I also get these long shape line in tSNE plot .

I don't know what that means or how I can get rid of that. I suspect some structure in the data causing that. But I could not find a useful resource explaining what structure in data could cause this.

My data is single cell data, each row is a cell, each column is a biology marker (e.g. CD3, CD4), so the data is the marker expression on each cell.


texts: Python: ValueError: Unknown layer: Functional

texts:
model_path = './models/VGG16_res.h5'

  model = load_model(model_path)

This is the code which I'm using to load a model
TensorFlow version: 2.3.0
I'm not sure why I'm getting this error:
Using TensorFlow backend.
2020-09-19 04:49:24.824483: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-19 04:49:24.843953: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "app.py", line 19, in <module>
    model = load_model(model_path)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\models.py", line 243, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\models.py", line 317, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\layers\__init__.py", line 55, in deserialize
    printable_modul## Heading ##e_name='layer')
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\utils\generic_utils.py", line 138, in deserialize_keras_object
    ': ' + class_name)
ValueError: Unknown layer: Functional

This is my model h5 (VGG16)


texts: Properties for building a Multilayer Perceptron Neural Network using Keras?

texts:
I am trying to build and train a multilayer perceptron neural network that correctly predicts what president won in what county for the first time. I have the following information for training data.

Total population Median age % BachelorsDeg or higher Unemployment rate Per capita income Total households Average household size % Owner occupied housing % Renter occupied housing % Vacant housing Median home value Population growth House hold growth Per capita income growth Winner

That's 14 columns of training data and the 15th column is what the output should be.

I am trying to use Keras to build a multilayer perceptron neural network, but I need some help understanding a few properties and the pros of cons of choosing different options for these properties.

ACTIVATION FUNCTION

I know my first step is to come up with an activation function. I always studied neural networks used sigmoid activation functions. Is a sigmoid activation function the best? How do you know which one to use? Keras additionally gives the options of using a softmax, softplus, relu, tanh, linear, or hard_sigmoid activation function. I'm okay with using whatever, but I just want to be able to understand why and the pros and cons.

PROBABILITY INITIALIZAIONS

I know initializations define the probability distribution used to set the initial random weights of Keras layers. The options Keras gives are uniform
lecun_uniform, normal, identity, orthogonal, zero, glorot_normal, glorot_uniform, he_normal, and he_uniform. How does my selection here impact my end result or model? Shouldn't it not matter because we are "training" whatever random model we start with and come up with a more optimal weighting of the layers anyways?


texts: Right ML mode and metric to minimize FN and FP on imbalanced dataset

texts:
So I have a dataset in which I have to predict class binary label (1 or 0), the problem, out of 120k data points, only 200 have the label '1'.
the aim is to minimize FN and FP.

Which  ML model should I use? Gradient boost or XGBoost or logistic regression. How do I calculate the class weights?
Which accuracy metric captures the minimization of FP and FN?


texts: How to split data in R using dplyr if we want to have rows of the same group to belong to the same split?

texts:
In my current pipeline, I have sensed that there is data leakage. This is because the same person, though with slightly different values, is in both training and testing set. As a result, my model is overfitting.
For eg my data looks like this:
PID       Var_1   Var_2
Person A     0      1
Person B     0      1
Person C     0      0
Person A     1      3
Person B     1      2
Person D     0      1
Person C     0      1

I want to split this data such that the rows of the same person will be in either training or testing set, i.e I want the split to look like this:
Training:
PID       Var_1   Var_2
Person A     0      1
Person B     0      1
Person A     1      3
Person B     1      2

Testing:
PID       Var_1   Var_2
Person C     0      0
Person D     0      1
Person C     0      1


texts: Material Science dataset with feature-dependent inputs

texts:
I'm dealing with a material science/chemistry dataset where I have a bunch of duplicates inputs formulas corresponding to different values of a specific features like temperature. It looks something like this:
Formula           Temperature(K)         (Other features)         target
  NaO2                 280                                         7.5
  NaO2                 340                                         5.6
  ZnO                  370                     ...                 6.7
  LaCuO3               300                                         5.6
  ZnO                  278                                         6.8

of course, I want to perform a regression task here, my question is, how should I treat the inputs?
I would like to train some model that takes into account different temperature values for the same compound. All the other features, based on chemical properties will be of course the same for the duplicates compounds, while the only thing that will vary on the same compounds is the temperature..
Basically the target varies for the same compound based on the temperature.


texts: Transitioning from a maths background, is it possible to find a data science job, by just being able to write my own code (kmeans, kNNs, logistic regression etc) without using any popular commercial software?


texts: Converting pandas dataframe to scipy sparse arrays

texts:
Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning.

Now, if my pandas' data frame consists of only numerical data, then I can simply do the following to convert the data frame to sparse csr matrix:

scipy.sparse.csr_matrix(df.values)

and if my data frame consists of ordinal data types, I can handle them using LabelEncoder

from collections import defaultdict
d = defaultdict(LabelEncoder)
fit = df.apply(lambda x: d[x.name].fit_transform(x))

Then, I can again use the following and the problem is solved:

scipy.sparse.csr_matrix(df.values)

Categorical variables with a low number of values is also not a concern. They can easily be handled using pd.get_dummies (Pandas or Scikit-Learn versions).

My main concern is for categorical variables with a large number of values.

The main problem: How to handle categorical variables with a large number of values?

pd.get_dummies(train_set, columns=[categorical_columns_with_large_number_of_values], sparse=True)

also takes a lot of time.

This question seems to be giving interesting directions, but, it is not clear whether it handles all the data types efficiently.

Let me know if you know the efficient way. Thanks.


texts:
When ML algorithms, e.g. Vowpal Wabbit or some of the factorization machines winning click through rate competitions (Kaggle), mention that features are 'hashed', what does that actually mean for the model? Lets say there is a variable that represents the ID of an internet add, which takes on values such as '236BG231'. Then I understand that this feature is hashed to a random integer. But, my question is:

Is the integer now used in the model, as an integer (numeric) OR
is the hashed value actually still treated like a categorical variable and one-hot-encoded? Thus the hashing trick is just to save space somehow with large data?


texts: Product classification according to description

texts: I have some products, along with their description. I wish to assign USPSC code to each product. I have a really basic doubt here. What exactly is my test file and training file? Eg. Should the training file be entries of product description along with manually entered codes assigned to each product? And the test file only product descriptions?


texts: Stacked barchart, bottom parameter triggers Error: Shape mismatch: objects cannot be broadcast to a single shape

texts:
I am working in python3 and I want to obtain a stacked barchart plot, showing three different variables on 5 different columns.
My code works fine if I do not add the 'bottom parameter' in plt.bar (but I need to, in order for the stacks to appear in the correct order):

import numpy as np
import matplotlib as plt

columns=['a','b','c','d','e']
pos = np.arange(5)
var_one=[40348,53544,144895,34778,14322,53546,33623,76290,53546]
var_two=[15790,20409,87224,22085,6940,27099,17575,41862,27099]
var_three=[692,3254,6645,1237,469,872,569,3172,872]

plt.bar(pos,var_one,color='green',edgecolor='green')
plt.bar(pos,var_two,color='purple',edgecolor='purple')
plt.bar(pos,var_three,color='yellow',edgecolor='yellow')
plt.xticks(pos, columns)
plt.show()

However, once I add the bottom parameter in bar.plot (as shown below):

import numpy as np
import matplotlib as plt

columns=['a','b','c','d','e']
pos = np.arange(5)
var_one=[40348,53544,144895,34778,14322,53546,33623,76290,53546]
var_two=[15790,20409,87224,22085,6940,27099,17575,41862,27099]
var_three=[692,3254,6645,1237,469,872,569,3172,872]

plt.bar(pos,var_one,color='green',edgecolor='green')
plt.bar(pos,var_two,color='purple',edgecolor='purple',bottom=var_one)
plt.bar(pos,var_three,color='yellow',edgecolor='yellow',bottom=var_one+var_two)
plt.xticks(pos, columns)
plt.show()

the code triggers the error

ValueError: shape mismatch: objects cannot be broadcast to a single shape

How could I fix it?


texts: Modelling promotions for demand forecasting

texts:
I am trying to develop a model to predict future demand for a product. Now, there are always some promotional events that affect the sales. I am trying to solve this problem using dummy variables. Here's how:

Supposing that the firm runs 7 promotional events on a particular product. So, I construct 7 dummy variables, that are boolean. For supposing that a for a particular week, promotion 3 was running. So, my training data-point becomes [0,0,1,0,0,0,0] and the corresponding sales. I construct a linear regression model for promotions in this way.

Now, here is my problem. When we model seasonality using this method, we construct a base linear model, after deseasonalising the data, and  then use the two models to predict the final output. In case of promotions, how do I 'depromotionalise' the data?

Any tips in solving the problem are appreciated. Thanks!


texts: Calculation and Visualization of Correlation Matrix with Pandas

texts:
I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of area of activity (theater, cloth stores, food ...) and other data.

I tried to create a new data frame and insert a column with the income of all kinds of stores that belong to the same category, and the returning data frame has only the first column filled and the rest is full of NaN's.
The code that I tired:

corr = pd.DataFrame()
for at in activity:
    stores.loc[stores['Activity']==at]['income']

I want to do so, so I can use .corr() to gave the correlation matrix between the category of stores.

After that, I would like to know how I can plot the matrix values (-1 to 1, since I want to use Pearson's correlation) with matplolib.


texts: Is it valid to shuffle time-series data for a prediction task?

texts:
I have a time-series dataset that records some participants' daily features from wearable sensors and their daily mood status.

The goal is to use one day's daily features and predict the next day's mood status for participants with machine learning models such as linear regression.

I think cross-validation could be a good way for me to evaluate the performances. However, would shuffling the data randomly be fine?

Someone told me that because I am using a time-series dataset and I am trying to do a prediction task, shuffling the data randomly will cause some mix-up of future and past, which makes my model meaningless.
However, I think I can still use the strategy of randomly shuffling the dataset because the learning model is not a time-series model and, for each step, the model only learns from exactly 1 label value instead of a series of labels.


texts: Classification: How to manage data sets where one data row depends on another data row

texts: I am trying to classify  heading, image and image caption  of a webpage. I am preparing data by scraping selected URLs(around 1000) using XPath of DOM elements I need. Each data row in the CSV file contains tag name, x-coordinate, y-coordinate, text-size etc with three target labels heading, image, image caption but a text simply will not become image caption until unless it is under an image. Is there any way to create this dependency among rows.


texts:
Let's say I have a sentences that goes like this:

  Hi, how are the kids. I will be going to Los Angeles next Friday and
  will come back the following Monday.

If the date today is October 16 (Wednesday), then next Friday would be October 25th. The sentence later says that the person will come back the following Monday. We know they mean the Monday after next Friday so the 28th of October. If I use the dateutil library, then it would classify both Friday and Monday, but it won't take into account the next before Friday or that we are saying the following Monday.

I was thinking of creating a solution that will split the sentence up into words and then check what comes before actual dates. However, I'd like to capture not only this type of adverb of time, but others as well. For example, next day, this Thursday, 2 days from today, etc. How do I build a framework that will detect these types of date texts and then print out their respective dates?


texts: Best OCR approach on documents with different formats to find one specific information

texts:
Unfortunately, because of confidential data, I can't give a more specific explanation.

The Problem

So I've got a few documents that in general contain the same information but have different formats. In most cases, the value I am looking for is near a keyword on the document. The OCR itself is taken care of by the Google Cloud Vision API but what is the best approach to handle the different formats?

My idea

... was to train a classifier that detects what format I am dealing with and then picks the appropriate way of finding the target value, I implemented beforehand by hand. This is not handy nor scalable. So I am looking for some algorithm I tell e.g. where the target value is, what it looks like etc.

What is the best ML-approach for this problem or what are your ideas?

As an example of the type of data: Let's say I have receipts from 20 different supermarkets and I am looking to find the total cost, with the problem that every companies receipt looks different.


texts: What is the difference between GPT blocks and Transformer Decoder blocks?

texts:
I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same?
In the original Transformer model, Decoder blocks have two attention mechanisms: the first is pure Multi Head Self-Attention, the second is Self-Attention with respect to Encoder's output. In GPT there is no Encoder, therefore I assume its blocks only have one attention mechanism. That's the main difference I found.
At the same time, since GPT is used to generate language, its blocks must be masked, so that Self-Attention can only attend previous tokens. (Just like in Transformer Decoders.)
Is that it? Is there anything else to add to the difference between GPT (1,2,3,...) and the original Transformer?


texts: R: substituting only whole statements (not parts of them)

texts:
I have a data set DS.
I want to substitute the statements contained with others following a pattern-replacement schema like the following one:

I have used the followng function:

from_distincts_to_origins <- function(patternST, replacementST,originalFile){

  #---------------------------------------
  patternST<-paste0('\\b',patternST,'\\b')
  #---------------------------------------
  # we have to point back to the original data set
  file_origin_DF <- as.data.frame(originalFile)

  file_output <- file_origin_DF
  for(i in 1:length(patternST))   {
    where <- file_output[[1]]#vector
    file_output <- apply(file_output,
                         2,
                         function(x) sub(patternST[i],
                                         replacementST[i],
                                         where,
                                         #ignored.case=FALSE,
                                         perl=FALSE,
                                         fixed=FALSE, useBytes=FALSE)
    )

    file_output <- as.data.frame(file_output, stringsAsFactors =F) # in order to work
  }

  return(file_output)
}

What will happen is that R will start substituting all the "I like football" with "football". When it will pass to the second row it will look for "I like football, W Lazio" but it will only find "football, W Lazio", so those rows won't be substituted with "Lazio".

One possible solution is to order the Pattern such to have the longest strings at the top, but it would only reduce the substitution missings.

Is there another method in R to do that?


texts:
I am working on a project which processes big data (size ~3TB) every day. The first stage of our data pipeline copies the data from our MS-SQL Server to a host machine(linux server) using a tool called freebcp. More info on freebcp is here

Using this tool from the linux server, we run a set of stored procedures on SQL Server and export the data and transfer it in bulk. Recently I am observing that if the data is huge (~200GB), the data transfer is getting stalled after sometime. I ran a couple of commands (sp_who2, and dbcc inputbuffer(spid)) to monitor the execution of stored procedure on SQL server. We observe the CPU time and Disk IO utilized by this procedure. If this is not changing for a few minutes, we assume the job is stalled, and manually kill these stored procedures to continue our data processing tasks.

What are the probable reasons for this stalling of data copy?

Is there any better way to copy the data in bulk from SQL server to linux host? May be an alternative to freebcp. After this, we load this data into hadoop file system run our map reduce tasks.


texts: I try to understand my time series data. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. So I have a vector x with length 10.000. This I repeat with certain time intervals. Since the vectors are big, I want to create subsequences of them for my model later to predict. But now I do not understand if I have to see the whole time series data with a shape (number of samples, vector x length) or (number of samples*vector x length, 1) to move on.


texts: Tokenize text with both American and English words

texts:
I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English.

Consequently, I get 2 tokens for “organization” and “organisation” or “color” and “colour”.
Examples : https://en.oxforddictionaries.com/spelling/british-and-spelling

Do you know a (python) library converting “British English” to “American English” (or vis versa) ?

I would be happy to that ... (but I am french and my english is not soo good)

Thanks.


texts: How do I create a new column to an existing data frame which is the product of other columns?

texts:
I have data estimating what kind of food is used on a daily basis at school to feed kids.

Pizza   Apples  Lemonade  Lemon   Banana  Milk  Soup
 12        3       10       1      9      10     6
 14        5       8        0      13     11     7
 ....................................................

I want to create a new variable Fruits_total which is a sum of variables Apples, Lemon, and Banana, and add it to an existing data frame.

  df['Fruits_total'] = df.Apples + df.Lemon + df.Banana

However, it doesn't work. Where is my mistake?


texts: What is the relationship between MDP and RL?

texts:
What is the relationship between Markov Decision Processes and Reinforcement Learning?

Could we say RL and DP are two types of MDP?


texts:
A lot of websites on CNN for large datasets of images talk about starting with the pretrained model for 1.2 million images in 1000 categories available via AlexNet / Imagenet. These sites seem to imply that this dataset is freely available, but I'm having trouble actually getting access to it. For example, I tried going to https://github.com/deep-diver/AlexNet but couldn't get the code in alexnet.ipynb to run. Consider the following code:

for f in data_file.iterdir():
    data = pickle.loads(f.read_bytes(), encoding='bytes')
    if 'meta' in str(f):
        meta = data
    elif 'test' in str(f):
        test = data
    elif 'batch' in str(f.name):
        train_dic = merge(train_dic, data)

This seems to imply that there should be some data files which include the text strings 'meta', 'test', and/or 'batch', but I couldn't seem to find those files.

Where is the best place to get access to this dataset--ideally somewhere where there is good documentation as to how to download/use it?


texts:
I am looking for the best method to go from a sequence of events such as

time event
1 a
2 b
3 a
4 b
5 c
6 d
7 c
8 d
9 e

Where each letter corresponds to a certain event that occurs at a time. I want to reduce the number of events by aggregating frequently occurring events into a new event. A possible solution data set would look like,

1 a'
2 a'
3 a'
4 a'
5 b'
6 b'
7 b'
8 b'
9 e

where the clusters are created because they occur in a sequence following each other.

I was looking at the text mining algorithms in R with tm or RNA sequenceing
with edgeR. But I have no experience in this so I was hoping that someone can
shed me some light on a common approach for this type of problems.


texts:
I am referring Ehthem Alpaydin, 'Introduction To Machine Learning' book.

Under the chapter 'Decision Trees', I need help with understanding the concept of discriminant and how it is being used in this paragraph:

  Each $f_m(x)$ defines a discriminant in the d-dimensional input space dividing it into smaller regions that are further subdivided as we take a path from the root down. $f_m(.)$ is a simple function and when written down as a tree, a complex function is broken down into a series of simple decisions. Different decision tree methods assume different models for $f_m(.)$, and the model class defines the shape of the discriminant and the shape of regions. Each leaf node has an output label, which in the case of classification is the class code and in regression is a numeric value.


texts: Using python to identify common paths that users follow among different activities

texts:
I have a click-stream data of users regarding their behaviour in a platform. There are activities provided in a temporal order and users can go back and forth between these activities. Below is an example:

Here, there are 3 activities in a temporal order. Blue arrows represent one user and the green arrows represent another user. As you can see, these two users followed a different path.

In my scenario I have over 500 users , which are divided into 2 categories (let's Group A and Group B), and more than 50 activities. I want to identify the common patterns for Group A and Group B users and make a comparison among them (I am hoping to see patterns distinct to each user group).

I wonder if there are some useful (free) tools out there that I can take advantage of. I prefer Python but I could not find a python library for this. I am also looking for a tool (or python library) to visualise the paths that users follow. Any suggestions?


texts: If ReLU is so close to being linear, why does it perform much better than a linear function?

texts:
ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers.

I'm a beginner to deep learning research and methodologies but I've already seen several examples that claim that using ReLU as activation functions on a network will be superior than other common functions, including simply linear ones, of course.

ReLU has 1 point of "drastic" change, and is "otherwise linear".

My questions are:

Is there research or opinions on why this change significantly affects the performance of a network?
Is there something unique about $0$, the point of change? What if we move the point of change to $1$ instead for instance?

My intuition is that, as $0$ separates the line, it separates the sign function and that may have high affect on certain types of data that rely on positive/negative classification, perhaps.

How relevant is the dataset being learned? Do we see that ReLU is "better" on most examples of datasets?

Why ReLU is better than the other activation functions

The above question is relevant. Mine is more focused on the difference between ReLU and a linear function.


texts: How to use inverse_transform in MinMaxScaler for pred answer in a matrix

texts:
I am working on a data, for preding output, I used SVR by bellow code:
from sklearn.svm import SVR
regressor = SVR(kernel = 'linear')
regressor.fit(trainX,trainY)

from sklearn.metrics import r2_score
pred = regressor.predict(testX)
print(pred)

The answer is : [0.58439621 0.58439621 0.58439621 ... 0.81262134 0.81262134 0.81262134]. I'm trying to inverse the scaling to real amount.
I search it in StackOverflow and reach this: https://stackoverflow.com/questions/49330195/how-to-use-inverse-transform-in-minmaxscaler-for-a-column-in-a-matrix, I implement every 2 answers in my code, but I get error yet. Can anyone help me with this?
I write this from above source:
import sklearn
from sklearn.preprocessing import MinMaxScaler
scale=sklearn.preprocessing.MinMaxScaler()
scale.min_,scale.scale_=scaler.min_[0],scaler.scale_[0]
scale.inverse_transform(pred)

but, I got same error as:

Blockquote
Expected 2D array, got 1D array instead:
array=[0.58439621 0.58439621 0.58439621 ... 0.81262134 0.81262134 0.81262134].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Blockquote


texts: How to sum all price values having in count the quantity of each one

texts:
Using pandas, I would like to sum all of the price to get the count value for each one. The Dataframe looks like this:

   price  quantity
0     25         2
1     20         3
2     20        -1

My code:

import pandas as pd

df = pd.DataFrame({'price': [25, 20, 20], 'quantity': [2, 3, -1]})
print(df)

So, I would expect the output to be something like $(25*2)+(20*3)+(20*-1)$:

>>> 90


texts: Business-related metric for an optimization problem

texts:
We have an optimization problem on hand that is the following.
Let's say we have 10 different treatments that we might offer, all of them are equally good for us, but people have different propensity to take those treatments.
Before we've built the model the rule was to just randomly choose one of the treatments, offer it to a person, and if the person takes it - yaay, if not - so be it.
Based on this historic data we've created a binary classifier that predicts a person's propensity to agree to each treatment type and the idea is to offer a person a treatment that has a max propensity score from the model.
The model built shows great GINI and AUC numbers, but what we struggle with is a more business-related metric that we could use to evaluate the model's performance.
I'm sure some of you had been in this situation before. What metrics did you use?
Thank you!


texts: Gap leaderboard score and model scoring on a Competition

texts:
I'm working on a Veolia challenge on Ens Data Challenge ens-data (equivalent to Kaggle) the goal is to classify very rare binary events (the failure of a pipeline ) for 2014 and 2015 (y={2014,2015}). In input we have 5 features 3 categorical features (which I turned into dummy variable) and two continuous. The score is average AUC, $0.6*AUC_1 + 0.4*AUC_2$.

My problem is the following, when I compute each AUC (for 2014 and 2015) with a stratified kfold cross validation and I compute the average AUC I get roughly 0.88 and when I submit on the website I end up with 0.67, I guess there is a problem in my code.

Here is my code for choosing the best model for 2014:

Rk: to predict on the test set (2014,2015 unknown), I first predict with all 5 features, 2014. Then I add the prediction of 2014 to my feature to predict 2015

# Spot Check Algorithms
models = []
models.append(('LG', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('RF', RandomForestClassifier()))
models.append(('GBC', GradientBoostingClassifier()))
models.append(('SVM', SVC()))
# evaluate each model in turn
results = []
names = []

# stratifiedkfold is defined by default when there is an integer
scoring = 'roc_auc'
num_folds = 10

for name, model in models:
    cv_results = cross_validation.cross_val_score(model, X, Y, cv=num_folds, scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)


texts: How to use zero-inflated negative binomial regression for binary classification task?

texts:
I am working on a binary classification problem and I am currently employing XGBoost. The dataset consists of several variables which are count variables. The problem is, these features are highly skewed on counts. For example, these are the counts of each value of one of the variables:

0.0     98.175855
1.0      1.275902
2.0      0.348707
3.0      0.199535

I was suggested here to apply Zero-inflated Poisson or Zero-Inflated Negative Binomial regression models. I would like to know how these models work.

Are these used for feature transformation which will give me features which I can then feed to XGBoost or should I use them as classifiers?
My data is a mixture of count and continuous features. Should I employ these models only on count variables, or all variables.

If someone could suggest a good reference to understand these models intuitively, that would be very helpful.


texts: Comparing Efficiency of Machine Learning Algorithms

texts: I know that the algorithmic complexity of CNNs and other methods of deep learning can't be fully expressed in simple terms, like big oh complexity. That said, how would one go about comparing the efficiency/complexity of CNNs to standard machine learning methods (say decision trees, LDAs, Naive Bayes, etc.)? Deep learning methods are expensive, but how do we know how expensive they are comparatively?


texts: What's the fastest way to do a text analysis over user reviews on a website for a beginner?

texts:
I want to analyse user reviews for certain products as part of a research project without having to learn analytics from scratch, as my requirement is temporary. I need to do the following:

The user reviews are formatted in separate columns per user and indexed over multiple pages (almost 2000 reviews per product) - so first extract all the text intelligibly.
Analyse frequency of certain words and phrases(once per user), the overall connotation of the comment, most frequent words of all the reviews combined, etc.

Questions:

Where do I begin doing this?
How should a beginner approach this?
How long is it going to take?
Could there be an easy to use software or a bot where I can just give in my requirements and would just give me the results?


texts: Reproduce Figure 3.2 in Introduction to Statistical Learning

texts:
Has anyone reproduced Figure 3.2 in Introduction to Statistical Learning (James et al)?
https://trevorhastie.github.io/ISLR/ISLR%20Seventh%20Printing.pdf
They have a contour plot with circles.
Here is my code, which produces a contour plot with ellipses. Some of the numbers appear close, but some others are very different.
library(tidyverse)

# dataset can be found here: https://trevorhastie.github.io/ISLR/data.html
advert <- read_csv('Advertising.csv')

calc_ssr <- function(b0, b1){
  r <- advert$sales - (b0 + b1 * advert$TV)
  sum(r^2)
}

b0_range <- seq(5, 9, length.out = 25)
b1_range <- seq(0.03, 0.065, length.out = 25)
params <- crossing(b0 = b0_range, b1 = b1_range)
params$ssr <- map2_dbl(params$b0, params$b1, calc_ssr)

params %>%
  ggplot(aes(x = b0, y = b1, z = ssr)) +
  geom_contour_filled(breaks = c(2100, 2150, 2200, 2300, 2500, 3000))
```


texts:
I just wrote an MLP in Python. After having trained it, I pass in some test data to see the result, and I get an array of decimal numbers at the output, rather than the desired binary output.

For example, I'll get [0.43, 0.56, 0.1, 0.8], rather than [0, 1, 0,1].

What can I do to achieve this?

Thanks

Liam


texts: Training AI to do repetive retouching job

texts:
I'm new to AI, so bear with me... What would be the easiest way to do this using AI? Where do I start? What technologies are already available?

Sample data:
50.000 before images ex. there's a red ball that needs to be removed
50.000 after images the red ball is removed out of the image (by a human in photoshop)

Feed the data
AI learning magic

Feed the AI Input:
1 other before image featuring the red ball
Output expected:
1 image where the red ball is removed from the image.


texts: Does the training set of one topic will be useful to predicate the sentiment for any other topic?

texts:
I am working on Twitter sentiment analysis. Now, I should prepare a training data set that fits for any kind of Twitter data that predicts the sentiment of a tweet as pos, neg, and neu.

From googling, I found an air-line tweets training data set with needed labels (pos, neg, neu, and sentiment). When I checked the sentiment of tweets with that training data set, it gave me mixed results. I found that some positive tweets are given as negative and negative as positive.

Is the method that I am following correct or not?

Please suggest your views for preparing a training data set that can predict any kind of the twitter data.


texts: Can I do bagging method as improvement technique to decision tree in research?

texts: Bagging use decision tree as base classifier. I want to use bagging with decision tree(c4.5) as base as the method that improve decision tree(c4.5) in my research that solve problem overfitting. Is that possible because some lecturers said not right as bagging is other classifier not hybrid between two?


texts:
I have the following dataset with the dimensions:

Rows: 41174

Columns: 439316

The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.

However, I get an error saying that

  Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition

But I ran the same computeSVD on the following dataset and it ran perfectly fine.

Rows: 3502

Columns: 103301

In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?


texts: "Hand pose estimation in-the-wild" vs Normal hand pose estimation

texts:
So far I have seen several article that mention "Hand pose estimation in-the-wild" and just "Hand pose estimation." What is the difference between the two?
Thank you for your help


texts:
I am working on an image classification problem.
There are 876 images in the training and 600 in the test dataset.
It is a multi class classification for plants.

Since this is my first CNN problem, I started working with tensorflow and keras to build my model and then started using transfer learning to improve my performance.
The best model which I built so far using Keras is getting me a score of 0.68 ( since the test dataset has no labels, I have to upload my results over and they generate the score using log-loss).

Also, I tried using Fastai and Pytorch which significantly improved my single model score to 0.46.

I have used Resnet50 image dataset for both cases but i cannot experiment using Resnet101 or increasing the batch size since my gpu goes out of memory( also tried using kaggle and google colab).

Is their any way which i can try to further improve the performance?
I am basically stuck now since i can't experiment by increasing either batch size or using a better model like Resnet101 etc.

I have uploaded both codes at : https://github.com/akshatshreemali/ImageClassification


texts: What loss function to use for imbalanced classes (using PyTorch)?

texts:
I have a dataset with 3 classes with the following items:

Class 1: 900 elements
Class 2: 15000 elements
Class 3: 800 elements

I need to predict class 1 and class 3, which signal important deviations from the norm. Class 2 is the default “normal” case which I don’t care about.

What kind of loss function would I use here? I was thinking of using CrossEntropyLoss, but since there is a class imbalance, this would need to be weighted I suppose? How does that work in practice? Like this (using PyTorch)?

summed = 900 + 15000 + 800
weight = torch.tensor([900, 15000, 800]) / summed
crit = nn.CrossEntropyLoss(weight=weight)

Or should the weight be inverted? i.e. 1 / weight?

Is this the right approach to begin with or are there other / better methods I could use?

Thanks


texts: In a residuals vs fitted plot, how do I interpret a homoscedastic variance that is not randomly distributed above/below the line?

texts:
I'm learning linear regression, and I ran a step function for linear regression and checked out the residuals vs fitted plot for the final equation. The residual looks homoscedastic but it's not randomly distributed above and below the line. I'm not sure how to interpret that. The residuals vs leverage plot doesn't indicate I have any points with a high cook's distance. My R-squared value is pretty awful...here's the summary output:

Residual standard error: 2.337 on 58588 degrees of freedom

Multiple R-squared:  0.3692,    Adjusted R-squared:  0.3671

F-statistic: 174.1 on 197 and 58588 DF,  p-value: < 2.2e-16

But I don't quite understand what this residuals vs fitted plot means. And what my next step in developing a better model would be.


texts:
I work for a highly-regulated entity, so I have to obfuscate what I'm working on; I'll provide the following as examples on what I'm doing.

I am training an XGB model for NLP comments about breeds of dogs (Tensorflow, sklearn). I have a dataset about dogs that I'm using to train the algo and cross-validation testing is promising.

However, my test data is not just about dogs, it's about animals in general. So that data can be about dogs, cats, elephants, whatever. Right now, the XGB model is giving all of the animals it does not recognize a default value (a category). How can I make my XGB just say "I don't know" instead? Is there a way to assign a default value to test values that don't make sense in any category?

BTW, I realize that another possibility is to create a model to classify the test data but I want to see if there is a way to solve this within the XGB model.


texts: How to create a Tensorflow dataset for inferencing (not training) for fashion_mnist-trained model from an arbitrary input JPG/PNG image

texts:
OK so what I am trying to do is make a inference.py TensorFlow script that will load a saved model that has already been trained, take an input .jpg image, perform inferencing on it (on the existing model) and then print the prediction result to the screen.

Here is the train.py script that I ran to build my model in the first place:

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

print(tf.__version__)

fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

train_images.shape

len(train_labels)

plt.figure()
plt.imshow(train_images[0])
plt.colorbar()
plt.grid(False)
# plt.show()

train_images = train_images / 255.0

test_images = test_images / 255.0

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[train_labels[i]])
# plt.show()

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=10)

test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)

print('\nTest accuracy:', test_acc)

model.summary()

# Save the weights
model.save_weights('./checkpoints/my_checkpoint')

This works and produces a checkpoint file as expected (and is based on Tensorflow docs).  OK now for the harder part - I want to take an arbitrary .JPG or .PNG input file and then run inferencing on it.  So far I have created inference.py that looks like this:

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

# Restore the weights
model.load_weights('./checkpoints/my_checkpoint')

probability_model = tf.keras.Sequential([model,
                                         tf.keras.layers.Softmax()])

### How do I make a dataset to pass in here from a .jpg?

predictions = probability_model.predict(  ...  )

So how do I actually create a dataset based on a single image and pass it into the predict() function?  Note that this image could have any size/resolution and will not have any transforms applied to it.

Any help with this would be greatly appreciated!


texts: Interpretation of Loss and validation Loss in Keras


texts: Orange reading in new features from Tab-Delimited File

texts:
In my tab delimited file I have 65 features and 1 target variable that are numerical and continuous. My data is supposed to be used for regression models. There should be 65 features and 1 target variable in the columns when I upload the file to orange.

However,there are actually 68 features, 2 new features called "Feature 1" and "Feature 2" show up that shouldn't be there. Both of them are categorical. Why are these two new features showing up? I counted the amount of variables I put and the amount of domain and attribute types corresponding to each variable. They both add up to 66 so I don't know where the extra features are coming from.


texts: How to calculate pedigree function in diabetes prediction?

texts:
I am developing a model for diabetes prediction using this dataset using Logistic Regression.

I have completed the model and my input variables are -

Pregnancies, Glucose, blood pressure, BMI, DiabetesPedigreeFunction etc.

The model gives an accuracy of 78% which is quite good for me.

Now when I predict new values which are not in the dataset I have all the required inputs except Pedigree function.

I searched for it and I understand what it is but how can I calculate that?


texts: How should labeled data from multiple annotators be prepared for ML text classification?

texts:
My specific question is how NLP data from multiple human annotators should be aggregated - though general advice related to the question title is appreciated. One critical step that I've seen in research is to assess inter-annotator agreement by Cohen's kappa or some other suitable metric; I've also found research reporting values for various datasets (e.g. here), which is helpful for baselining.

How many annotators should work on each data point depends on time, personnel, and data size requirements/constraints, among other factors (I may ask a followup question for how to find optimal n). However, once n annotators have finished a dataset, how should those n datasets be unified for a "ground truth"? A couple approaches that I have seen used or seem reasonable to me:

Take averages over all annotators. Classification problems are sometimes hard to restate as graduated ones, although that seems necessary if an average is to be taken.
Express some level of uncertainty in the data for controversial labels, or even omit them from training and evaluation.
Add an arbitration step to unify or discard controversial labels. I am not sure this would be worth the annotators' time.
Choose some "principal annotator(s)" (possibly determined by IAA scores) who get the final word in conflicts.

Guidance/references for the above and any other steps I can take to make a high quality dataset are much appreciated. I am mostly interested in efficiently removing individual annotator bias even when n is low.


texts: Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?

texts: In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that enables to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?


texts: What is the best way to calculate score based on input features?

texts:
Lets say we have certain products. We also have certain input features regarding these products like inventory, sales, price, cost etc. based on these features we want to score these parts. The score should reflect whether it is in shortage or not. Shortage is when there is high demand (Sales) and low inventory/supply. Score of 100 means it is in shortage. Score of 1 means it is not in shortage.
Right now i use a random forest model and use the probability from the model as the score. However this leads to a lot of products having score between 0-10 and 90-100. Very few products have score between 10-90.
Is there a better way to do this so that the scores are more distributed?


texts: URL classification problem with small training set

texts:
I have a set of a few thousand organisations (org) with some attributes (name, location, and type) and I want to identify the official URL for each organisation from a set of 10 URLs retrieved from Google.
The features to characterise org/URL pairs may include a few different string-matching scores, URL length, top-level domain, etc. Given the project constraints, I can generate a training/test set of correct pairs up to 100 or 150.
This can be seen as a binary classification problem between an organisation and a set of URL (true=official website). Another approach could be learning to rank, where the algorithm ranks the URLs and I pick the top one.
What algorithm/method would you recommend for this scenario, considering the small training set?


texts: Generalization between 2 datasets of same domain

texts:
Suppose that I'm have 2 different datasets of same domain and trained a model with d1 dataset. Can we generalize the model to predict d2 dataset ? Is it possible or not. For example consider Dataset D1, Dataset D2 with images of cats and dogs and model is trained on D1. Then can we use the model to predict D2.
Any Kind of reference is helpful
Thanks in Advance


texts: outlier detection in time serie without using windows

texts: I would like to know if it's possible to detect outliers in a time-serie with an outlier score computed given the whole dataset and not given windows


texts: From logistic regression to XGBoost - selecting features to run the model with

texts:
I have been asked to look at XGBoost (as implemented in R, and with a maximum of around 50 features) as an alternative to an already existing but not developed by me logistic regression model created from a very large set of credit risk data containing a few thousand predictors.
The documentation surrounding the logistic regression is very well prepared, and as such track has been kept of the reasons for exclusion of each variable. Among those are:

automated data audit (through internal tool) - i.e. detected excessive number of missings, or incredibly low variance, etc.;
lack of monotonic trend - for u-shaped variables after attempts at coarse classing;
high correlation (>70%) - on raw level or after binning;
low GINI / Information Value - on raw level or after binning;
low representativeness - assessed through population stability index, PSI;
business logic / expert judgement.

A huge number of the variables are derived (incl. aggregates like min / max / avg of the standard deviation of other predictors) and some have been deemed too synthetic for inclusion. We have decided to not use those in XGBoost either.
The regression was initially ran with 44 predictors (the output of a stepwise procedure), whereas the final approved model includes only 10.
Because I am rather new to XGBoost, I was wondering whether the feature selection process differs substantially from what has already been done in preparation for the logistic regression, and what some rules / good practices would be.
Based on what I have been reading, perfect correlation and missing values are both automatically handled in XGBoost. I suspect monotonicity of trend should not be a concern (as the focus, unlike in regression, is on non-linear relations), hence binning is likely out; otherwise I am a bit unsure about handling of u-shaped variables. Since GINI is used in deciding on the best split in decision trees in general under the CART (“Classification and Regression Trees”) approach, maybe this is one criterion that it is worth keeping.
I have been entertaining the idea of perusing our internal automated data audit tool, removing std aggregates (too synthetic as per above), removing low GINI and low PSI variables, potentially treating for very high (95+%) correlation, and then applying lasso / elastic net and taking it from there. I am aware that Boruta is relevant here but as of now still have no solid opinion on it.


texts:
I build personal recomendation system for choosing games. In website on main page on special place there is collection of personal games recomendation. And after AB test(between 2 recommend system)  I don't understand, what events I should collect. Only events after click on recomend icon or all events(recommend events plus events without choosing recommend game-user can choose game on other places such as finder)?
For example, one of the metric is sum payment per user per game. Should I collect payments only on recommend games, which was clicked, or all payments? As I understand, in theory, recomend system can give negative effect for site(user see advises and not to click for them).  And If I collect only recommend payments, I can skip watching negative effect. On other hand, I build  recommend system, and want to see metrics only for players, who click on advice icon.
What way is more correct?


texts: Connection between prob output LogisticReg/SVM and ROC

texts:
I have the following ROC generated using LPOCV and Logistic regression or SVM (l2 norm). Now, let's say I have a test set containing 10 patients and I get that the probabilities of those patients to be sick range from 0-100% (of course). What I want is to define a point in the ROC curve (i.e. have a sensitivity specificity pair) which will give me a threshold for the probability used to classify a patient as 0 or 1 (or the other way around, choose a probability, say 60%, and use that probability to calculate the sensitivity and specificity given by my ROC).

Thanks


texts: What is the benefit of using Max pooling in convnets as opposed to just using convolution layers? (from Francois Chollet's Deep Learning with Python)

texts:
I am reading Francois Chollet's Deep learning with python, and I came across a section about max pooling that's really giving me trouble.

I simply don't understand what he means when he talks about "What's wrong with this setup?". How does removing the max pooling layers "reduce" the amount of the initial image that we're looking at?


texts: What is the difference between normalization and re-scaling?

texts: This site does not describe the nature of the normalization tag. How does it differ from re-scaling? Many authors use the two terms interchangeably. I can not understand normalization's operational meaning.


texts: Is there an algorithm for categorizing unlabeled samples into K classes?

texts:
I am not sure if this would be considered unsupervised, or semi-supervised learning.  I am looking for an algorithm that will take N input arrays of features, and then cluster samples(not features) into a user-specified amount of classes.

For example, let's say we had data that we knew were images of circles, squares, and triangles.  In this case, I would be able to specify 3 classes, and the desired outcome would be that it would separate out the circles, squares, and triangles.

Also, if there is such algorithms, are they any good?  Thanks.


texts: Finding predictions using biglm without finding errors

texts:
I'm using the biglm R package for linear regression.

click/impression is the output required. But the test data does not contain click and impression.
The predict function of biglm gives the error

  Error in eval(expr, envir, enclos) : object 'click' not found

I assume this is because predict tries to compute the standard errors also.

Is there a method to just obtain the predictions ? I tried assigning values to se.fit and type attributes, but I get the same error.


texts:
I am having trouble constructing a stem and leaf plot in "R"

The data is: 1717, 1719, 1645, 3739, 3024 (This is only a subset of a larger sample but it is sufficient to show my problem)

The output I get is:

The decimal point is 3 digit(s) to the right of the |

1 | 677

2 |

2 |

3 | 0

3 | 7

This does not reconstruct my data as the numbers would be 1600, 1700...
It seems my numbers have been rounded.
I want to make the stem to be one digit as the thousands number and the leaves to be 3 digits to represent the hundreds, tens and ones.
For ex) I want to 1717 to be 1|717

And I have tried scaling but that doesn't fix my problem, How do fix this issue?


texts:
I have millions of user ratings on about 2k products. I want to use Machine Learning to analyse these ratings and recommend products to users based on other users ratings of the same and different products.

i.e. user A likes 1,2 & 3, user B likes 1&2 so probably likes 3 based on A's ratings.

Is this possible to do through AWS ML? I'm looking at AWS ML as I'm a developer, not a data scientist and looking to keep this simple.

I've been playing with this all day and think I have figured it out, but just looking for some further advise / guidance.

Based on my experimenting so far, the following format of dataset is what AWS needs

CustomerID, Prod1Rating, Prod2Rating, Prod3Rating,.....

And then I would create a model for each product with the target on each being the row containing that products ratings. To make a recommendations engine from this I then just need to loop through every product for a user and ask each model the score.