texts:
I'm trying to use the Graphical Lasso algorithm (more specifically the R package glasso) to find an estimated graph representing the connections between a set of nodes by estimating a precision matrix. I have a feature matrix containing the values of multiple features for each of the nodes, and the sample covariance matrix obtained from the product between this matrix and its tranpose is used as the input for the glasso function, along with the l1 regularization coefficient $\lambda$.

However, when testing this with some simple examples I'm not getting the expected results. For example, when using the following feature matrix (5 nodes with 2 features each) as input:

$D=\begin{bmatrix} 1 & 10 \\ 2 & 10 \\ 3 & 10 \\ 4 & 10 \\ 5 & 10 \end{bmatrix},$

with $\lambda=0.01$, I get (approximately) the following inverse covariance matrix as output:

$P=\begin{bmatrix} 19.8 & -18 & -6.3 & 0 & 2.2 \\
-18 & 34.5 & -12.1 & -4.5 & 0 \\
-6.3 & -12.1 & 36.1 & -12.8 & -3.8 \\
0 & -4.5 & -12.8 & 34.8 & -15.9 \\
2.2 & 0 & -3.8 & -15.9 & 20.7 \end{bmatrix}.$

As far as I understand, a value of 0 in this precision matrix indicates that the two corresponding nodes are conditionally independent. As such, it makes sense for entries $p_{1,3}$ and $p_{2,4}$ (along with their symmetrical equivalents) to be zero. However, I don't understand why entry $p_{1,4}$ has a greater value than entry $p_{1,3}$, for example, considering that the feature values in the first node are closer to those of the third node than the fourth. Moreover, I would like have more entries with a value of zero, which means I want a sparser matrix. Therefore, I tried increasing the value of $\lambda$ to 0.03, which results in the following precision matrix:

$P=\begin{bmatrix} 8.3 & -5.8 & -3.2 & -0.6 & 0 \\
-5.8 & 11.9 & -3.6 & -2.5 & -0.1 \\
-3.2 & -3.6 & 13.2 & -3.3 & -2.4 \\
-0.6 & -2.5 & -3.3 & 12.6 & -4.7 \\
0 & -0.1 & -2.4 & -4.7 & 9.6 \end{bmatrix}.$

Now there are less entries with a value of zero (so the sparsity decreased rather than increasing), and while there was a general decrease in the value of every entry, the larger entries decreased at a higher rate, leading to a more evenly distributed matrix. This is not consistent with the feature selection I'm used to seeing in standard lasso regularization, and looks more like some sort of l2 norm regularization.

I feel like there must be something fundamental that I'm missing completely here. Is this method not supposed to be applied in this way?

The code I'm using:

D=matrix(c(1, 10, 2, 10, 3, 10, 4, 10, 5 , 10),nrow=5,ncol=2, byrow=TRUE)
m=dim(D)[2]
D = D - mean(D)
covar = (1/m)*D%*%t(D)
D = D/sqrt(max(covar))
covar = (1/m)*D%*%t(D)

a=glasso(s=covar,rho=0.01)


texts: Best practices for avoiding spurious artifacts in image cluster detection / color quantization

texts:
I want to know whether there are some common best practices for unsupervised detection of clusters / colors in images, in order to avoid spurious artifacts. To understand what I mean by 'spurious', here's an example.
Consider the following image:

A "reasonable" human would infer 5 statistically significant colors in this image (4+background).
However, it may not be that simple for a computer to reach the same conclusion. For example here's what I get when I apply k-means clustering with the elbow method:

The method detects 8 colors instead of the expected 5. Upon inspection of the clusters, one finds that k-means finds 2 different backgrounds:

I suspect that this has to do with a slight color variation in the background which nevertheless gains statistical significance due to the fact that the image is mostly background.
Secondly, another 2 of the detected clusters have to do with edge artifacts. Here's one of them:

I hypothesize that this occurs due to aliasing, but I am not certain.
After manually disregarding the spurious clusters, one finally gets the expected answer, which is 5 clusters. However, this manual pruning may not be feasible in images with a larger number clusters.
My question therefore is: Are there common or "standard" best practices to avoid the above (or other commonly encountered) artifacts in automatic image cluster / color detection?


texts:
I have a CSV file that, by all appearances, is totally normal except that each line ends with ,^M.  Not sure if that has anything to do with my issue or not, but I try reading in the file in R with the usual command

df <- read.csv('file.csv')

and then when I try to inspect it by typing df it prints out this result that I don't know how to interpret:

function (x, df1, df2, ncp, log = FALSE)
{
    if (missing(ncp))
        .External(C_df, x, df1, df2, log)
    else .External(C_dnf, x, df1, df2, ncp, log)
}
<bytecode: 0x000000000b083720>
<environment: namespace:stats>

Anybody know what's going on here?  I've read in CSV files before and had no problem.


texts: Loss function returns nan on time series dataset using tensorflow

texts:
This was the follow up question of Prediction on timeseries data using tensorflow.
I have an input and output of below format.
(X) = [[ 0  1  2]
       [ 1  2  3]]

  y   = [ 3  4 ]

Its a timeseries data. The task is to predict the next number. Basically input was crafted by the below snippet
 def split_sequence(arr,timesteps):
     arr_len = len(arr)
     X,y = [],[]
     for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component =  arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

     return np.array(X), np.array(y)

Now i want to train the model on the input and predict the next number. For instance x = [81,82,83] and the predicted output would be y = 84. In the previous problem, i had confronted the shape issue. Fortunately, i got a quick fill.
Now, when i am training the model,I observe my mse values are nan.
My implementation in keras is working but not in tensorflow. Most of the solutions in stackoverflow was pointing out to learning rate. Irrespective of giving different learning rate, my mse values are still nan. I have gone through many times reading my implementation but nothing turned up. Below is my tensorflow implementation.
 import tensorflow as tf
 import numpy as np

   # Create a dataset
  def split_sequence(arr,timesteps):
    arr_len = len(arr)
    X,y = [],[]
    for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component = arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

    return np.array(X), np.array(y)

 # data generator
def generate_batch(X,y,batch_size):
    m = X.shape[0]
    indexes = range(m)
    n_batches = m // batch_size
    for batch_index in np.array_split(indexes,n_batches):
       yield X[batch_index],y[batch_index]

  # parameters
   n_inputs = 3
   n_epochs = 400
   batch_size = 500
   learning_rate = 0.0005
   n_steps = 3
   input, output = split_sequence(range(10000),n_steps)

   # input and output and theta variable
   X = tf.placeholder(tf.float32,shape=(None,n_inputs),name='X')
   y = tf.placeholder(tf.float32,shape=(None),name='y')
   theta = tf.Variable(tf.random_uniform([n_steps,1],-1.0,1.0),name='theta')

   # Predictions and the residual
   y_predictions = tf.matmul(X,theta,name='predictions')
   error = y_predictions - y

   mse = tf.reduce_mean(tf.square(error),name='mse')

   # Training the algorithm
   optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    training_op = optimizer.minimize(mse)

   # initialize the variables
   init = tf.global_variables_initializer()

  with tf.Session() as session:
  # create  a session

     session.run(init)
     for epoch in range(n_epochs):
        for X_batch,y_batch in generate_batch(input,output,batch_size):
          #print(X_batch,y_batch)
          mse_val,_ =  session.run([mse,training_op],feed_dict={X:X_batch,y:y_batch})

          if epoch % 10 == 0:
             print('epoch',epoch,'MSE=',mse_val)

And the output i got was

epoch 0 MSE= nan
epoch 10 MSE= nan
epoch 20 MSE= nan

Any help is greatly appreciated. Thanks


texts: Should scaling be done for mixed data (categorical and numerical)?

texts:
My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations

NumFeature1,NumFeature2....NumFeature10,CatFeature1,CatFeature2,CatFeature3, Target

All Categorical are non-ordinal and each have the following categories:

CatFeature1: 0/1

CatFeature2: 0/1/2

CatFeature3: 0/1/2/3

It is a binary classification problem where we have to predict the probability for each class of the target class.

I have 3 Questions for above dataset:

Q1- For the categorical Feature, Should I use LabelEncoder() or OneHotEncoder() or df.get_dummies() or should I just combine custom label encoder with one-hot encoder?

Q2- Should scaling be done for Numerical features only or it should be done for all the features including categorical after encoding

Q3- What should be the best model to get the probability of the binary classification. So far, I have tried kNN, LogisticRegression, and RandomForestClassifier with predict_proba, but log_loss score was 0.301 the best.


texts:
I am trying to implement  audio recognition that can sport key words. I came across google speech dataset but it has a limited data set.

Is it possible to buy a new keyword data set ? Can you guys suggest few websites ?


texts:
Why is a coupling layer called so, in NICEand RealNVP density estimation?
Does it have anything to do with the coupling of probability distributions?
What is the intuitive goal of using these layers?


texts: Effect of discounting parameter on Language Model Perplexity

texts:
The general formula for absolute discounting for calculating language model probabilities subtracts a discounting parameter d from the count of the ngram before calculating the probabilities.

The perplexity of the model first decreases as d increases, and then later starts increasing again.

What is the reason behind this change? Why does the perplexity decrease first and then start increasing again?


texts: Need recommandations of using timing variables for forecasting sales

texts:
I have a data.table that contains many timing variables.

Date: it gives the date of Sales
Promo2(week, year): It describes the calendar week and the year when the store has started to participate in the Promo2 (reduction of prices).
CompetitionOpenSince(Month, Year): Gives the approximate year and
the month of the moment when the nearest competitor was opened.
PromoInterval: Describes the consecutive intervals Promo2 is launched,
naming the months when the promotion was launched again. For example.
"Feb, May, Aug, Nov" means that each round starts in February, May,
August, November of a given year for this store.

The Goal is to predict sales for next few week. I have other factor and binary variables such as the type of store(a b,c..), the level of Assortment, CompetitionDistance, number of Customers...

In my work I suggest some variable that uses timing variables above:

Compute the number of days between the day of sales and the first
day when the store started the promo2(Day-Promo2).
Compute the number of days between the day of Promo2 and the day
   when the competitor opened the promo2(Promo2-CompetitionOpenSince).

Note: Dont surprised when I wrote (Day-Promo2) and (Promo2-CompetitionOpenSince) because I transform Promo2 and CompetitionOpenSince to date type using some hypothesis to simplify the work.

What can you suggest more than that? I find it really hard to create new other useful explaining variables mostly the  PromoInterval variable!

Thank you for your suggestions!


texts:
I was recently asked this in a ds interview pertaining to product thinking -
If a product feature is being rolled out, but an a/b test is unable to be performed due to whatever reason, how can we measure the efficacy of this feature?
My response was along the lines of an exploratory standpoint in terms of comparing data pre and post rollout, but was curious as if there were better methods for this, thanks and thanks so much for even looking at my question!


texts: Why Imblearn pipeline gives very different results when used scaler and under sampling method swapped

texts:
I am using the Kaggle's credit card fraud detection dataset (https://www.kaggle.com/mlg-ulb/creditcardfraud)
In order to create a balanced datasets I was testing RandomUnderSampler() and NearMiss(). I am running a make_pipeline() from imblearn. I get very different results when I used RobustScaler() before vs after Neamiss() method. This drastic difference with LinearSVC(). Is this something wrong here, it is expected?
import pandas as pd
from sklearn import model_selection, preprocessing
from imblearn import under_sampling
from imblearn.pipeline import make_pipeline

credit = pd.read_csv() # Please use data from above link

X_train, X_test, y_train, y_test = model_selection.train_test_split(credit.drop('Class', 1), credit.Class, test_size = 0.2, random_state = 100)
# 1)
pipe = make_pipeline(under_sampling.NearMiss(), preprocessing.RobustScaler(), LinearSVC(dual = False))
score = model_selection.cross_val_score(pipe, X_train, y_train, cv = 3)
print(score, '\n', score.mean())

"""
 results are
[0.88720062 0.8111471  0.81310897]
 0.8371522304707543 """

# 2)

pipe = make_pipeline(preprocessing.RobustScaler(), under_sampling.NearMiss(), LinearSVC(dual = False))
score = model_selection.cross_val_score(pipe, X_train, y_train, cv = 3)
print(score, '\n', score.mean())

"""
results-
[0.33242044 0.46160531 0.35399221]
 0.38267265137357126
"""

# 3)

pipe = make_pipeline(under_sampling.RandomUnderSampler(), preprocessing.RobustScaler(), LinearSVC(dual = False))
score = model_selection.cross_val_score(pipe, X_train, y_train, cv = 3)
print(score, '\n', score.mean())

 """
results-
[0.97021686 0.96007795 0.9701638 ]
 0.966819533466512
"""
# 4)

pipe = make_pipeline(preprocessing.RobustScaler(), under_sampling.RandomUnderSampler(), LinearSVC(dual = False))
score = model_selection.cross_val_score(pipe, X_train, y_train, cv = 3)
print(score, '\n', score.mean())
"""
results-
[0.97234987 0.96139464 0.95404751]
 0.9625973369943192
"""


texts: Is it possible to have a validation error less than train error for a while followed by the reverse behaviour?

texts:
I am solving for a regression (using tensorflow's DNNRegressor) problem. When I sampled out 20% data (randomly) and divided it further into train-eval (90-10%, random but mutually exclusive), I observed the following graphs-

Here, blue line corresponds to eval set loss and orange line is for train loss. The total length of data is nearly 4.8 million rows, thus 20% being nearly 1 million data points. How to interpret this behavior?


texts: Should the weights for CrossEntropyLoss be exactly the inverse of the propotions of training data?

texts:
I have a classifier network which chooses one of three classifications, and uses cross entropy loss as the loss function.  If the proportions of training data are 100:10:5 for each classification, should I automatically set the weights to 1/100,1/10,1/5 ?
If not, what other issues are there to consider?


texts: How to model spatiotemporal data with ANN?

texts:
I have traffic jam intensity of many location in different times of eabch day. So I have kind of 3D data, where the intensities in each location is 2D like an image. Then there is time as a third dimension.

Now, I want to predict traffic jam intensity at a give location and time.
The intensity is from 0.0 to 10.0. high the intensity means high traffic jam.

Example. Whats the jam intensity of X latitude and Y longitude on 3:45 PM?

My question is how to feed that 3D data to my neural network?

NB: both the time and location contextual data. The intensity of the traffic jam is behavioral.


texts: Brown clustering, graph partitioning, agglomerative clustering - libraries/software

texts:
I need to do some experimenting with Brown clustering, graph partitioning, agglomerative clustering.

1) Are there Python/Matlab libraries for that? I know sklearn.cluster but it doesn't have algorithms I need.

2) Is it possible to install graphical interface for Cluto on Mac OS?

3) Overall, are there useful tutorials on using Cluto?

4) Other software for clustering that I could learn within a couple of hours?

Thanks


texts: Weighted samples in Tensorflow for convolutional neural networks

texts:
For my binary classification problem (A vs B),  each image in either class has its individual weight. This means, for example, if I have 10000 images for A, not all of the images are equally important. Weight of "0.9" would mean it is very important, and a weight of "0.01" means that it is not that important.
Can this information be communicated to the training in Tensorflow?
Thanks for the help.


texts: How to dealt with combine form of words in Natural Language processing?

texts:
This problem is related to natural language processing out of a specific language context.
Combination of words having their own self meaning, like North America, South Asia, Albert Einstein etc.
Some Problems I faced:

Tokenization, would separation these words too. (so it is meaningless)
N-Gramization, makes some garbage and non-meaningful combinations too plus bags of words will be also increased.
TFIDF, in that case, first name and last name probability will differ and increase feature words too. Means South and Africa both words having different probability scores and in the case of n-gramization, it will be generated irrelevant words probabilities too. If I ignore fewer probability words it will make some sense, but the question still left. how many n_gram words should be generated and will increase also processing time and corpus length and how to dealt with first, middle, and last name. because many irrelevant words make even high probabilities list, in general, it is.
Named Entity Recognition, mostly dealt with full naming corpora, means I need heavy corpus with proper labels to recognize these names, and every day new names generate, so it is also unhandled.

I want a solution to that problem, in an optimized manner to dealing this type of issue.
Note: I am working on language, don't have a rich corpus like the English language.


texts: Clustering product searches into products

texts:
I've been tasked with clustering searches from our website into the different searches for the same product. e.g. such different searches for the same product may be "product name" / "product name 1st release" / "product name 1995" / "product name 2004" / "prodct name" (mis-spelling intended) / "author 1995 part-of-product-name" and so on.

I seem to have a million or so searches like this for various products with various search strings. I don't have category or any other grouping data.

How can I cluster these search strings into their respective products / items with as minimum input from me as possible?


texts:
Is there someway to create a bubble plot with seaborn? I already know how to do it with Matplot lib, as in this tutorial.
# libraries
import matplotlib.pyplot as plt
import numpy as np

# create data
x = np.random.rand(40)
y = np.random.rand(40)
z = np.random.rand(40)

# use the scatter function
plt.scatter(x, y, s=z*1000, alpha=0.5)
plt.show()

I would like to replicate this with Seaborn. I am able to plot, but cannot control the size of marker.
import seaborn as sns
import numpy as np
%matplotlib inline
data=pd.DataFrame({"x":x, "y":y,"z":z})
sns.lmplot(x="x", y="y",data=data,  fit_reg=False)


texts: Keras Sequential model returns loss 'nan'

texts:
I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value.
I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.

This is the code:

def data_generator(batch_count, training_dataset, training_dataset_labels):
  while True:
    start_range = 0
    for batch in batch_count:
      end_range = (start_range + batch[1])
      batch_dataset = training_dataset[start_range:end_range]
      batch_labels = training_dataset_labels[start_range:end_range]
      start_range = end_range
      yield batch_dataset, batch_dataset

mlp = keras.models.Sequential()

# add input layer
mlp.add(
    keras.layers.Input(
        shape = (training_dataset.shape[1], )
    )
)
# add hidden layer
mlp.add(
    keras.layers.Dense(
        units=training_dataset.shape[1] + 10,
        input_shape = (training_dataset.shape[1] + 10,),
        kernel_initializer='random_uniform',
        bias_initializer='zeros',
        activation='relu')
    )
# add output layer
mlp.add(
    keras.layers.Dense(
        units=1,
        input_shape = (1, ),
        kernel_initializer='glorot_uniform',
        bias_initializer='zeros',
        activation='sigmoid')
    )

print('Compiling model...\n')

mlp.compile(
    optimizer='adam',
    loss=listnet_loss
)

mlp.summary() # print model settings

# Training
with tf.device('/GPU:0'):
  print('Start training')
  #mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)
  mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),
                    steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)

How can I do?


texts: What is the correct way to call Keras flow_from_directory() method?

texts:
In the following article there is an instruction that dataset needs to be divided into train, validation and test folders where the test folder should not contain the labeled subfolders. Instead it should only contain a single folder (i.e. Test_folder).

When I use the following code, I get the output message refering that no image were found.

Ver.1:
test_generator = test_datagen.flow_from_directory(
    "dataset\\test\\test_folder\\",
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=1,
    class_mode=None,
    shuffle=False,
    seed=10)

Output message: "Found 0 images belonging to 0 classes.".

Instead, if I use the same folder structure (dataset\test\class_a\test_1.jpg etc) as in the train and validation folders, everything seems to be OK and I manage to evaluate my model.

Ver.2:
test_generator = test_datagen.flow_from_directory(
    "dataset\\test\\",
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=32,
    class_mode='categorical',
    shuffle=False,
    seed=10)

Output message: "Found 1500 images belonging to 3 classes.".

I also tried the recommendation where 'classes' attribute is specified but still 0 images were found.

Ver.3:
test_generator = test_datagen.flow_from_directory(
    "dataset2\\test\\test_folder\\",
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=1,
    classes=['test'],
    class_mode=None,
    shuffle=False,
    seed=10)

Output message: Found 0 images belonging to 1 classes.

Thus, what is the correct way to call flow_from_directory() method and why am I getting the message that no files were found? Is my model not correctly evaluated when I use the Ver.2 solution?


texts: Does image's background matter for detector training (CNN)?

texts:
Does an image's background matter for detector/localisation in the training part (using CNN)?

For example, if I want to make a face detector, which one is better as training dataset?

Faces cropped dataset
Faces in a global scene dataset

Does it matter?


texts: Generating a sequence based on value in another column in Python

texts:
I have the following data frame:

I would like to add a column with value equal to 1 if flag is 0 and incrementally add 1 in the following rows until the next 0 is encountered (as given in the example below).

I have been able to generate the sequence, but the code is extremely slow, so is there a faster way to generate the sequence?


texts:
I'm building a Matrix Factorization model for MovieLens dataset with batch-wise training. Loss function for the batch:
$$
L_{batch} = 1/|B|\sum_{(u,i)\in{B}}(r_{ui} - \mu - b_u -  b_i - p_u^Tq_i)^2 + \lambda(||p_u||^2 + ||q_i||^2)
$$
$$
L_{batch} = (L_{base\_loss} + L_{reg\_loss})/|B|
$$
$$
L_{base\_loss} = \sum_{(u,i)\in{B}}(r_{ui} - \mu - b_u -  b_i - p_u^Tq_i)^2
$$
$$
L_{reg\_loss} = \sum_{(u,i)\in{B}}\lambda(||p_u||^2 + ||q_i||^2)
$$
where $r_{ui}$ is the observed rating, $\mu$ is the global average rating, $b_u$ and $b_i$ are the average deviations of user $u$ and item $i$ from the global average rating respectively, $p_u$ and $q_i$ are the learned user embeddings and movie embeddings respectively, $B$ is the batch.

What should I infer from regularization loss going up? Model is not able to capture the underlying information using the current embedding size, or the scale of regularization loss is too less compared to the base loss?


texts: Output size is too small for SpatialAveragePooling in Unet

texts:
Tried this to use Resnext as encoder in Unet from here, but keep getting  RuntimeError: Given input size: (4320x4x4). Calculated output size: (4320x-6x-6). Output size is too small at /opt/conda/conda-bld/pytorch_1525796793591/work/torch/lib/THCUNN/generic/SpatialAveragePooling.cu:63

(input image 128*128 batch size 32)

class UNetResNext(nn.Module):
    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d
    if encoder_depth == 34:
        self.encoder = resnext34()
        bottom_channel_nr = 512
    elif encoder_depth == 101:
        self.encoder = resnext101()
        bottom_channel_nr = 2048
    elif encoder_depth == 152:
        self.encoder = resnext152()
        bottom_channel_nr = 2048

    else:
        raise NotImplementedError('only 34, 101, 152 version of Resnext are implemented')

    self.pool = nn.MaxPool2d(2, 2)

    self.relu = nn.ReLU(inplace=True)

    self.conv1 = nn.Sequential(self.encoder.conv1,
                               self.encoder.bn1,
                               self.encoder.relu,
                               self.pool)

    self.conv2 = self.encoder.layer1
    self.conv3 = self.encoder.layer2
    self.conv4 = self.encoder.layer3
    self.conv5 = self.encoder.layer4
    self.center = DecoderCenter(bottom_channel_nr, num_filters * 8 *2, num_filters * 8, False)

    self.dec5 =  DecoderBlockV2(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8,   is_deconv)
    self.dec4 = DecoderBlockV2(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
    self.dec3 = DecoderBlockV2(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2, is_deconv)
    self.dec2 = DecoderBlockV2(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                               is_deconv)
    self.dec1 = DecoderBlockV2(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
    self.dec0 = ConvRelu(num_filters, num_filters)
    self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

def forward(self, x):
    conv1 = self.conv1(x)
    conv2 = self.conv2(conv1)
    conv3 = self.conv3(conv2)
    conv4 = self.conv4(conv3)
    conv5 = self.conv5(conv4)
    center = self.center(conv5)
    dec5 = self.dec5(torch.cat([center, conv5], 1))
    dec4 = self.dec4(torch.cat([dec5, conv4], 1))
    dec3 = self.dec3(torch.cat([dec4, conv3], 1))
    dec2 = self.dec2(torch.cat([dec3, conv2], 1))
    dec1 = self.dec1(dec2)
    dec0 = self.dec0(dec1)

    return self.final(F.dropout2d(dec0, p=self.dropout_2d))

class DecoderBlockV2(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels, is_deconv=True):
        super(DecoderBlockV2, self).__init__()
        self.in_channels = in_channels

        if is_deconv:
            """
                Paramaters for Deconvolution were chosen to avoid artifacts, following
                link https://distill.pub/2016/deconv-checkerboard/
            """

            self.block = nn.Sequential(
                ConvRelu(in_channels, middle_channels),
                nn.ConvTranspose2d(middle_channels, out_channels, kernel_size=4, stride=2,
                                   padding=1),
                nn.BatchNorm2d(out_channels), ##me added
                nn.ReLU(inplace=True)

            )
        else:
            self.block = nn.Sequential(
                nn.Upsample(scale_factor=2, mode='bilinear'),
                ConvRelu(in_channels, middle_channels),
                ConvRelu(middle_channels, out_channels),
            )

    def forward(self, x):
        return self.block(x)

I'm wondering how to properly set everything to start it working? Maybe input image size is to small - 128*128


texts: XG Boost Regression to model Log of Dependent Variable

texts: I'm working on a data set that has continuous dependent variable. I used XG Boost to model the dependent variable. However, when I transformed the dependent variable by applying Log transformation and then modeled it using XG Boost, the results were drastically improved. I am getting results close to 100% of actual on Test Data. Is there any explanation to this ?


texts: How to evaluate performance of a new feature in a model?

texts:
I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.

When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see an AUC improvement only in one of the models (LR scikit and Xgboost).

I also come across articles online that says AUC or F1-score aren't strict scoring rules. We could use log-loss metric but it's only applicable for logistic regression. but we can't use log-loss metric for Xgboost or RF or SVM right? So, is there any common metric which I can use to compare. Am I missing something here?

Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models?

Please note that I split the data into train and test and did 10 fold CV on train data.

So, how do I know that this newly added 7th feature is really helping in improving the model performance?

update based on answer

from statsmodels.stats.contingency_tables import mcnemar
# define contingency table
table = [[808,138],    # here I added confusion matrix of two models together (I mean based on TP in model 1 is added with TP in model 2 etc)
         [52, 416]]
# calculate mcnemar test
result = mcnemar(table, exact=True)
# summarize the finding
print('statistic=%.3f, p-value=%.3f' % (result.statistic, result.pvalue))
# interpret the p-value
alpha = 0.05
if result.pvalue > alpha:
    print('Same proportions of errors (fail to reject H0)')
else:
    print('Different proportions of errors (reject H0)')


texts: mask layer in speech emotion recognition

texts: I am trying to build a speech emotion recognition model and I have files of different length, thus my feature will have a variable length I learned that adding a mask layer could be a solution but I didn't understand its utility and how to use it


texts: Is there a pandoc for data manipulation?

texts:
You know how pandocs converts markdown to HTML and pdf etc.
Is there a pandocs for data manipulation? Like from SQL to pandas and SQL to dplyr etc?


texts: normalize identification values properly

texts:
I'm building a neural network to analyze a business' sales. I'm normalizing all input values to the range {0,1}.

I'm struggling with the day of the week column. Business days are identified by a number ranging {1-5} (1=Monday). Normalizing these values to the range {0,1} is straightforward, but results in a major bias in the final output.

The reason is that the full range of normalized values for the business day column is explored with every week worth of data, whereas other price-related column explore their full range of normalized values infrequently.

The business day column ends up being the largest contributor to the final output.

How can I normalize it to make its contribution more in tune with the rest of the inputs?


texts: How were auto encoders used to intialize deep neural networks?

texts:
In a document on deep learning about auto encoders, it is said that these networks were used back from 2006 to 2010 for deep neural networks initialization.

Can somebody explain how this was done?


texts: Question on What I would use for a Piece of Data

texts: I have 3 categories that we can call A, B, and C. In almost every case we B/A = C. B and A are given in the Dataset and C is what I am find an trying to use elsewhere. The higher number C is the better it is for the given individual who scored C. The only problem is that sometimes A is 0 and C doesn't have a true value. When A is 0 the individual has performed extremely well, so I don't want to remove the data. Should I just make it equal to the highest number of C that I have for others, or disregard C for that individual? What should I do with C for these cases?


texts: Time series clustering using dynamic time warping and agglomerative clustering

texts:
I'm new to data science and I'm currently working on a project to classify electricity consumption profiles.
This consists of electricity meter readings taken from sites on a half-hourly interval over a year (17520 readings in total). I am using Python to analyse the data.
I have settled on the current process of z-normalising the data, applying dynamic time warping on the entire dataset (using DTAIDistance) and then using agglomerative clustering as suggested by answers to similar questions on the site. I have produced the dendrogram below using 'complete' linkage using scipy.

I have some questions regarding the process.

What is the "best"* metric to determine the quality of clustering
(i.e. where do I set the cut off on my dendrogram?) and is there any
iterative method to determine the "optimal" cut off? (* I am aware
that "best" is quite subjective for this type of clustering)
Following up on 1. - say if I want to produce an elbow plot from my
DTW distance matrix. How would I go about doing this in Python?
How can I find out if I have selected the correct linkage method? I
have discounted Ward linkage as it apparently relies on an Euclidean
distance matrix and have plotted dendrogram using other linkage
methods, but I am unsure as to assess their suitability for my
dataset.
I have read that z-normalising data is required for most
applications of DTW - when is z-normalising not required for
DTW?
Are any other clustering methods I should consider?


texts: Image Super-resolution Connecting Subimages

texts:
I'm working with image super-resolution on terrain height data. Currently, I'm cutting the input data into smaller pieces (20 x 20 rather than 10800 x 10800).

After the upscale 20x20 -> 40x40, piecing the resulting images together causes artifacts along the borders of the smaller pieces. This is because the outputs of the model don't flow well together (we are using the output to run simulations, so it's important that they fit seamlessly).

Is there a recommended way of stitching these smaller images together? Or is there a better approach to this problem than using these smaller cut-down images?


texts: How to include user features in a recommender system?

texts: I'm novice in that matter but I was thinking about the formulation of a recommender system. Let's take the example of a movie recommendation system. We have a column dedicated to movies ID (or names), a matrix related to the rates the users gave to each movie, and a matrix with movie features (romance, drama, etc..)- joined a photo that shows this formulation. What about I would like to use users features to improve my recommendation? If I had information like age, profession, revenue of each user I would like to use it in my formulation. But if I include user features, this formulation is not Content-‐based neither Collaborative filtering anymore. Anyone knows what kind of formulation it can assume?


texts:
I have a case like below:

  col1
  {hjjlhllllh
  :59:kjkjkjdd
  :50:/kjhjhjhsa:ddd
  :70:jnjjh

  kjkjytdvdjjh:fhghghghg
 :52:jkjkt
 :50K:kkcknkknn
 :72:nbnbds

  mvmmbmnnm{hghghgsl
 :65A:jjhjhjh
 :50B:iut6565vdjbj

:kjkjkj

I want it like below:

  col1                                                            col2

 {hjjlhllllh
  :59:kjkjkjdd
  :50:/kjhjhjhsa:ddd
  :70:jnjjh
                                                               /kjhjhjhsa

 kjkjytdvdjjh:fhghghghg
 :52:jkjkt
 :50K:kkcknkknn
 :72:nbnbds
                                                               kkcknkknn

 mvmmbmnnm{hghghgsl
:65A:jjhjhjh
:50B:iut6565vdjbj:kjkjkj                                     iut6565vdjbj

Now, I am able to get col 2 value for the 1st row by using a logic that whatever appears between '50' & ':', extract it. The Same logic does not stand correct for the other 2 rows where in I have to get that appears between '50K' or '50B' & ':'. I want to extract irrespective whatever be the prefix be for '50' without hardcoding. Can anyone help me in this?


texts: On fitting a Poisson distribution to make sense of data

texts:
Hi guys I am working with a regular network which has the shape of a square grid and contains 100x100=10000 nodes. The edges (links) between these nodes simply follow the shape of a chess table: each node which is not placed in the corner or along the boundary has 4 connections only, all of them involving its nearest neighbors. Accordingly, the nodes on the boundary have 3 connections only, while the nodes in the four corners have 2.

Now, this brings me to my problem. If you plot such data you will figure out you have 4 nodes with links=2, 98*4 nodes with links=3, and 10000-(98*4)-4 with links=4. This translates into three tuples to plot: (x,y)=(2,4),(3,392),(4,9604). The resulting histogram is heavily skewed toward the right hand side:

My question is: what kind of distribution do you think would fit this dataset? I was thinking of a Poisson distribution (the x-axis values are discrete and not continuous) skewed to the right. I will appreciate any kind of help/guidance. Thank you!


texts: Some Confusions Regarding Variable Importance Extraction of Several Machine Learning Models

texts:
I'm trying to apply several machine learning algorithms in R using caret (decision trees, ensemble methods (bagging, boosting, random forests), logistic and penalized logistic regression models, svm and neural networks) to an imbalanced dataset (0: 90%, 1: 10%) with 1621 observations containing ordinal and nominal features. My primary concern is to extract the importance of each level in the mostly categorical dataset in the black box models but my sources of confusion are:

For two level categories, one hot encoding is unnecessary but how can I know which level is more important and which one of them affects negatively or positively the model's prediction? Logistic regression has a good way of dealing with that but what about neural networks for example? I'm trying to apply the Olden algorithm [1] in order to extract the importance and the way each level affects the latter model's prediction. Would one hot encoding for two level categories in this case be justified?

Should I use any oversampling methods here? If so, to which algorithm or set of algorithms should they be applied to? Would their use significantly impact the model's interpretability and explainability?

Can we compare the variable importance output of each model to one another and draw conclusions even if the methods used to compute these variable importance values are different (Permutation for random forests, Olden for neural networks, Shapley values for svm, etc.)?

[1] https://www.rdocumentation.org/packages/NeuralNetTools/versions/1.5.2/topics/olden


texts: How to Include Features that Apply to Specific Classes

texts:
I'm predicting hours that will be worked for building tasks. Due to the overall low sample size, I've stacked multiple related tasks together into a single model. (There may be 100 total samples in a single model, each task having 10 to 20 samples individually) An example would be - how long will it take a worker to complete each task associated with installing 2 different sizes of pipe in a hospital.
There are many tasks associated with installing a pipe -

Cutting the pipe
Welding the pipe
Bending the pipe
Riveting the pipe

We know from experience that the more bends a pipe has - the more difficult it is weld. But the difficulty of cutting and riveting are completely unrelated to the number of bends. Additionally there are multiple sizes of pipe in a single model, and the above tasks are completely unrelated between different sizes of pipe. An example of the data is:

| Task       | Pipe Size | Amount | Ratio of Bends to Welds | Predicted Hours |
|------------|-----------|--------|-------------------------|-----------------|
| Cut Pipe   | 3 inches  | 5      | NULL                    | 2               |
| Weld Pipe  | 3 inches  | 10     | 2                       | 4               |
| Bend Pipe  | 3 inches  | 20     | NULL                    | 8               |
| Rivet Pipe | 3 inches  | 10     | NULL                    | 2               |
|            |           |        |                         |                 |
| Cut Pipe   | 10 inches | 1      | NULL                    | 1               |
| Weld Pipe  | 10 inches | 2      | 5                       | 2               |
| Bend Pipe  | 10 inches | 10     | NULL                    | 15              |
| Rivet Pipe | 10 inches | 1      | NULL                    | 0.5             |

There are many different types of these "ratio" features within a single model, my current plan is to include them and null out the feature in all other tasks where it isn't relevant. It's the first time I've stacked this many classes together in a single model, and also the first time I've encountered features which are only applicable to some rows and not others. I'm currently using a random forest model. Is there anything conceptually wrong with doing this?


texts: Are there any activation functions which on inputting integer data will produce the output as integers?

texts: Idea is to create the model for ethereum mining which deals with only integer data.


texts: predict_classes() returning only 0 or 1 for multiclass image classification

texts:
I am trying to build a multi class image classifier but the only returns 0 or 1 .

Why is it not returning "Rock" , "Paper" , "Scissor" ?
and why only 0 and 1 but not 2?

CODE:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten
import numpy as np

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical', classes = ["Rock", "Paper" , "Scissor"])

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical',
    classes = ["Rock", "Paper" , "Scissor"]
    )

model = Sequential()
model.add(Flatten(input_shape = (28,28,3)))
model.add(Dense(128,activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'],)
filenames = validation_generator.filenames
nb_samples = len(filenames)
desired_batch_size = 1

model.fit_generator(training_generator,epochs=20,validation_data = validation_generator)
predict = model.predict_classes(validation_generator, batch_size = None)

print(predict)

Output:

[0 1 1 ... 1 0 1]


texts:
I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other?

Is "Lemmatization" always better than "Stemming"?


texts: Should we do Feature selection in parallel with feature engineering?

texts: I'm working with LightGBM on a large data set about 3M row and about 8 columns. When i started feature generation after every new feature i was measuring the RMSE of the model and if it was the same or slightly worse i was removing it so is that procedure right ? or i should do all the feature generation and then apply feature selection on them and why ?


texts:
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.cassandra.CassandraSQLContext
object Test {
  val sparkConf = new SparkConf(true).set("spark.cassandra.connection.host", <Cassandra Server IP>)
  val sc = new SparkContext(sparkConf)
  val cassandraSQLContext = new CassandraSQLContext(sc)
  val numberAsString = cassandraSQLContext.sql("select * from testing.test").first().getAs[Int]("number").toString()
  val testRDD = sc.parallelize(List(0, 0))
  val newRDD = testRDD.map { x => numberAsString }
}

This is the code that I have written in Spark. I was expecting it to work properly because i am evaluating the numberAsString value and then using it in the map function but it gives me task not serializable error. I am running the job in local mode.


texts: How do I use Cohen's Kappa as a metric in a keras model for continuous output?

texts:
I am using keras to train a CNN on a dataset. The output should be a value from 0 to 4 and my training y values are integers in that range. I want to create a custom metric that rounds and clips my conintuous output to an integer value in that range and calculates a cohen kappa value.

custom_metrics.py

def cohen_kappa_integerized(y_true, y_pred, num_classes=5, weights=None, metrics_collections=None, updates_collections=None, name=None):
    kappa, update_op = tensorflow.contrib.metrics.cohen_kappa(
        y_true,
        clip(rint(y_pred), 0, 4),
        num_classes,
        weights,
        metrics_collections,
        updates_collections,
        name
     )
     K.get_session().run(tensorflow.local_variables_initializer())
     with tensorflow.control_dependencies([update_op]):
         kappa = tensorflow.identity(kappa)
     return kappa

My code returns a value that does not match up to my accuracy metric very well so it "seems" to not be working. I pieced this together from a few sources and have never written a custom metric before, so a basic mistake is likely.

Here are the metrics from my run.

cohen kappa

mean squared error


texts: What are the key differences between a MLP with lagged features and a RNN

texts:
I've been working with MLP's for a while. Whenever I assumed that the past values of a feature might be useful for predicting the future values of Y, I would just create a new column in my data frame with Feature(t-1). This process would be repeated for further lags t-2,t-3...t-n.

Besides the obvious problem of the curse of dimensionality, I am worried that the MLP doesn't know how to weight those time lagged features that are now in a new separate column.

So in a nutshell:

Is the above approach wrong?
How does an RNN solve this?


texts:
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter Projects from 2009 -2016. I wanted to filter the data by year including only projects that launched between jan - jun and count the frequency of the categories. I would then perform multiple tests for each year with what kickStarter posted.
All of the tests give me 0 for the p-values. My goal is to check sampling of the data I got from kaggle by comparing it with kickstarter statistic. Any advice?
Kickstarter Projects
Kickstarted Stats
#Data was collected 6/19/2020
_ = {'Art' : (39310, 44.41), #(Number of projects, Success rate by percent)
    'Design' : (41510, 38.37),
    'Technology' : (42993, 20.65),
    'Film & Video' : (74760, 37.59),
    'Music' : (62545, 49.99),
    'Fashion' : (31840, 28.16),
    'Publishing' : (50215, 33.30),
    'Food' : (30106, 25.15),
    'Comics' : (16280, 59.24),
    'Photography' : (12448, 32.39),
    'Theater' : (12282, 60.01),
    'Crafts' : (1149, 25.20),
    'Journalism' : (5762, 22.81),
    'Dance' : (4265, 61.71),
    'Games' : (53029, 41.10)
}

tmp2 = [_[tmp][0] for tmp in sorted(_)]  # Frequency of main catagories for the year 2020. Six months jan - jun

for year in range(2010, 2017):           # Get the first 6 months of every year
        tmp1 = list(clean_df[(clean_df['launched'].dt.year == year) & (clean_df['launched'].dt.month <= 6)].main_category.value_counts().sort_index())
        print(year, chisquare(f_obs= tmp1, f_exp= tmp2))


texts:
I have written a simple python code that opens a csv files and then clusters the values of one column. There around 10k rows

This is my code

import pandas as pd
import numpy as np
import random as rd
import matplotlib.pyplot as plt

data = pd.read_csv('file.csv', encoding='unicode_escape')

data.head()

feature_names = ['numbers_col']
X = np.asarray(data[feature_names])

from sklearn.cluster import KMeans

labels = KMeans(5, random_state=10).fit_predict(X)
plt.scatter(X[:, 0], X[:, 0], c=labels,
            s=50, cmap='rainbow');

This is how the result looks like

The result doesn't look that good it looks all liner and the clusters are shown just like dots.

I am asking for some advice so my output would look more  towards what a normal K-Means output looks like: Example


texts: Stopping training if accuracy is below x at epoch y

texts:
In addition to the basic early stopping features in KERAS, I would like to stop training if the validation accuracy is still below 0.80 at epoch 40.

How can I do that?


texts: How can machine learning algorithms solve this particular problem?

texts:
Let's think of a case of an e-commerce website which lists products for sale. Now a person can come on a particular product page and decide to add it to the shopping cart or not. If we look at it as a classification problem then, specifications of the product can be its features(e.g. for a cell phone, camera resolution, screen size, price, warranty period) and for a particular product page view by a customer either he clicked on add to cart button or not can be the success criteria(classification label).

An observation is that for the same set of features, X for a product page view, very few customers click on add to cart button (y = 1). That means for the same X, there will be many Y=0 and very few Y=1.

My peer says that machine learning algorithms like XGBoost will be able to classify this. While I am not convinced of the fact that how will the algorithm be able to predict that for an X, Y will be 1 when in reality itself it has a very slight chance of being 1 as many customers will not add it to cart.

I know it will give me a probability score of y being 1 but that will be too low to make use of.


texts: Automatic labelling of text data based on predefined entities

texts:
I'm new to NLP.
I have a folder containing .txt files which are legal and specific documents. I want to label all these files based on four predefined labels. How can I do that automatically?


texts:
Why take softmax at all at the final layer for multi-class classification problems? For example softmax of the vector [1, .5]

Is [.621, .379]

I mean if we just took the straight ratio, it'd give me
[.667, .333] instead

Does that really make a difference?

Is it cause the vector can have negative numbers that we softmax things? What benefit do we get from making an odder way to give a ratio/probability to certain numbers as opposed to just taking a ratio of the numbers?


texts: What is exactly psi and segZ function in "segment" function in R

texts:
I am using "segment" package in R for fitting piecewise linear segments to a sample parabolic curve in R. I am not able to able to interpret the functions
"psi" and "segZ" from the R documentation. I have two variables X and Y .
If we consider X as dependent and Y as independent variable then,

  reg<- lm(Y~X, data = curved_data)

I am creating a parabola from the sample data given below-

  x<- c(-37, -36, -35, -34, -33, -32, -30, -28, -26, -25, -24, -23, -22, -21, -20, -19, -18, -17, -16, -15, -14, -12, -10, -8, -7, -5, -4, -2, -1, 0, 1, 2, 4, 5, 7, 8, 10, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 30, 32, 33, 34, 35, 36, 37)

y=4x^2

That is I am constructing a parabola with the help of the above equation in R.

So in this case y is independent and x is the dependent variable.

I want to have a piece wise linearity fit for this parabolala.

I have tried using two different codes of R i.e.

  seg_model<- segmented(reg, seg.Z = ~curved_data$X, psi = c(-35, 4, 30), control = seg.control(display = TRUE))

-35, 4, 30 are the 3 breakpoints that are chosen by me. Are they correctly chosen? How should they be chosen?

I get the error

    Error in segmented(reg, seg.Z = ~curved_data$X, psi = c(-35, 4, 30),  :
      psi' should be a list with more than one covariate inseg.Z'

What does this error means?

Also I have tried using code by writing psi = NA

I have got the below error

    Error in segmented(reg, seg.Z = ~curved_data$X, psi = NA, control = seg.control(display = TRUE)) :
      psi' should be a list with more than one covariate inseg.Z'

I am not able to understand psi and segZ from the R documentation. Please can I get an elaborate functioning of these two functions?
What is the relation between psi and segZ? What should I do to correct this error? Does number of points in vector of psi means number of breakpoints OR what is it?


texts: Hyperparameter tuning of neural networks using Bayesian Optimization

texts: One of the assumptions for finding good hyperparameters using Bayesian optimization (GP) is that the unknown function is smooth. Is this assumption valid for neural networks or at least for most of the neural networks? Can we find any reference?


texts: Why is Reward Engineering considered "bad practice" in RL?

texts:
Reward engineering is an important part of supervised learning:

  Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature engineering. — Andrew Ng

However tweaking the reward function by hand is generally frowned upon in RL. I'm not sure I understand why.

One argument is that we generally don't know, a priori, what will be the best solution to an RL problem. So by tweaking the reward function, we may bias the agent towards what we think is the best approach, while it is actually sub-optimal to solve the original problem. This is different from supervised learning, where we know from the start what we want to optimize.

Another argument would be that it's conceptually better to consider the problem as a black box, as the goal is to develop a solution as general as possible. However this argument could also be made for supervised learning!

Am I missing anything?


texts: How to deal with discrete variables in Multivariable Time Series forecasting?

texts:
I am tackling this time series forecasting problem to basically predict number of sales in the future

training dataset looks like this:

id   date         store   item    sales
 0. 2018-01-01.     1.      1       13
 1  2018-01-02      1       1       12
 2  2018-01-03      1       1       16
 3  2018-01-04      1       1       18

Use this to predict sales on these days:

id     date     store    item
 0   2018-01-01   1       1
 1   2018-01-02   1       1
 2   2018-01-03   1       1
 3   2018-01-04   1       1

I am not sure how to incorporate store and item variables into the model. Some idea I came up so far is:

Use embedding to represent categorical variables to continuous vectors
Feed it into NN model to learn relationships between these variables and sales
create univariate time series with sales and add lags, feed through LSTM to learn
somehow combine these two for prediction

Any thoughts would be super appreciated.


texts: How to choose the features for an algorithm from the given attached screenshot?

texts:
How to choose the features from the given attached heat map & correlation factor for the classification algorithm?

I have 6 different features i.e., ac233fc01403, ac233fc02eaa, ac233fc015f6, ac233fc044aa, ac233fc0455d, ac233fc0451f. From the 6 different features, I should classify in which segment(i.e.,class) the test data is?

How to choose from the given plot, which are the feature has to be considered for the algorithm to achieve the better efficiency?


texts: Libraries for Bayesian network inference with continuous data

texts:
Is there any good libraries that allow me to:

Construct a Bayesian network manually
Specify the conditional probabilities with any continuous PDF, not just Guassian
Perform inference, either exact or approximate

I looked at the following libraries so far, none of them meet the 3 requirements:

pgmpy: only work on discrete distribution or linear Guassian distribution
bnlearn: same as pgmpy
gRain: only discrete distribution
Huggin: only discrete distribution and Guassian
deal: no support for inference
abn: same as deal
libpgm: only discrete distribution and Guassian


texts: Reasons that LIME and SHAP might not agree with intuition

texts:
I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making on new, tabular data. WLOG, the explanations generated by both methods do not agree with user intuition.

For example, when leveraging the methods in a healthcare setting, they might list the presence of a comorbidity (a disease that frequently co-occurs with the outcome disease of interest) as a factor that decreases a patient's risk of an adverse event.

Intuitively, such behavior is incorrect. We shouldn't see that history of heart attacks lowers the risk of adverse events, for example. What are some reasons that we might see these inconsistencies?

Some of my ideas

Class label imbalance: tried balancing the dataset, did not solve the issue
Kernel width for LIME: working on tuning this, but cursorily, no benefit
Relationship to training data: for tabular data, both lime and shap require the training dataset as input to build the explainer class. If there are instances in which a feature such as history of heart attacks were associated with a no adverse event outcome, such instances would "confuse" the methods, so to speak. However, I'm not sure I have the intuition correct there.
Error in understanding on my part: there may be nuances in intuition here that I've missed. Specifically, I am trying to make sure I correctly understand the relationship between the generated explanations and the training dataset used to build them.


texts:
How can I approach a regression problem where the input data is not noisy but the target variable is noisy? Are there any regression algorithms that are robust to a noisy target variable?
Also, is it possible to de-noise the target variable somehow? If so, how?


texts: Keras single sample prediction returns different values

texts:
I'm using a pre-trained ResNet50 model in keras and am trying to see predictions for single samples

The issue is that model.predict() is returning different values, depending on when I input a single sample vs the same sample within a larger array.

My model outputs a 1x6 vector of probabilities

I run on a single sample:

Case A: model.predict(X_test[0:1])

and within an array of multiple samples:

Case B: model.predict(X_test[0:2])

The prediction for X_test[0] is

Case A: [9.999e-01, 4.228e-06, 8.278e-05, 1.121e-06, 1.061e-06, 2.958e-05]

Case B: [1.000e+00, 7.702e-13, 2.969e-13, 6.343e-11, 7.477e-14, 1.304e-10]

Why are the predictions different? (predictions here being the probabilities themselves, not argmax)

As requested, the main code is below:

from keras.models import load_model
from my_utils import load_dataset
import numpy as np

X_train, Y_train, X_test, Y_test, classes = load_dataset()

model = load_model('model.h5') # ResNet50 with pre-trained weights

model.predict(X_test[0:1]) # Case A

model.predict(X_test[0:2]) # Case B

The model architecture of ResNet50 can be found here; the one difference with the implementation I'm using is that I'm using 6 classes not 1000

Thanks!


texts: Minimization algorithm that can consider gradient close to solution

texts:
I want to minimize a function which has sharp gradients close to each local minimum. Due to process tolerances, I want to find solutions which meet some minimum criterion (e.g. lower than x), but have a shallow gradient near to the found minimum.

My question is: is this simply a case of me assigning a penalty factor to minima with sharp nearby gradients, or is there a class of algorithm that can handle this sort of constraint as part of its optimization routine?


texts: Classification of Colors using Machine Learning from an Array of Pixels

texts:
From a very high level of possible machine learning methodologies, what method of classification would be good to identify objects or situations based on the color sensor pixel array (i.e., 16x16 array) being fed into Python?

For example, if I want to identify certain fruits based on the shape and color detected in the array:

banana is curved and yellow
apple is circle and red
orange is circle and orange

If I am reading in a constant stream of data in this array form, are there any similar machine learning and classification applications that might be best suited for this type of application?


texts: Extracting structure and content from invoices

texts:
Lately, I have been largely inspired by this https://rossum.ai/, which is able to extract text from invoice documents.

Do you have any ideas on how this could be implemented? It's clear that they did a lot of research to reach this performance level, but in my case I am interested in the overall approach to such problems.

If I understand correctly, the first part of the pipeline is to extract different blocks from the document. In that case, is object detection the right approach to get bounding boxes around the blocks? I guess it might not be really good at extracting tabular data.

If not object detection, what is the correct way to tackle the problem?

Thanks.


texts: Optimization methods used in machine learning

texts: I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was wondering why more advanced methods don't seemed to be used, such as SQP algorithms or interior-point methods. Is it because training a neutral net is always a simple unconstrained optimization problem, and the above-mentioned methods would be unnecessary? Any insight would be great, thanks.


texts: How to evaluate the "importance" of a variable in a function

texts:
Let's say that we have
$$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$
$$k = constant \in ]0,1[$$
And  I need to show in some way that the variable $x$ is more important in  some metric that I don't know which one could be good. I thought about  analyze the partial derivatives of that functions, but I don't think  that is a good way, because one will only see some restricted path  through the surface.
Another  approach would be do monte carlo simulation in some domain of interest  (it exists, actually is a real world problem) and see that the variance  of the function value increases more varying x fixing the another  variables rather than the same simulation with $x$ fixed.
In some extent one can thinking in this function as a model, and I need to calculate the variable importance, it could be a way as well.
But I'm really  puzzled how to approach this problem.
Any metric or some approach is welcomed!
Thank you very much!


texts: Around how much difference between two classes can be accounted as an imbalanced dataset?

texts: I am building a binary classification model which has 17K values as class A and 10K values as class B. I want to know when a dataset can face the issues of "IMBALANCED Dataset" ?


texts: Choosing the size of the network for Neural Collaborative Filtering (NCF)?

texts:
I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system. After doing some hyperparameter tuning with various sizes for embedding and dense layers sizes, from 16 all the way to 2048, I realized less complex models tend to work better and usually do a better job at avoiding overfitting. This is not so surprising until now, I suppose.
I know it's generally very contextual and data-dependent. The less complex relationships in the data, the higher chance of overfitting in training the networks with large capacity. But I can't help but wonder, besides halving the size of dense networks, if there is any rule of thumb for the size of such networks. How do you usually go about choosing the size of the network for NCF models?


texts: What is considered best practice when importing data in a 'pipeline'?

texts:
Let's say I have some sort of process that takes data from a CSV that is located on my local machine. This process will eventually need to be scheduled via some cron job. I need to import that data obviously and if I'm writing an automated script there are two main solutions for this.

Create a user defined function that includes the import statement in the body.

    import_data_and_clean <- function(full_path){
         data = read_csv(path)
         ....
    }

Create a dataset_builder.R file that not only imports the data, but also cleans and prepared the data for analysis.

data = read_csv(path)
....

My question pertains to which of these solutions was the reccomended way to do this in R?


texts: Why don't we use Manhattan distance instead of euclidean distance in linear regression?

texts: When I am explaining concept of linear regression to one of my peers, I got stuck in answer this question. Why don`t we use Manhattan distance instead of euclidean distance in linear regression? Can anyone give intuition behind this?


texts: Measuring Difference Between Two Sets of Likert Values

texts:
I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets? I was considering a Pearson correlation, but after thinking about things more I don't believe that it is the most appropriate correlation metric.
My data looks similar to below:

Before   After
1         2
1     &nbsp   2
2     &nbsp   2
3     &nbsp   3
3     &nbsp   4
4     &nbsp   2

Result: XXX?

Thanks all


texts: What is "SwarmPackagePy.cso.cso at 0x187cf21e340"

texts:
def rosen(x):
"""The Rosenbrock function"""
return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1])**2.0)

from SwarmPackagePy import cso

cso(1,rosen,10,1000,2,10000,pa=0.25,nest=100)

Out[41]: <SwarmPackagePy.cso.cso at 0x187cf21e340>

How can I access outputs of cso function? There aren't docs on this.


texts: How can the performance of a neural network vary considerably without changing any parameters?

texts:
I am training a neural network with 1 sigmoid hidden layer and a linear output layer. The network simply approximates a cosine function. The weights are initiliazed according to Nguyen-Widrow initialization and the biases are initialized to 1. I am using MATLAB as a platform.

Running the network a number of times without changing any parameters, I am getting results (mean squared error) which range from 0.5 to 0.5*10^-6. I cannot understand how the results can even vary that much, I'd imagine there would at least be a narrower and more consistent window of errors.

What could be causing such a large variance?


texts: Extract editing history from Microsoft Word documents?

texts:
Is there a tool to computationally extract the editing history of a given Microsoft Word Document?

I have been using Apache Tika, but can only extract the last version of the text, and meta-information about the initial and last edit – even if edits are still being tracked via Word.


texts: Is MLlib compulsory to work with distributed data?

texts:
I am new to the field of machine learning and recently learned the basics and working out various algorithms in python using libraries such as pandas, numpy, matplotlib, scikitlearn , etc.
                    I started learning about working Bigdata by distributing it and using Apache Spark's library MLlib to load and apply algorithms on it. So is working with MLlib the only way on Spark or is there any other way to use pandas and other libraries on distributed data?


texts: Binary classification problem with imbalanced dataset, how to compare to random classifier

texts:
We have a very imbalanced dataset (2% of class 1). To the best of our knowledge, there is no baseline in the literature to the problem we want to solve - so we thought of comparing our performance to a random classifier. We evaluate our model as a combination of precision and recall - we vary the threshold at which data points are classified as 1 and compute the rolling threshold and recall. We could use F1-score as well.

What would be an acceptable way to define a random predictor that we can compare to our model such that the comparison is as fair as possible?


texts: How to modify the resnet 50 in Pytorch (pre-trained) to give Multiple Outputs for Multi-Label Classification

texts:
class ResNet(nn.Module):
def __init__(self):
    super(ResNet, self).__init__()
    resnet = models.resnet50(pretrained=True)
    modules = list(resnet.children())[:-1]
    self.resnet = nn.Sequential(*modules)
    self.fc=nn.Linear(2048,10)

def forward(self, x):

    x1 = self.resnet(x)
    x1=self.resnet(x)
    x1 = x1.view(x1.size(0), -1)
    x1=self.fc(x1)

    x2 = self.resnet(x)
    x2= x2.view(x2.size(0), -1)
    x2=self.fc(x2)

    return x1,x2


texts: How to show progress of sklearn.multioutput.MultiOutputRegressor and XGBRegressor?

texts:
Is it possible to show the training progress of the MultiOutputRegressor in sklearn? When a huge dataset is processed, my program runs a long time and I have no clue how long it will take. I have shortened my program to a minimal working example below.

import numpy as np
from sklearn.multioutput import MultiOutputRegressor
import xgboost as xgb

df = np.arange(50).reshape(10,5)
train = df[:8]
test = df[8:]
X_train = train[:,0:-2]
X_test  = test[:,0:-2]
y_train = train[:,-2:]
y_test  = test[:,-2:]

eval_set = [(X_test, y_test)]
multioutputregressor = MultiOutputRegressor(xgb.XGBRegressor(eval_set=eval_set, verbose_eval=True))
multioutputregressor.fit(X_train, y_train)
predictions = multioutputregressor.predict(X_test)
print(predictions)


texts: What do you call the ratio of positive to negative samples?

texts:
I am working with a binary classifier and I want to express the "balance" or "skewness" of the training data using a metric.

I want to reflect this ratio in a report, like this:

Accuracy: 80%
Recall: 78%
Precision: 62%
*The Ratio of Positive to Negative Samples*: 62%

I feel like there's probably a standard name for the metric expressing "The Ratio of Positive to Negative Samples".

My main question is: What is the name of this metric?

(This question assumes that there is a standard name for this metric.)

Extra Information:

Here are some example values for the metric and their interpretations:

1.0 = The sample (of training data) is balanced.
0.5 = There are twice as many negative samples as positive samples.
2.0 = There are twice as many positive samples as negative samples.

My secondary question is: Is there another metric that can be used to form similarly meaningful interpretations?


texts: Where can i download a benign PE dataset? or at least which website is the best candidate for crawling and downloading normal executables?

texts:
I'm planning to gather a benign dataset for my ML malware detection model
the problem I'm having is finding benign PE files, i just need a source that has a dataset of normal executables, i will scan them with VT and extract benign ones, but i cant find anything useful
if there is nothing out there, then at least what is the best website that has the potential to be useful for a PE downloader crawler? (meaning its easy to crawl and automatically download .exe files without running into problems)
also another problem of using a download website is Installers, considering most of their files are installer and i need to install the program first, is there  any good solution to this? is there any AutoIT script that somehow can install all types of installers ?
(I tried looking at surveys on using ML in malware detection like [1], but seems like non of the papers have released any useful benign dataset other than simple windows files which anyone can gather and is less than 10k, and very small amounts like 1000, i need to gather a large benign dataset, more than 50,000 benign files because my malware dataset is really large)
[1] https://www.sciencedirect.com/science/article/pii/S0167404818303808


texts:
From a behavioral study data was extracted. The study was about how people change their eating behavior, following visual cues. There were to groups of people: One was shown visual cues and then it was recorded what they chose to eat and the other group was just shown random stuff or nothing and then it was recorded was they chose to eat. This experiment was repeated with the same people a large number of occasions, at different times of day.
I have a dataset that precisely recordes personal information about each individual as well as their eating behaviour during each time the experiment was carried out.

The goal is to predict whether the visual cue will make someone eat the stuff presented in the cue or not. The problem is that it is unclear to me how to separate the influence of the treatment from the behavior that they might have shown anyway. I.e., suppose a person sees an image of a cake and then, when he is presented with a variety of different foods, eats cake. How can we know that it was actually the image that influenced him and that he did not wanted to eat cake anyway, so the image change actually nothing?

Thus I can't directly treat it as a binary classification and define a categorical feature, where I assign "1" if someone ate what was in the picture and "0" if he didn't, since by that way I may also identify those who wanted to eat what was in the cue before it was shown to them. How can I solve this problem?


texts: How do we define the noisy-MAX canonical aggregator for Bayesian networks?

texts:
In the book Probabilistic graphical models - principles and techniques, Daphne Koller and Nir Friedman introduce the noisy-OR canonical model for CPDs (in the independence of causal inference family of aggregators) and go on to say this can be extended to noisy-MAX. However the details on how to do this are omitted.

The OpenMarkov project (http://www.openmarkov.org/) notes talk about a noisy-MAX model but again the details on how this is implemented are absent.

The noisy-OR model takes in a number of binary variables $X_i$, for $i=1,2,...n$. It then transforms them to an intermediate set of binary random variables $Z_i$ such that $Z_i=1$ with probability $\lambda_i$ when $X_i=1$ and 0 otherwise. Adds in the binary variable $Z_0$, called the leak, which is 1 with probability $\lambda_0$ (and 0 otherwise). The resulting (binary) output is then the OR over all the $Z_j$, $j=0,1,2,...,n$.

It is unclear how one should extend this to MAX, since directly replacing OR with MAX will yield exactly the same as with OR. (A series of 1s and 0s OR'd is the same result as MAX'ing it.)

What is the methodology/algorithm/formula used for calculating the noisy-MAX?

Answers should also whether the inputs and output can be generalised to non-binary (or even numerical/continuous) types.


texts: How to find the optimal number of clusters for K-modes? : Python

texts: I have a categorical dataset (survey data on a likert-scale: never to extremely often). So, I need to cluster them. Therefore, I am planning to use K-modes. But, I am stuck at selecting the optimal number of clusters. I noticed that Elbow Method, Silhouette, and sqrt (n/2) are in use to determine the number of clusters for K-means. But could not find anything proper or K-modes. I am using Python and also not a clustering expert.


texts:
An example of what I'd like to do is identify the price of a product on a product page.

While I can train a CNN to identify prices, it's likely that it would recognise every instance of a price on a given page and end up picking up the prices of recommended products in addition to the price of the primary product.

Are there examples of visual neural architectures which take the context of the object into account? I'd imagine that would be something like a combination of an RNN and a CNN.

EDIT:
Maybe a simpler approach would be to find a way to apply multiple labels to the same bounding box?

The reason I say this is that it if there are 10 prices on the page, it would be easier for the network to identify them first as prices, and secondarily apply a label for the primary price


texts: To continue training an existing tensorflow model do I need to restore the session?

texts:
I'm trying to work out the best way to integrate with EC2 spot instances that can be started and stopped.

Do I need to store the tf.session or can I just do load_model('myfile.h5') and continue with mymodel.fit(....checkpoint=...) which saves the model every epoch back to myfile.h5?


texts: Spacy Text classification (Binary Classification)

texts:
I have a dataset of two folders. One of them contains the documents(text, pdfs) related to personal information (like name,email,address etc), the  other contains non-personal information.

I have to train a model using Spacy, based on these two folders. So, when we predict a given document, it should predict among these two folders.

I have tried writing many codes taking reference from Github, but nothing seem to be worked.

So, can anyone  give me a code sample to train a model based on the information given above and predict ?

I have done some hands on, on the below code

import spacy
from spacy import displacy
from spacy.util import minibatch, compounding

train_data = [("This has names, emails, addresses ", {'cats': {'POSITIVE': 1}} ), ("This has games, food, etc", {'cats': {'POSITIVE': 0}})]

nlp = spacy.load('en_core_web_sm')

if 'textcat' not in nlp.pipe_names:
    textcat = nlp.create_pipe("textcat")
    nlp.add_pipe(textcat, last=True)
else:
    textcat = nlp.get_pipe("textcat")

textcat.add_label('POSITIVE')

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']

n_iter = 1

with nlp.disable_pipes(*other_pipes):
    optimizer = nlp.begin_training()
    print("Training model...")
    for i in range(n_iter):
        losses = {}
        batches = minibatch(train_data, size=compounding(4,32,1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(texts, annotations, sgd=optimizer,
                      drop=0.2, losses=losses)

Here in the above code, I have trained the model using two simple sentences. I need to train on two folders, as mentioned in the question.
This code just says model has trained.
And also how can i save this model and test it for documents to predict ??


texts: I have a set of images of various products from different websites. I want to cluster the images based on the product shown in the image. How can I generate a suitable feature vector for an image for this purpose?? I just need to know how to generate a feature vector given an image. I tried NetVLAD, but it is very slow. I would like something that is fast and gives high accuracy for clustering in the scenario I have described. Please help me.


texts:
I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?

Edit: I am sorry, question may have been a bit misleading. What I meant by "unfamiliar environment" is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?


texts: Should I create thousands of models (one per user) or a single one?

texts:
I have application data for a mobile app and I have thousands of users:
Should I create one ML model for each user or a single ML model for the entire application? Here is a cartoonish version of my actual use-case.
Suppose each user can come up with their own color: it can be a typical color (red, blue, green, etc.), or it can be an obscure color, like Aureolin.
For each user session (say, the last 5 minutes of using the application), I would like to predict the most likely color the user will pick (I'll be looking at their previous chosen color, and some other behavioral attributes).
The rule is: a prediction should only be EITHER one of my 10 globally available colors (the usual ones: red, blue, green, etc.), OR one of the obscure colors that the user had already come up with.
That is, I should never suggest an obscure color to someone who hasn't come up with it before.
My thinking is that because the space of right answers is more limited than the entire space of answers, each user should have its own model (trained on a subset of the data that includes its own colors and perhaps the global colors as well). If that's the right direction, I'll have to think about coordinating all thousands of models together.
Is this the correct framing?


texts: Detect the time at which deviation occurs in time series data

texts:
I working on multivariate time series data. I have sensor data generated by a machine every time it is operated. Data set consists of machine_ID(machines of same model), hours_ operated, measurements from various sensors. The machine starts to degrade after operating for certain hours. I would like to find the hours after which there is step change after which the performance starts to degrade.

I want to do this using machine learning approach preferably and would like to plot the graph marking the deviation. Which ML techniques could be used for this approach.

I have performed exploratory data analysis where I could find the point at which there is deviation occurrence. Now, I want to confirm this by running a model to detect the occurrence of step change. In the figure above, the decline starts somewhere at 100 and decline gradually. Now, is there any way I could find this pint through models.

I greatly appreciate any links or suggestion to deal with this.

Thanks in advance.


texts: How to merge all the data to have a final dataset

texts:
I am working with a problem that has different tables, end goal to predict if a customer will end up subscribing based on purchases.
Mother table containing user_id,register_reason
|user_id|reason_reg|source|
|-------|----------|------|
|   1   |     2    |  3   |
|   2   |     3    |  1   |

I have then the purchase data where a customer can have more than one entry
|user_id|product_id|
|-------|----------|
|   1   |     A    |
|   1   |     B    |
|   1   |     C    |
|   1   |     D    |
|   2   |     A    |
|   2   |     E    |

Ideally I want to have one dataset where the uniqueidentifier is the user_id and there are no duplicated rows based on this value:
The final dataset (in my head), could look like
|user_id|reason_reg|source|product_id_A|product_id_B|product_id_C|product_id_D|product_id_E|
|------------------------------------------------------------------------------------------|
|   1   |     2    |   3  |      1     |      1     |      1     |      1     |      0     |
|   2   |     3    |   1  |      1     |      0     |      0     |      0     |      1     |

My questions are:

Is the approach correct?

Is there a dataframe or library that does that automatically? or I do that myself by panda before feeding it to a algorithm.

In your opinion, is there a better way to approach the problem(I could add also add aditional columns like total_products with the sum of how many products the user bought)


texts: Why does np.linalg.eig produce an opposite-signed eigenvector?

texts:
I am learning SVD by following this MIT course.  In this video, the lecturer is finding the SVD for

$$
\begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix},
$$

which involves finding the eigenvalues for

$$
C^T C =
\begin{pmatrix}
26 & 18 \\ 18 & 74
\end{pmatrix}.
$$

In the example (at the time in the link above), the lecturer finds eigenvalues

$$\begin{pmatrix}-3/\sqrt{10} \\ 1/\sqrt{10} \end{pmatrix},
\begin{pmatrix} 1/\sqrt{10} \\ 3/\sqrt{10} \end{pmatrix}.$$

But np.linalg.eig produces the opposite vector to the second one:

w, v = np.linalg.eig(C.T*C)
v
matrix([[-0.9486833 , -0.31622777],
        [ 0.31622777, -0.9486833 ]])

Why?


texts: Is it possible to implement a Recommender System without having a ratings/previous purchases similar data?

texts:
I'm trying to implement a recommender system for a website that hosts a wide variety of softwares and you can search the website to find what you need. The need is to implement a recommender system to better recommend softwares to the users except that there is no ratings system and users don't really buy the softwares from the website...
A first approach would be like this: let's imagine a user clicks on software A, we can search for the most clicked on softwares by all the users who also clicked on software A. But this would lack meaning since these users that clicked on software A may just search the category of this software and would click on the first suggestion from a list that may be sorted alphabetically.
Also the users don't really interact with the softwares. They can't bookmark it or post a comment so we can't just take these actions and give them a weight to simulate a ratings system.
I don't really find ideas about solving this issue and I don't have any experience with user tracking data so any idea would be really helpful.


texts: What is a good way to store processed CSV data to train model in Python?

texts:
I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

HDFS
HDF5
HDFS3
PyArrow


texts: Reversing data through a tensorflow feature_column.embedding_column

texts:
I am building an variational autoencoder in Tensorflow, and one of my columns has object data. The data is too sparse (on the order of 2^16 possible values) to use one-hot encoding, it's not ordinal in nature, and I have strong misgivings about some of the other options available through libraries like CatgoryEncoders, but an embedding column through the feature_column api seems to be a remarkably good way to represent all of the possibilities in a mathematically sound way.
Now, I have no problem using the difference between the output of the DenseFeatures layer and the final layer as my loss, but the end goal is to use the VAE to generate data similar to the original. To do that, I'd need to feed the columns representing the object column back through the embedding_column and get a representative value back out. But nothing I've seen does this, and I'm not certain why.
Does anyone have any suggestions? I'm fairly new to the feature_column api, so there may be a simple solution I'm just not aware of. I'm also willing to dig into the DenseFeatures layer, get the weights, and figure out how to do it myself. But before I do, I want to check to see if there's anything that already exists.
Thanks!


texts:
I need to perform clustering in a given dataset.
There are atrributes with numerical as well as categorical values.

What is the best way to convert categorical to numeric value ?

as an example one field is colour and the values are red, green, blue
so can I assign a  mapping like:

red : 1, green : 2, blue : 3 or
red : 11, green : 25, blue : 30

and if I provide a mapping like this will this affect the Euclidian distance for clustering ?

or is there any other way ?


texts: How to explain ANN can predict much larger output values (e.g., y>2.5) when it was only trained with small output values (y>=2.5)

texts:
I have trained models with both ANN and XGBoost. I am wondering that whether ANN has the ability to predict much larger output values (e.g., $y>2.5)$ when it was only trained with small output values $(y\le2.5$). Actually, my experiment confirms this but I do not have theoretical evidence to support it (I am not very familiar with computing science).
I trained both ANN and XGBoost models and the experiment shows that XGBoost outperforms ANN at both validation and test data sets (y values are all less than 2.5). But when I apply the models to other cases without true y labels (expected larger y values), ANN seems to be able to give more reliable results while XGBoost can only give y values less than 2.5.
One point should be mentioned is a prior knowledge is that the cases of both small and large y values obey the same physical function (e.g. $y=e^x+lnx$, the real function is much more complex than this), can I argue that ANN has the ability to predict extended y values while XGBoost does not?


texts: Unsupervised learning to identify most common basketball plays from spatial data

texts:
The NBA has a system called Sports VU that tracks x-y coordinates of every player and the ball every 1/10th of a second for every game of the 2013-2014 and 2014-2015 seasons. With some fancy web scraping I now have access to this data and -- because I'm such an avid fan of the NBA -- I would like to identify each team's most common plays. Assume I don't have any knowledge about each team's plays beforehand (so I don't think supervised learning would work here). What would be the best unsupervised learning techniques to use?

If I could trace each player's path over the course of the play, I imagine the problem would be similar to what you would see with image recognition/classification. Anyways, should I use PCA, some kind of neural  network? I understand this is a very broad question -- I don't need to know how to code it (I'm a proficient coder and machine learning practitioner); I'm just looking for high level unsupervised machine learning details.


texts: Euclidean distance between time series in Python

texts:
While thinking about similarity between two time series, one can use DTW to approach the issue. There is a Python package for that mlpy.

It is also said to compare time series via simple euclidean distance.

Is there a Python package to this?


texts: How to prepare data for LSTM time series prediction

texts:
I have a binary classification task for time series data.
Every 14 rows in my CSV is relevant to one time slot. How should I prepare this data to be used in LSTM? In other word how to feed the model with this data?


texts: In XGBoost would we evaluate results with a Precision Recall curve vs ROC?

texts:
I am using XGBoost for payment fraud detection.  The objective is binary classification, and the data is very unbalanced.  One out of every 3-4k transactions is fraud.

I would expect the best way to evaluate the results is a Precision-Recall (PR) curve, not a ROC curve, since the data is so unbalanced.

However in the eval_metric options I see only area under the ROC curve (AUC), and there is no PR option.  https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

Also the documentation recommends AUC  http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html

Does it make sense to not use a Precision-Recall (PR) curve?


texts: Why does my Keras model learn to recognize the background?

texts:
I'm trying to train this Keras implementation of Deeplabv3+ on Pascal VOC2012, using the pretrained model (which was also trained on that dataset).

I got weird results with the accuracy quickly converging to 1.0:

5/5 [==============================] - 182s 36s/step - loss: 26864.4418 - acc: 0.7669 - val_loss: 19385.8555 - val_acc: 0.4818
Epoch 2/3
5/5 [==============================] - 77s 15s/step - loss: 42117.3555 - acc: 0.9815 - val_loss: 69088.5469 - val_acc: 0.9948
Epoch 3/3
5/5 [==============================] - 78s 16s/step - loss: 45300.6992 - acc: 1.0000 - val_loss: 44569.9414 - val_acc: 1.0000

Testing the model also gives 100% accuracy.

I decided to plot predictions on the same set of random images before and after training, and found that the model is encouraged to say everything is just background (that's the 1st class in Pascal VOC2012).

I'm quite new to deep learning and would need help to figure out where this could come from.

I thought that perhaps it could be my loss function, which I defined as:

def image_categorical_cross_entropy(y_true, y_pred):
    """
    :param y_true: tensor of shape (batch_size, height, width) representing the ground truth.
    :param y_pred: tensor of shape (batch_size, height, width) representing the prediction.
    :return: The mean cross-entropy on softmaxed tensors.
    """
    return tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=y_pred, labels=y_true))

I am a bit uncertain on whether my tensors have the right shape. I'm using TF's dataset API to load .tfrecord files, and my annotation tensor is of shape (batch_size, height, width). Would (batch_size, height, width, 21) be what's needed? Other errors from inside the model arise when I try to separate the annotation image into a tensor containing 21 images (one for each class):

tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [12,512,512,21] vs. [12,512,512]
         [[Node: metrics/acc/Equal = Equal[T=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"](metrics/acc/ArgMax, metrics/acc/ArgMax_1)]]
         [[Node: training/Adam/gradients/bilinear_upsampling_2_1/concat_grad/Slice_1/_13277 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_62151_training/Adam/gradients/bilinear_upsampling_2_1/concat_grad/Slice_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

Thank you for your help!


texts: How can I do nested cross validation for multivariate Time series forecsting

texts: I'm trying to do nested CV for my multivariate time series but I'm really confused how to do it. I have 7 Time series which are the inputs of my CNN model and one time series as target.Always when I read about it I found they talk about univariate.Any advices


texts: Is there any advanced collocation tool based on NLP+DL?

texts:
I am not native English speaker and often times I use Ozdic to find the correct word choice. It's good overall but its database is quite limited. I think a tool based on Word2Vec should be able to do a great job for finding collocations/cooccurance.

  With skip-grams, given a window size of n words around a word w, word2vec predicts contextual words c; i.e. in the notation of probability p(c|w). Conversely, CBOW predicts the current word, given the context in the window, p(w|c).

I am looking for a tool that takes a sentence with a blank spot as input and gives alternative options for the blank spot as an output.


texts: Training neural network to generate realistic terrain for video games

texts:
I've recently had an idea to create a tool that makes it easier for environment artists to generate highly realistic terrain for video games. I've seen approaches using GANs and I'm familiar with the publication by Guérin et al "Interactive Example-Based Terrain Authoring with Conditional Generative Adversarial Networks".
However I don't like the limited level of control the proposed solution gives and the fact that they had to artificially generate terrain sketches later used for training. What I'd like to have would be the possibility to use different brushes to place different objects - lakes, valleys, roads etc and a separate texture which roughly represents the desired height and erosion. For that I'd need a dataset of high resolution aerial maps of terrain and correlated elevation maps. I'd be very thankful for insights (if my approach makes sense at all) and some help finding such dataset - so far I've been failing to find it and either there's no such dataset or I'm looking in the wrong place.


texts: Popular classification algorithms over time

texts:
In the book "Deep Learning with Python" by Francois Chollet (2018), in section 1.2.4 one can find:

Decisions trees learned from data began to receive significant research interest in the 2000s, and by 2010 they were often preferred to kernel methods.
...
In particular, the Random Forest algorithm introduced a robust, practical take on decision-tree learning that involves building a large number of specialized decision trees and then ensembling their outputs. Random forests are applicable to a wide range of problems—you could say that they’re almost always the second-best algorithm for any shallow machine-learning task. When the popular machine-learning competition website Kaggle (http://kaggle.com) got started in 2010, random forests quickly became a favorite on the platform—until 2014, when gradient boosting machines took over.

It sounds for me like the author trying to draw this kind of the evolution of classification methods:
| Method            | From | To           |
|-------------------|------|--------------|
| Kernel Methods    | ...  | 2000         |
| Decision Trees    | 2000 | 2010         |
| Random Forest     | 2010 | 2014         |
| Gradient Boosting | 2014 | Today (2019) |

Is Gradient Boosting the most popular library nowadays?
Is it can be universally applicable for any cases?
What do you think?


texts:
Much of the vocabulary here is new to me so forgive me if I misspeak.

I'm attempting to find an approach to a very simple classification problem. I have a set of description such as:

House : Building with several walls some windows and a roof

Car : Vehicle with four wheels, a steering wheel, and an engine

We have a third party giving a description of what they:

My House : Red walls, a balcony, a roof and a great view

Give these loose natural language descriptions what would be a good approach to classifying "My House" as a "House". Would a simple word counting approach be applicable, perhaps with some sort of emitting dictionary {"the", "a", etc}. Any alternative algorithms to consider?


texts: As simple as the title: which one is best for outlier detection between DBSCAN and RANSAC? What are pros and cons of each model?