texts: How do I create a dok matrix with split files correctly?

texts:
I let my model run normally and have defined an early stopping as a callback. The model breaks off, I let it run through without early stopping, loss and val_loss go further and further apart (see photos below). This is an indication of overfeeding.
I checked the model. It is due to the splitting of the data because it ran without errors without the validation data.
My data set consists of several purchases. A user can buy several items. A purchase by a user looks like this
            purchaseid  itemid
0           0           3
1           0           8
2           0           2

I would like to split my data frame in a ratio of 80:20. However, complete purchases should always be made and not split up at random, so I wrote myself a method.
For example what I want
   purchaseid  itemid
0           0       3
1           0       8
2           0       2
3           1      10

What I do not want
   purchaseid  itemid
0           0       3
1           0       8

3           1      10

Dataframe:
d = {'purchaseid': [0, 0, 0, 1, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 8, 9, 9, 9, 9],
     'itemid': [ 3, 8, 2, 10, 3, 10, 4, 12, 3, 12, 3, 4, 8, 6, 3, 0, 5, 12, 9, 9, 13, 1, 7, 11, 11]}
df = pd.DataFrame(data=d)

   purchaseid  itemid
0           0       3
1           0       8
2           0       2
3           1      10
4           2       3

I don't know whether it is due to the splitting of the data or the creation of the dok matrix. Can someone tell me if they see a bug?
The splitting takes place without errors. I tested this with several data sets and always got the desired result. Is it probably because of the dok matrix, or the get_train_samples()?
Code:
PERCENTAGE_SPLIT = 20
NUM_NEGATIVES = 4
def splitter(df):
  df_ = pd.DataFrame()
  sum_purchase = df['purchaseid'].nunique()
  amount = round((sum_purchase / 100) * PERCENTAGE_SPLIT)

  random_list = random.sample(df['purchaseid'].unique().tolist(), amount)
  df_ = df.loc[df['purchaseid'].isin(random_list)]
  df_reduced = df.loc[~df['purchaseid'].isin(random_list)]
  return [df_reduced, df_]

def generate_matrix(df_main, dataframe, name):

  mat = sp.dok_matrix((df_main.shape[0], len(df_main['itemid'].unique())), dtype=np.float32)
  for purchaseid, itemid in zip(dataframe['purchaseid'], dataframe['itemid']):
    mat[purchaseid, itemid] = 1.0

  return mat

dfs = splitter(df)
df_tr = dfs[0].copy(deep=True)
df_val = dfs[1].copy(deep=True)

train_mat = generate_matrix(df, df_tr, 'train')
val_mat = generate_matrix(df, df_val, 'val')
.
.
.
def get_train_samples(train_mat, num_negatives):
    user_input, item_input, labels = [], [], []
    num_user, num_item = train_mat.shape
    for (u, i) in train_mat.keys():
        user_input.append(u)
        item_input.append(i)
        labels.append(1)
        # negative instances
        for t in range(num_negatives):
            j = np.random.randint(num_item)
            while (u, j) in train_mat.keys():
                j = np.random.randint(num_item)
            user_input.append(u)
            item_input.append(j)
            labels.append(0)
    return user_input, item_input, labels

num_users, num_items = train_mat.shape

model = get_model(num_users, num_items, ...)

user_input, item_input, labels = get_train_samples(train_mat, NUM_NEGATIVES)
val_user_input, val_item_input, val_labels = get_train_samples(val_mat, NUM_NEGATIVES)

hist = model.fit([np.array(user_input), np.array(item_input)], np.array(labels),
                 validation_data=([np.array(val_user_input), np.array(val_item_input)], np.array(val_labels)))

Result:


texts: Help with the following error: Variable already exists, disallowed. Did you mean to set reuse=True in VarScope?

texts:
I am not sure how to handle this error. This is from an RNN tutorial found here.

I vaguely understand that the variables need to be able to be reused, but I don't know how to implement this fix. Train.py ran for several hours and finished successfully; this particular error comes when running sample.py

Given the long training time, I would very much like to get this working, rather than starting over.

ValueError: Variable rnnlm/softmax_w already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
  File "C:\Users\wlt42\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1269, in __init__
    self._traceback = _extract_stack()
  File "C:\Users\wlt42\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\wlt42\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
    op_def=op_def)


texts: Scalling and unscalling of data for SVR prediction

texts:
I'm trying to use SVR to predict a certain feature. I create the model with the following code:

from sklearn.svm import SVR
from sklearn.preprocessing import StandardScaler

X = data
# this is the outcome variable
y = data.iloc[:, 10].values

sc_X = StandardScaler()
sc_y = StandardScaler()

X2 = sc_X.fit_transform(X)
y = sc_y.fit_transform(y.reshape(-1, 1))

# my_custom_kernel looks at certain columns of X2 / scaled data
my_regressor = SVR(kernel=my_custom_kernel)
my_regressor = regressor.fit(X2, y)

After creating the model, I want to test it to see if the prediction is good. The first thing the code does is scale the row that I want to test, using the same scaler as above (sc_X). Then I try to reverse the scaling of the prediction result (using sc_y). During this process, I get datatype errors. Here is the code:

line1 = X.iloc[0].as_matrix().reshape(1, -1)
line1_scaled = sc_X.fit_transform(line1)
res = regressor.predict (line1_scaled)
pred_line1 = sc_y.inverse_transform (res) # The error appears to be here

Error:

ValueError: non-broadcastable output operand with shape (1,) doesn't match the broadcast shape (24,)

Thanks.


texts: Is there a rule of thumb for the initial value of loss function in a CNN?

texts:
I notice that most of my successfully trained CNNs initiate (i.e. first epoch) loss is in the 1-2 range. Typically, these are image classifiers, either for discrete classes (cat, dog, ship, etc) or semantic (pixelwise) segmentation. Occasionally, either because of poor design or architecture misunderstand on my part, I see an initial loss in the 60s; the other day one started at 200 (and didn't converge).

So my question is: is there a heuristic that would allow me to determine a ball park value for my initial loss value? If so, on what aspects of my trained model is it contingent?


texts: How shouild I save model weights in Pytorch?

texts:
### Create / load model

# Faster - RCNN Model - pretrained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
num_classes = 2

# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features

# replace the pre-trained head with a new one
model.roi_heads.box_predictor =  FastRCNNPredictor(in_features, num_classes)

Training the model doesn't use model.fit() function, it uses cycles.
    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

How can I save the model?


texts:
I would like to clarify my understanding of learning curves with two example plots below. I am experimenting with small data sets here between 500 and 1500 samples to clarify my understanding.
My understanding from the learning curve below is that underlying data must have a lot of noise and therefore the learning algorithm is not able to generalize the underlying function. This learning curve indicates neither high bias or variance. Just the overall data samples are not a good indicator of the outcome. Is my interpretation of this learning curve above correct? Would getting more data help?

From the learning curve below, I am interpreting this as a good learning curve. Because both the training and cross validation misclassification errors are going down as we add more samples. Around ~240 samples we get the least difference between the training and validation sets. Is this indicative of high variance since the training and validation never converge even though the misclassification error delta is pretty low between the two?


texts: Techniques for Collection/Graph Conversion to Cluster-able Data

texts:
Here are my problems; any techniques/papers as to how to approach this problem would be much appreciated. I also apologize for the vagueness of my question title; I do not really know if there is a term for what I am trying to do.

Problem 1:

Suppose we have a $n$-dimensional space $\mathcal{X}$. Say we have a set $X$ of data points $x$ which lie in $\mathcal{X}$. Now, suppose we have a set $\mathcal{C}$, which is a collection of sets $\{C_i\}$ which partition $X$. There is no restriction on the size of a given $C_i$. I would like to find some way to cluster $C_i$ based on the data points inside them. This means I would like some way of comparing the $C_i$'s.

My original thought to solve this was to have some way to convert each of the $C_i$ to a vector given the points inside them. This would be easier to do if the $x$ were discrete datapoints but this is not the case; therefore I have reached a roadblock. It should also be noted that the clustering of the $C_i$ is not my problem; I know many techniques to do this and am mostly just stuck on how to vectorize the $C_i$.

UPDATE: My second thought was that maybe instead of vectorizing each $C_i$, we could instead form a probability distribution over the space $\mathcal{X}$ for each of the $C_i$ and then to compare them, use something like KL-divergence to find the similarity. This seems like it would be relatively effective, and I know of techniques of how to form the probability distributions, so it seems tractable as well.

Problem 2 (a more advanced version of Problem 1):

This problem is very similar to Problem 1, however there is extra structure imposed on the $C_i$. For this problem, we define the $G_i = (C_i, w)$, where $w : \mathcal{X} \times  \mathcal{X} \mapsto \mathbb{R}$ is a distance metric imposed on the $x$'s in a given $C_i$. Basically, $G_i$ is a weighted graph with the data points as vertices. In this problem, similar as to problem 1, I would like to see if there is some way to compare the $G_i$ so that I can eventually cluster them.

If there are any clarifications I can make please let me know in the comments; otherwise as I said, I would be sincerely grateful for any suggestions as to how to go about these problems. Thanks in advance!


texts: kde plot for interpreting the correlation

texts:
i have created some new features for my model. I found people use kde plot to find out the correlation between the created feature and the target variable, but I am not really sure how to find the correlation from kde.

Any help on how to interpret the correlation from kde plot will very helpful


texts:
According to a benchmark of GBM vs. xgboost vs. LightGBM (https://www.kaggle.com/nschneider/gbm-vs-xgboost-vs-lightgbm) it is possible to implenet xgboost with the argument

tree_method = 'hist'

in R.

However doing so gives me always an error:

Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) :
Invalid Input: 'hist', valid values are: {'approx', 'auto', 'exact'}

What am I missing?


texts: How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

texts:
My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50.
I would like to identify if there is a relationship between the independent variables and the dependent variable and to explore which variables/set of variables have the largest impact on the result.
Most of the solutions I have checked so far (Nominal Variable Association and Cramér's V) don't support more than one independent variable.


texts: Why there are specific number of layers with same depth in any CNN architecture?

texts:
why each architecture have specific number of layers with same depth

like vgg had 2 with 64 depth and 2 with 128d and 4 layers

same with ResNet and 34 layer plain like 6 layers with depth of 64 and 8 layers with depth 128.

and why depth increasing by last depth *2 ?

is there is way or equation to decide number of layers with each depth ?


texts: About finding pre-trained faceswap model for converting the Lionel Messi to baby face

texts:
I am a newbie in deepfake field, but I am  trying to use faceswap to build one video of deepfake for converting Lionel Messi face to a baby face. So I tried faceswap it in colab to do this by this link, like showing below:

However, according to my previous try, it seems too long to do it, especially with colab (the disconnecting problem could be solved by this post method!), but I like to find some better way like to find some pre-trained model to use for this propose (maybe by some methods of this post!).
So, now I like to use some open-source deepfake packages with a pre-trained model, for this purposes, which I don't test others GitHub projects like Deepfakelab or ..., and don't know about their speed and the possibility of existing the pre-trained models on those open-source community, for my usage (converting Lionel Messi face to baby face).
So, I would appreciate, if you give me some suggestions or guidance, to find proper open-source project or some internet sources to find pre-trained models for this usage, like said below:

Thanks.


texts: Determining size of FC layer after Conv layer in PyTorch

texts:
I am learning PyTorch and CNNs but am confused how the number
of inputs to the first FC layer after a Conv2D layer is calculated.
My network architecture is shown below, here is my reasoning using
the calculation as explained here.
The input images will have shape (1 x 28 x 28).
The first Conv layer has stride 1, padding 0, depth 6
and we use a (4 x 4) kernel. The output will thus be (6 x 24 x 24),
because the new volume is (28 - 4 + 2*0)/1.
Then we pool this with a (2 x 2) kernel and stride 2 so we get an output of (6 x 11 x 11),
because the new volume is (24 - 2)/2.
Same thing for the second Conv and pool layers, but this time with a (3 x 3) kernel in the Conv layer, resulting in (16 x 3 x 3) feature maps in the end.
My assumption would then be that the first linear layer should have 144 inputs (16 * 3 * 3),
but when I calculate the inputs programatically, I get 400. What did I miss?
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 4)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.fc1 = nn.Linear(400, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, len(classes))

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features *= s
        return num_features # 400, not 144

Related but less so: is there a reasoning used by people to get a good kernel size, number of layers
and number of pool layers or does everyone just look at what the SOTA papers do?


texts: Retrieve user features in real time from UserId for prediction

texts:
Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date.

Here is the proposed architecture -

Let's say I have a pre-trained model hosted as a service. The part I'm struggling with is, how do I get the user features from the database in realtime from the RiderID to be used by the prediction service (XGBoost Model)? I'm guessing a lookup in a SQL database will take too long, considering I have 1M+ users and rides.

Thanks in advance!


texts: Data Augmentation for Regression ANN with low Sample Size

texts:
There is a Dataset of 65 tuples.
I want to Augment new Data from this set and validate my ANN on the original Data.
Is there a possibility, that my ANN already overfits on the augmentet Data. For example that the augmentation Is simulated by the ANN, and how does one prevent this ?


texts: Covariates in machine learning classificatoin

texts:
I would like to classify a number of people as sick and healthy. I have all the body measures as features. I have to control for age and sex because body features are affected by that. How do I adjust for age and sex?
Should I just add all the variables including age and sex as variables and try to classify? Or, should I do something else?


texts: Identifying persistent clusters within a series of graphs

texts:
The task is to identify persistent clusters, i.e., groups of nodes that "persist" as clusters (tend to form a cluster) in a series of graphs.
This is how I approached the problem: I form a supergraph out of a series of input graphs, such that whenever an edge occurs again, its weight is increased by 1. Then, I apply Walktrap algorithm on the weighted supergraph (random walks are now dependent on weights), and the resulting clustering corresponds to persistent clusters.
Another idea is to go in embedding space (embedding available for input graphs in the series), and then apply similar idea: 1) form clustering on each input embedding, 2) form a supergraph comprising all the nodes, and where edges exist between all nodes forming a specific cluster (a clique is formed), and, as above, whenever an edge occurs again, its weight is increased by 1. In the end, I apply Walktraph algorithm.
However, although with pretty high running times, the above approaches do not yield satisfactory results (perhaps this is due to Walktrap, but a lot of non-persistent data tend to pollute the clusters). Do you have some other ideas on how to approach the problem?


texts:
So I have a set of images with 16 layers each.
Is there a good reason to split the channels in a format like
example :   [images_length,16,image_width , image_height]
instead of creating something more simple like
example 2 : [images_length,1,image_width * 16 , image_height]

In the second example I use a single channel and lay each channel next to the other in order to create an image that has 16 times the length of the original image, but same height etc.

So here are the actual questions:

Are there any negative trade offs on using the second method ?
Which method is more memory efficient ( on the graphics card side of things ) ?

I tagged both keras and tensorflow as I use tf as backend for keras.

Thank you for your time!


texts: Pivot DataFrame while calculating new values

texts:
I have reduced the data set to only the columns I need:

| yearID | POS | PO | A | E |
|--------|:---:|:--:|:-:|:-:|
|  1871  | SS  | 0.0|3.0|1.0|
|  1871  | 2B  |30.0|1.0|0.0|
|  ...   |  .. | ...|...|...|

source: Sean Lahmans 2015 Baseball Data set Using the Fielding.csv file.

I am trying to calculate the Fielding Percentage:

values = df['E'] / (df['PO'] + df['A'] + df['E'])

Where there are multiple records for each 'yearID'. I am not sure if I need to transpose, apply a function or map one. Additionaly, in what order should I be moving the pieces around in.

data.loc[:,('C')] = middle_infielders.PO + middle_infielders.A + data.E
data.loc[:,('FP')] = 1 - (data.E / data.C)

| yearID | POS | PO | A | E |  C  |  FP  |
|--------|:---:|:--:|:-:|:-:|     |      |
|  1871  | SS  | 0.0|3.0|1.0| 123 | .960 |
|  1871  | 2B  |30.0|1.0|0.0|  12 | .452 |
|  ...   |  .. | ...|...|...| ... |  ... |

I would like it in this form to plot a line graph:

| yearID |  SS  |  2B  |
|--------|:----:|:----:|
|  1871  |0.3745|0.1245|
|  1872  |0.8940|0.3366|
|  ...   |  ... | ...  |

End result: One mean for each 'POS' (SS and 2B) each year.

UPDATE

Expecting pivoting to aggregate the values='FP' from the statement:
data.pivot(index='yearID', columns='POS', values='FP')
But, I get the error:

"ValueError: Index contains duplicate entries, cannot reshape."

Should I apply a Lambda to calculate the Fielding Percentage ('FP') I want in the values instead of pre-calculating it?


texts:
I have a data.frame that contains two columns that I want to group and sum the direct labor for employees.

Employees   Actual work
Joe Smo      8
Joe Smo      7
Joe Smo      5
Sam Adams    7
Sam Adams    5
Sam Adams    3

The desired outcome is

Results:
Joe Smo      20
Sam Adams    15

Here is the code I used that doesn't work.

aggregate(directLaborNov18[c('Employee')], directLaborNov18['Actual_work'], sum)


texts: ValueError: cannot reshape array of size 136415664 into shape (2734,132,126,1)

texts:
I have a data set I loaded with cv2 but when I try to format it I get the above error. I start by moving the data into X_train and X_test (the loaded data is in x_train and x_test).
X_train = []
X_test = []

# Image matrices are different sizes so I am making them the same size
for i in range(len(x_train)-1):
  resized = cv2.resize(x_train[i], (img_width, img_height))
  X_train.append(resized)
for i in range(len(x_test)-1):
  resized = cv2.resize(x_test[i], (img_width, img_height))
  X_test.append(resized)

# Convert to numpy arrays
X_test = np.array(X_test)
X_train = np.array(X_train)

# Gather statistics
print(X_train.shape)    # -> (2734, 132, 126, 3)
print(X_train.size)     # -> 136415664
print(len(X_train))     # -> 2734

# Convert to black and white
X_train = X_train/ 255.
X_test = X_test/ 255.

# First line throws error
X_train = np.reshape(X_train, (len(X_train), img_height, img_width, 1))
X_test = np.reshape(X_test, (len(X_test), img_height, img_width, 1))

What am I doing wrong?


texts: sklearn.neighbors.NearestNeighbors - knn for unsupervised learning?

texts:
From basic theory I know that knn is a supervised algorithm while for example k-means is an unsupervised algorithm.

However, at Sklearn there are is an implementation of KNN for unsupervised learning (http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html#sklearn.neighbors.NearestNeighbors).

What is exactly this unsupervised version of knn at SkLearn?

Is this a knn algorithm?

If yes how it is unsupervised since by definition knn is supervised?
If no what is it then?


texts: Why in deep learning or another AI technique python is used rather than other languages?

texts: I want to know, why python? Instead of C#, Java, C\C++ and so on?


texts: Why does measuring for Asymmetric Binary Attributes exclude t (Values which are 0 for both)?

texts:
I'm teaching myself Data mining and now struggling from this problem.

According to this table(http://slidewiki.org/slide/22352),
if the attributes are Asymmetric, dissimilarity is calculated without t (Values which are 0 for both)

Let me give an example.

There are 2 objects, called Products.

They have 2 attributes, COLOR and SIZE.

COLOR is either RED(1) and BLUE(0) (They are binary).
SIZE is either BIG(1) or SMALL(0) (They are binary too).

When there are RED-BIG(1-1) & RED-SMALL(1-0) objects, d is 1/2, regardless of if the attributes are symmetric or asymmetric.

However, if they are BLUE-SMALL(0-0) and BLUE/BIG(0-1), d is 1/2 for symmetric BUT 1/1 for asymmetric.

How does the difference come from? In both cases, there is only one difference, BUT dissimilarity isn't the same.


texts: can I use public pretrained word2vec, and continue train it for domain specific text?

texts:
I have a set of reviews from apparel domain, about 100K reviews (2M words).
And I want to train word2vec to do some cool NLP staff with it.

However the size is not enough for creating adequate word2vec model, it requires billions of words.

So the idea is to use public corpora (e.g. wikipedia), or even use some pretrained models (e.g. from gensim cool framework) and add my domain specific text.
I assume the model will get aware of unseen-in-public words, and could correct vectors for common word.

Does it make sense ? will the 2M words makes any effect at all ?


texts: distinguishing two in phase components in a signal

texts:
Do you guys know some algorithems that can surely extract a component out of the following signal:

I have an IR-sensor that generates the Signal s(t), which consists out of two components: s(t) = c(t) + r (t).

In the picture you can see a model of the output signal. Both components are periodic with the same frequency and are in phase.
r(t) rises much quicker than c(t) (timeconstant_c: 15 sek, timeconstant_r: 0.1 sek) and is actually (other than shown in the picture) much smaller than c(t) (factor 25).
Due to thermal drifts the c(t) component varies in amplitude and time constant. The intresting Information that has to be extracted is the Amplitude of r(t)- component.
Is there any standard halfway ease to understand method to get rid of the c(t)-part in the signal?

I have naively implemented a wiener filter, but a change in r(t) in a test signal has no impact on the evaluated r(t).
Also I have looked at the fourier-spectrum, in the hope to find an increase in high frequencies when r(t) rises. But the effect is too small.
Now my approach is to look how the signals rises the very first 50 ms, because these are dominated by the r(t) component and then correct this for long any drifts, I try to guess out of the whole signal.
But still I would guess there is a more rigorous way.

I studied physics but, as you may have noticed, I have no background in signal processing whatsoever. And I would prefer not to spend like two month reading into it, when I am not sure that there is a practical solution for this.
Thanks for reply in advance.


texts:
I have an imbalanced data set consisting of some 10's of millions text strings, each with thousands of features created by uni- and bigrams, and additionally I have also the string length and entropy of string as features.

It is a multiclass data set (40-50 classes), but it is imbalanced. Some classes can be 1000x smaller compared to the largest class. I have restricted the data to 1 million strings per class as maximum, otherwise the imbalance could be even larger.

Because of this I want to use over-sampling to improve the data for the underrepresented classes. I have looked into ADASYN and SMOTE from the python imblearn package. But when I run it the process eats up all my RAM in the swap memory, and soon after the process gets killed. I assume because the memory is not enough.

My question is now how to best proceed. Obviously my data is too large to be over-sampled as it is. I have thought of two options, but I cannot make out which is the most "correct".

I sent in only one underrepresented class and the largest class, and repeated this for each underrepresented class. I am not sure if this could mean that classes might start to overlap though.
I instead under-sample the data, maybe down to 100k samples per class. This might reduce the data enough such that I can run oversampling on the less represented classes (with 1k-10k samples).

Any other options that are more appropriate that I have missed?


texts: Can I use the test dataset to select a model?

texts:
I'm trying to understand if the test dataset can be used to select a final trained model. Let's assume this scenario:
I first split the whole dataset: 70% training, 30% test. Then I fit several models (let's say NN, RandomForest, AdaBoost,..) on the training dataset with cross-validation and tune the hyperparameters to get the best performance on the train data. I know that these scores are biased, since I was tuning the hyperparameters on this data.
Then I use the test dataset to get the true performance on the unbiased data and select which model performs the best.
Is this a correct way to use the test dataset? Some confusion comes from the internet definition of the test dataset:

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

It seems like it should only be used to get the performance of the one final trained model. My teacher told me I can't select a trained model based on the scores of the test dataset and quoted the definition above. I'm struggling to believe that she is correct. Which dataset should be used to select the model then?


texts: Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

texts:
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select from for implementing the required analyses:

One approach is to extract the data from MongoDB to a Hadoop cluster and do the analysis completely in Hadoop platform. However, this requires considerable investment on preparing the platform (software and hardware) and educating the team to work with Hadoop and write map-reduce tasks for it.
Another approach is to just put our effort on designing the map-reduce algorithms, and run the algorithms on MongoDB map-reduce functionalities. This way, we can create an initial prototype of final system that can generate the reports. I know that the MongoDB's map-reduce functionalities are much slower compared to Hadoop, but currently the data is not that big that makes this a bottleneck yet, at least not for the next six months.

The question is, using the second approach and writing the algorithms for MongoDB, can them be later ported to Hadoop with little needed modification and algorithm redesign? MongoDB just supports JavaScript but programming language differences are easy to handle. However, is there any fundamental differences in the map-reduce model of MongoDB and Hadoop that may force us to redesign algorithms substantially for porting to Hadoop?


texts:
I am relatively new to Orange, trying to utilise it for linear regression, in particular partial least squares (PLS). My statistics knowledge is in the moment not good enough to know whether I could compose an equivalent by combinding PCA with ordinary linear regression, but I would anyhow have expected a dedicated PLS widget - or as an option in the Linear regression widget.

I even found a description of PLS functionality in Orange2 here, but this is not available (or moved to where I don't find it) in Orange3.

Well, I have a python/numpy PLS implementation that I could probably adapt into a python script block, but is this necessary? Please tell me that it's just a mouse-click/import away.


texts: Multi Class + Negative Class Image Classification Strategies

texts:
I have seen a recurring theme in real-world problems I've worked with, where the problem looks something like "build an image classifier that recognizes classes A, B, and C but if the input is not those classes, don't recognize it". This features in other questions here from time to time, notably here and here and here; however, I have had difficulty finding a general solution to what seems to be a common problem.

The approaches I've seen so far seem to fall into roughly two categories:

Treat it as an imbalanced class problem. Usual tricks of dataset balancing, under/oversampling, and class weight all apply here. However, the "other" class tends to be broad and diverse, essentially making the CNN try to encode all the information for "other" in the network. While this works "good enough" for some problems, I have seen it perform poorly and generally it seems like a square peg in a round hole.
One class classification/outlier detection based approaches. In particular, the use of siamese and triplet networks are in a similar line of thought by trying to separate image feature vectors more clearly. Most of the research I've seen tends to focus on image similarity search or verification rather than applying to a multi-class + negative class problem directly. However, the notion of a contrastative or triplet loss to separate a more tightly definable class from everything else seems like a much more sound approach. Perhaps something like training the feature vector and then using one class SVM or some such?

Is there a general strategy for this situation like finetuning CNN's for normal classification? It seems as though some general solution that works much like finetuning plus a contrastative loss of some sort at the feature vector layer likely exists but I'm just not aware of it.


texts: I would like to find the accuracy of predicted data. Which package in R can I use that has the accuracy function?


texts: Can we make an AI to fine tune other AI hyper parameters?

texts: Every time AI gurus talk about fine tuning hyper parameters, they more or less say it's trial and error. But can't we make an AI to tell AI what its hyper parameters should be?


texts: How to train a machine learning algorithm with multiple labels

texts:
I have the following challenge and I very much hope that there is a solution to it. I also suspect that there is a simple approach to it. I just don't see it at the moment. Any help or advice is highly appreciated.

So, I have the following situation:

I asked persons to label about 1000 data points (each twice) on a 5-point scale, whose scores are not equi-distant. Texts were assessed with regard to several qualitative characteristics (such as comprehensibility). As was to be expected, the labelers did not always agree on the assessment. By analysing the inter-rater reliability, however, a "substantial" reliability (according to Landis and Koch) could be determined.

Now I want to use the labelled data as input for a machine learning algorithm (e.g. SVM and Random Forest). The challenge now is how to optimize the data in advance. Currently it is the case that for the same sample there are also different labels available.

The average value between different labels does not seem reasonable to me. So are there standard procedures how I can adjust the data set in advance?

You would help me a lot!

Thanks a lot in advance.


texts: F-Value P-Value T-Value linear regression interpretation

texts:
The following is what I found and understood and put in use without really understanding the logic behind it!
The t-value measures the size of the difference relative to the variation in your sample data. Put another way, T is simply the calculated difference represented in units of standard error. The greater the magnitude of T, the greater the evidence against the null hypothesis.
The p-value for each independent variable tests the null hypothesis that the variable has no correlation with the dependent variable. If there is no correlation, there is no association between the changes in the independent variable and the shifts in the dependent variable. In other words, there is insufficient evidence to conclude that there is effect at the population level.
The F value is a value on the F distribution. Various statistical tests generate an F value. The value can be used to determine whether the test is statistically significant. The F value is used in analysis of variance (ANOVA).
My question is:
Is there a one simple example with explanation of how to interpret these values, and how do they relate and complete each other?


texts: How to get tf tensor value computed in loss function in keras train_on_batch without computing it twice or writing custom loop?

texts:
I have a model and I've implemented a custom loss function something along the lines:

def custom_loss(labels, predictions):
    global diff
    #actual code uses decorator so no globals
    diff = labels - predictions
    return tf.square(diff)

model.compile(loss=custom_loss, optimizer=opt.RMSprop())

...

model.train_on_batch(input, labels)

#

How to get diff after I've run train_on_batch without causing
it to rerun predict a second time behind the scenes(unnecessary slowdown) and mess up with trainable/batchnorm etc(possible problems)?

I want to avoid making a manual raw tensorflow train_op loop etc, keeping track of learning phase and whatnot. Is this my only choice?

I'm using tensorflow 1.14's keras module


texts: Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression?

texts:
I recently completed exercise 3 of Andrew Ng's Machine Learning on Coursera using Python.

When initially completing parts 1.4 to 1.4.1 of the exercise, I ran into difficulties ensuring that my trained model has the accuracy that matches the expected 94.9%. Even after debugging and ensuring that my cost and gradient functions were bug free, and that my predictor code was working correctly, I was still getting only 90.3% accuracy. I was using the conjugate gradient (CG) algorithm in scipy.optimize.minimize.

Out of curiosity, I decided to try another algorithm, and used Broyden–Fletcher–Goldfarb–Shannon (BFGS). To my surprise, the accuracy improved drastically to 96.5% and thus exceeded the expectation. The comparison of these two different results between CG and BFGS can be viewed in my notebook under the header Difference in accuracy due to different optimization algorithms.

Is the reason for this difference in accuracy due to the different choice of optimization algorithm? If yes, then could someone explain why?

Also, I would greatly appreciate any review of my code just to make sure that there isn't a bug in any of my functions that is causing this.

Thank you.

EDIT: Here below I added the code involved in the question, on the request in the comments that I do so in this page rather than refer readers to the links to my Jupyter notebooks.

Model cost functions:

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def compute_cost_regularized(theta, X, y, lda):
    reg =lda/(2*len(y)) * np.sum(theta[1:]**2)
    return 1/len(y) * np.sum(-y @ np.log(sigmoid(X@theta))
                             - (1-y) @ np.log(1-sigmoid(X@theta))) + reg

def compute_gradient_regularized(theta, X, y, lda):
    gradient = np.zeros(len(theta))
    XT = X.T
    beta = sigmoid(X@theta) - y
    regterm = lda/len(y) * theta
    # theta_0 does not get regularized, so a 0 is substituted in its place
    regterm[0] = 0
    gradient = (1/len(y) * XT@beta).T + regterm
    return gradient

Function that implements one-vs-all classification training:

from scipy.optimize import minimize

def train_one_vs_all(X, y, opt_method):
    theta_all = np.zeros((y.max()-y.min()+1, X.shape[1]))
    for k in range(y.min(),y.max()+1):
        grdtruth = np.where(y==k, 1,0)
        results = minimize(compute_cost_regularized, theta_all[k-1,:],
                           args = (X,grdtruth,0.1),
                           method = opt_method,
                           jac = compute_gradient_regularized)
        # optimized parameters are accessible through the x attribute
        theta_optimized = results.x
        # Assign thetheta_optimized vector to the appropriate row in the
        # theta_all matrix
        theta_all[k-1,:] = theta_optimized
    return theta_all

Called the function to train the model with different optimization methods:

theta_all_optimized_cg = train_one_vs_all(X_bias, y, 'CG')  # Optimization performed using Conjugate Gradient
theta_all_optimized_bfgs = train_one_vs_all(X_bias, y, 'BFGS') # optimization performed using Broyden–Fletcher–Goldfarb–Shanno

We see that prediction results differ based on the algorithm used:

def predict_one_vs_all(X, theta):
    return np.mean(np.argmax(sigmoid(X@theta.T), axis=1)+1 == y)*100

In[16]: predict_one_vs_all(X_bias, theta_all_optimized_cg)
Out[16]: 90.319999999999993

In[17]: predict_one_vs_all(X_bias, theta_all_optimized_bfgs)
Out[17]: 96.480000000000004

For anyone wanting to get any data to try the code, they can find it in my Github as linked in this post.


texts:
I’ve built a model that classifies digits from 0-9. My dataset is tf.keras.datasets.mnist. I use softmax as the activation function for the output layer.

Q1:

The output layer should consist of 10 neurons. Each representing a digit from 0-9. But even when I change the number of output layer neurons to 20, the predictions are accurate. Seems as if even though I only need to predict 10 digits, I can have more than 10 neurons in the output layer. Why’s that?

Q2:

Also, the prediction is a list of lists right. Each list containing probabilities corresponding to 10 neurons and we take the neuron having the highest probability. My question is, suppose the 6th neuron has the highest probability, how do I know what digit(label) is assigned to it?

I’m a complete newbie to deep learning, so please dumb it down for me.


texts: How to handle performance degradation due to domain transfers in data

texts:
I am training a classification CNN on a labeled dataset $\langle x,y\rangle$. The network reaches a 0.92% accuracy rate on testing and validation sets. After this process, I pre-process the data to simulate some reasonable conditions by potting it through some process $\hat{x}=\Psi\{x\}$ and getting a new dataset $\langle\hat{x},y\rangle$ with the same labels. An example of this process is a conversion between color domains, for example, $x\in RGB$ and $\hat{x}\in grayscale$.

In the example I gave, The process may reduce the performance of the network if it using color features that are lost. In my case, no data is lost though the data may be disfigured. After applying $\Psi$ I am training a new network with a degradation in performance with only 0.81% accuracy. I was guessing that I am experiencing a concept drift, though I am not sure how to show or visualize this. Also, I do not know how to mend concept drift in NN.

Unfortunately, I cannot reveal the exact case I am working with. Is there a way to analyze or and show the data distribution has changed in the new domain?

How do I address the case described and what can I do to try and improve the performance (regardless of the specific task)?


texts:
I got a very strange error when run conda env create -f environment.yml. Due to proprietary information, I cannot share the content of environemnt.yml, except that it contains a pip section

  - pip
  - pip:
      - sqlalchemy
      - pyyaml

It seems to run OK up to the creation of the requirements.txt file, and then throw the following error

Collecting package metadata: done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "C:\Users\***\AppData\Local\Continuum\anaconda3\lib\site-packages\conda_env\installers\pip.py", line 42, in _pip_install_via_requirements
        requirements.write('\n'.join(specs))
    TypeError: sequence item 11: expected str instance, list found

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\***\AppData\Local\Continuum\anaconda3\lib\site-packages\conda\exceptions.py", line 1003, in __call__
        return func(*args, **kwargs)
      File "C:\Users\***\AppData\Local\Continuum\anaconda3\lib\site-packages\conda_env\cli\main.py", line 73, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "C:\Users\***\AppData\Local\Continuum\anaconda3\lib\site-packages\conda_env\cli\main_create.py", line 101, in execute
        installer.install(prefix, pkg_specs, args, env)
      File "C:\Users\***\AppData\Local\Continuum\anaconda3\lib\site-packages\conda_env\installers\pip.py", line 52, in _pip_install_via_requirements
        os.remove(requirements.name)
    PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\**current\\work\\dir**\\condaenv.7syuvz37.requirements.txt'

At the end, it asks me to send the report to (conda) core maintainer. Before answering it, I checked the requirements.txt file, it was generated (0 bytes, not read only, owned by me) by this process (based on creation time). I cannot delete it before answering the y/N question. Then I typed N to deny sending the report. Afterwards, I can delete the file. (This seems to suggest that it is NOT locked by others).

I also killed all Python processes using Windows task manager, and rebooted Windows, and ran this command as the first thing (in Anaconda Prompt), with no luck.

Thanks for any suggestions or insights.


texts: What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

texts:
I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/standardizing, which are not sensitive to the presence of outliers and fit this case study. Based on scikit-learn 0.24.2 study about Compare the effect of different scalers on data with outliers, it has been stated here:

If some outliers are present in the set, robust scalers or transformers are more appropriate.

I'm using CTU-13 dataset, which you can see the overview of its distributions in the dataset here.
Concerning the synthetic nature of the dataset, I need to use categorical-encoding for some features/columns to convert them into numerical values for my representation-based learning model(e.g. using image form of data as inputs for learning algorithm like CNN), what is the best normalizing method I can use fits my research case which is Anomaly/outlier/fraud detection for imbalance data at the end of the day in preprocessing stage to have the robust outlier detection model or binary classifier?
Any help/update about the state-of-the-art in this concept will be appreciated!


texts: suggestion for good online source according to Syllabus

texts:
Hi, are there some online courses e.g. some classes in Coursera?

I have difficulty in following the professor's teaching because I have a weak statistics background. I want to catch up by reading some online complementary resource!

Thank you!


texts:
Mode Analytics has a nice heatmap feature, but it is not conducive to comparing maps (only one per report).

What they do allow is data to be pulled easily into a wrapped python notebook.  And then any image in python can easily be added to a report.
So my question is: how do I recreate a heatmap on an actual map in Python?  I've checked out follium and plotly, but neither seem to have similar functionality.


texts: Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?

texts:
I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same dimensions (i.e. 102 columns and 800000 rows for both the dataframes). I would like to depict the fact visually that the 2 dataframes are very similar/have a statistically similar distribution.

I am not sure of how this can be done visually/graphically in Python. The df.describe method provides count, mean, std, min, 25 %, 50 %, 75 % and max values for the dataframes, but it is difficult to infer from in the first instance. I would prefer a visual/graphical method of any type (I am not sure if Boxplot can help?). Any suggested method along with a Minimum Workable Example would be highly appreciated. Cheers!

Presently,

# df1 is dataframe 1
# df2 is dataframe 2
df1.describe() # this gives statistical values of df1 dataframe
df2.describe() # this gives statistical values of df2 dataframe


texts: How can I create a negation of the sentence?

texts:
What are some ways that I can generate the negation of a sentence such that the output sentence reflects the negation of original sentence?

If the sentence is: That is an apple.

Then, the expected negation could be: That is not an apple.

Another example:

Sentence: Rita is so hot.
Negation: Rita is boring.


texts: What is the ideal database that allows fast cosine distance?

texts:
I'm currently trying to store many feature vectors in a database so that, upon request, I can compare an incoming feature vector against many other (if not all) stored in the db. I would need to compute the Cosine Distance and only return, for example, the first 10 closest matches. Such vector will be of size ~1000 or so.

Every request will have a feature vector and will need to run a comparison against all feature vectors belonging to a subset within the db (which will most likely be in the order of thousands of entries per subset in the worst case scenario).

Which database offers the flexibility to run such a query efficiently ?

I looked into postgres but I was wondering if there were alternatives that better fit this problem. Not sure it matters much, but I'm most likely going to be using Python.

I found this article about doing it in SQL.

EDIT: I am open to alternative solutions for this problem that are not necessarily tied to SQL.


texts: Extraction of kernel from Generative model?

texts:
I have been working on a project where I am supposed to combine a generative and a discriminative model by exploiting their geometrical feature.
There is a method where you encode the generative information in the kernel and then use this kernel in a discriminative method.
I can't find a method on how to create a generative kernel.Can someone help me how to get generative kernel?I am using EEG Eye State from UCI repository.


texts: Why does the naive bayes algorithm make the naive assumption that features are independent to each other?

texts: Naive Bayes is called naive because it makes the naive assumption that features have zero correlation with each other. They are independent of each other. Why does naive Bayes want to make such an assumption?


texts: Is XGBoost better with numeric predictors?

texts:
I have a categorical feature that I one-hot encoded and used in my XGBoost model, but it consistently underperforms as a predictor compared to the other predictors.

Then I created a new variable that contains the same kind of information that the categorical feature has...

  Imagine I'm interested in predicting price of a house and the
  categorical feature is the town that it is in, and the numerical
  feature is a feature I generated ranking the town's relative
  expensiveness, based on some prior knowledge of the towns.

TownA - 100
TownB - 40
TownC - 65
TownD - 15

Now all of a sudden the new numerical variable that was directly derived from the categorical one is outperforming. Is this because XGBoost just works better with numerical variables and maybe some predictive capability is lost when I one-hot encode the variable?


texts: Top-K vs AUC - communicating results and next steps

texts:
I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of 0.33 when trained with about 1m rows of data. When I also used the AUC metric, it gave 96%, but I'm not comfortable reporting this to my client because they will tear it apart looking for near 100% accurate predictions.
Three questions:

What is a good way of explaining the AUC to non-tech savvy people?
The top-k score tells me there is room for improvement, however the AUC tells me I can chill out and boast about what I've built. From the model below, and given that I'm using transformers tokenizer to process the inputs, is there anything blindingly obvious that I could be doing better or is worth exploring further? Or should I just accept that the AUC% is very good and leave it at that?

input = Input(shape=(args.max_seq_len,))
        vocab_size = tokenizer.vocab_size

        initializer = tf.keras.initializers.HeUniform()

        model = Embedding(vocab_size, 600, input_length=args.max_seq_len, embeddings_initializer=initializer)(input)
        model = Bidirectional(LSTM(100, return_sequences=True, dropout=0.10, kernel_initializer=initializer), merge_mode='concat')(model)

        model = Flatten()(model)
        model = Dense(500, activation='relu',kernel_initializer=initializer)(model)
        model = Dropout(0.1)(model)
        model = BatchNormalization()(model)
        model = Dense(500, activation='relu', kernel_initializer=initializer)(model)
        model = Dropout(0.1)(model)
        model = BatchNormalization()(model)
        model = Dense(500, activation='relu',kernel_initializer=initializer)(model)
        model = Dropout(0.1)(model)
        model = BatchNormalization()(model)
        output = Dense(n_classes, activation='sigmoid')(model)

        model = Model(input, output)

        optimizer = tf.keras.optimizers.Adam(1e-3)
        if args.predict:
            model = load_weights(args, model)
            # Freeze layer weights during prediction mode
            model.trainable = False

        loss = tfa.losses.SigmoidFocalCrossEntropy()

        model.compile(loss=loss, optimizer=optimizer,
                      metrics=[tf.keras.metrics.TopKCategoricalAccuracy(k=5)])

I read that AUC is great at dealing with imbalanced datasets, but if the model is trained no-differently depending on which metric I use, would you still consider using i.e. MLSMOTE anyway to seek further improvement?


texts: empty fields to be filled with previous values

texts:
In the above sample data, I have empty fields and now the task is to fill the fields with previous values.
my columns are dates and the values are a number of items present for that particular article for the specific date.
which would be a faster way to interpolate the missing fields. Any suggestions to build the function is appreciated.


texts: Why do we use Latin letters instead of Arabic letters for Arabic ASR?

texts:
Most of the work on speech recognition for the Arabic language uses Latin letters, even Arabic ASR systems with end-to-end architecture use Latin letters instead of Arabic letters. But I don't think there is a difference because each token will be represented by a number.
However when I did some tests with the same model I found that my system with Latin letters gives better results compared to the other system with Arabic letters but I can't find an explanation for that.
Thanks in advance!


texts: How to Ensemble LGBM and XGBoost Machine Learning Models?

texts:
I want to Ensemble my predictions for the StratifiedKFold of LGBM and XGBoost into another LGBM Model. I had written the following code which works when the data set has an ID_COL, but in this data set I don't have any ID_COL. It is simply based on the index. How can I ensemble the model in such case?
The Train and Test data are stored in train and test file respectively.
CODE
train_new = train[[TARGET_COL]]

test_new = test[[ID_COL]] #ID_COL is not available in data.

train_new['lgb'] = lgb_oofs
test_new['lgb'] = lgb_preds

train_new['xgb'] = xgb_oofs
test_new['xgb'] = xgb_preds

After this I pass train_new and test_new to a LGBM Model.


texts: Not enough replicas available for query at consistency all (2 required but only 1 alive)

texts:
I have a Cassandra cluster of 2 nodes and a keyspace "test" with a replication factor of 2.  The consistency level is All. I am running a spark job which dumps data in Cassandra database.
So, the thing is in one spark job I dump the data multiple times in Cassandra database. Sometimes it shows me the following error:

not enough replicas available for query at consistency all (2 required but only 1 alive)

Problem is that if I run the same job again then it might or might not give this error.

Need help to understand why this error might be coming and why such an inconsistency in the error i.e. it is coming sometimes only.


texts: Identify social users using graph embeddings

texts:
I have a social network and I want to identify the most social people in the graph. In a typical graph experiments this could have done using different network features.

However, I recently came across graph embeddings and interested in knowing if my problem can be solved using graph embeddings.

If so, please letme know how I can do that?


texts: How to extract contents by topic from a document?

texts:
I am trying to extract information from resumes. I tried the pdfminer for the text extraction. But I need to extract the contents from a resume with respect to its title.
For example:
I will be giving my educational details under a title EDUCATIONAL BACKGROUND, so I have to extract the content topic wise.
Is it possible to extract like that?
What will be the process behind that?
Is it possible to approach the problem in a segmentation manner.


texts: Any idea on how to model this specific data distribution. See attachment

texts:
I am trying to fit a curve on the data( in the attached image). I see that there is a lot of variance in the response variable for each explanatory variable value. I am not sure how to model this. I tried a polynomial curve(GAM) but at best it is explaining only around 10% of the data. I understand that this indicates that I need to add more variables to the model. However, I need to avoid going down that path.

I have tried transforming the explanatory variable but haven't found much luck. I know there is very limited things we can do with such distribution but just throwing it out to all the brilliant minds out there if some one has had to do something similar or if someone has some ideas I could try.

I have attached an image of the distribution as I am not sure how to attach a data file.

Let me know if I haven't provided enough information in my request and I will try to add more details.

Thanks,
Suhail


texts: I am working on a data-science project related on social relationship mining and need to store data in some graph databases. Initially I chose Neo4j as the database. But it seams Neo4j doesn't scale well. The alternative I found out are Titan  and oriebtDB. I have gone through this comparison on these three Databases, But I would like to get more details on  these databases. So Could some one help me in choosing the best one. Mainly I would like to compare performance, scaling, on line documentation/tutorials available, Python library support, query language complexity and graph algorithm support of these databases. Also is there any other good database options ?


texts: How to visualise GIST features of an image

texts:
I am currently working on a image classification application using deep learning algorithms (either by using GIST features or CNN). I need help in understanding the below queries.

I have extracted the GIST features of an image (Reference Link). These extracted features will be given as input to deep learning algorithm to classify the images.
Is there a way to visualize the extracted features on top of the image?

CNN or GIST, Which is better for image classification? Is GIST outdated when compared to CNN?

Thank you,
KK


texts: Optimal implementation of vanilla DQN loss in Keras

texts:
I've implemented vanilla DQN for continuous/non-images (no CNN) states in keras. But, I'm not sure if my implementation of the loss computation is optimal.

For reminder the loss is defined as : $loss=(r+\gamma max_{a'} Q(s',a')-Q(s,a))^2 \frac{1}{2}$

Here is my implementation of the network + loss function :

self.network = Sequential()
self.network.add(Dense(256, activation='relu', input_dim=input_dim))
self.network.add(Dense(32, activation='relu'))
self.network.add(Dense(output_dim))

def q_loss(data, y_pred):
    # Extract the concatenated data tensor
    action, reward, next_state, done = K.cast(data[:, 0], 'int32'), data[:, 1], data[:, 2:-1], K.cast(data[:, -1], 'bool')
    # Compute Q(s,a)
    mask = tf.one_hot(action, depth=y_pred.shape[1], dtype=tf.bool, on_value=True, off_value=False)
    q_action = tf.boolean_mask(y_pred, mask)
    # Compute the max of values at next state except if done=True
    max_q_next = K.max(self.network(next_state), axis=1) * K.cast(tf.logical_not(done), 'float32')
    # Compute the TD-error, do not propagate the gradient into the next state value
    td_error = reward + 0.95 * K.stop_gradient(max_q_next) - q_action
    # Compute the MSE
    loss = K.square(td_error) / 2
    return loss

self.network.compile(loss=q_loss, optimizer=RMSprop(lr=self.learning_rate))

Here is my train function :

def train(self):
    # Sample a batch (a tuple of narray) from the replay buffer
    # States (B*S), actions (B), rewards (B), next_states(B*S), dones (B)
    # B=batch_size and S=state_size
    states, actions, rewards, next_states, dones = self.memory.sample(self.batch_size)
    # Concatenate actions, rewards, next_states, dones together because keras loss only accept one tensor
    data = np.concatenate([np.expand_dims(actions, axis=1), np.expand_dims(rewards, axis=1), next_states, np.expand_dims(dones, axis=1)], axis=1)
    # Train on a batch
    self.network.train_on_batch(states, data)

I find the way I compute the DQN td-error and loss being ugly and probably sub-optimal. Do you have a better workaround (maybe with a mix of Keras and Tensorflow combined) ?

I've checked multiple existing Keras implementations (like link), but sadly they mostly compute the td-error outside of keras in full python/numpy, witch is IMO sub-optimal.


texts: Continuous decision trees using logistic functions

texts:
Decision tree functions are discontinuous functions of the predictors. Have continuous decision trees with smooth transitions been studied? For example, a decision tree in two variables

f(x1,x2) = c1 if x1 <  t1 and x2 <  t2
           c2 if x1 <  t1 and x2 >= t2
           c3 if x1 >= t1 and x2 <  t3
           c4 if x1 >= t1 and x2 >= t3

could be replaced by

    f(x1,x2) = c1*lg(t1-x1)*lg(t2-x2) + c2*lg(t1-x1)*lg(x2-t2) +
               c3*lg(x1-t1)*lg(t3-x2) + c4*lg(x1-t1)*lg(x2-t3)

where lg is the logistic function.


texts:
I am trying to reproduce a research paper, where it is a classification problem, and they have introduced a custom loss function that I am unable to understand.

Now I think I have to implement the equation (8) or equation (7) and I am using Tensorflow framework, but I am not able to understand the equation (8) as we have to input both Actual and Predicted Features in custom loss, but there answer is klog(lambda+1)
Similarly in equation 7, they are only using pi and pj which are actuall output i guess, but not yi and yj.
I am using Tensorflow 2 framework. Can someone help me explain these equations and some help in implementation.
I know it is not a reproducable example, but I need some direction to make it reproducable.
Edit
Link for paper:
https://ieeexplore.ieee.org/document/9148944


texts:
I am trying to convert files stored in a CSV file dataset to grayscale. I run the code which worked on my previous dataset but I now get this error:
TypeError: Image data of dtype object cannot be converted to float
The code I produced is below:
import pandas as pd
import numpy as np
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt

data_path = "pe_section_headers.csv"

data1 = pd.read_csv(data_path);

data = data1

#data = data.drop("Malware", axis=1)
#data = data.drop("Name", axis=1)
data = data.values
data = data.reshape(data.shape[0], data.shape[1], 1)
data = np.tile(data, (1, data.shape[1]))

for i in range(data.shape[0]):
    plt.tick_params(left=False, bottom=False, labelleft=False, labelbottom=False)
    plt.imshow(data[i], cmap="gray")
    plt.savefig(f"output_image_{data1.iloc[i,0]}.png")
    plt.close()

#print(data[0].shape)

And the error message I receive is:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-9f9e3cc13a98> in <module>
     21 for i in range(data.shape[0]):
     22     plt.tick_params(left=False, bottom=False, labelleft=False, labelbottom=False)
---> 23     plt.imshow(data[i], cmap="gray")
     24     plt.savefig(f"output_image_{data1.iloc[i,0]}.png")
     25     plt.close()

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, filternorm, filterrad, resample, url, data, **kwargs)
   2728         filternorm=filternorm, filterrad=filterrad, resample=resample,
   2729         url=url, **({"data": data} if data is not None else {}),
-> 2730         **kwargs)
   2731     sci(__ret)
   2732     return __ret

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/__init__.py in inner(ax, data, *args, **kwargs)
   1445     def inner(ax, *args, data=None, **kwargs):
   1446         if data is None:
-> 1447             return func(ax, *map(sanitize_sequence, args), **kwargs)
   1448
   1449         bound = new_sig.bind(ax, *args, **kwargs)

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, filternorm, filterrad, resample, url, **kwargs)
   5521                               resample=resample, **kwargs)
   5522
-> 5523         im.set_data(X)
   5524         im.set_alpha(alpha)
   5525         if im.get_clip_path() is None:

~/opt/anaconda3/envs/jacksprojectnew/lib/python3.6/site-packages/matplotlib/image.py in set_data(self, A)
    701                 not np.can_cast(self._A.dtype, float, "same_kind")):
    702             raise TypeError("Image data of dtype {} cannot be converted to "
--> 703                             "float".format(self._A.dtype))
    704
    705         if self._A.ndim == 3 and self._A.shape[-1] == 1:

TypeError: Image data of dtype object cannot be converted to float

Could anyone suggest what this means? Thanks


texts: How to apply data binning on reviews data?

texts: I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or classification the only method to bin long-text data?


texts: With one pretrained CNN model do I get only one vector of descriptors for an image?

texts:
I am given a dataset of 2D medical images. I am asked to extract image descriptors from the hidden layer of the neural network pre-trained on the ImageNet dataset. I consider to use two networks: ResNet-50 with imageNet accuracy of X and VGG-16 with imageNet accuracy of Y (X < Y).
I was wondering if with one pretrained CNN model I would only get one vector of descriptors for an image.


texts: How to forecast product/item sales next one week using Xgboost regressor

texts:
I have trained xgboost algorithm to predict the number of items sale on a given day and got pretty good results, now I would like to forecast sales ahead of one week.

I tried re-training the algorithm by adding date as one of the independent variables, however, got an exception on date type.

So can someone please suggest how to predict forecast item sales count next one week.

In-sample predictions code:

train_x,X_test,train_y,y_test=train_test_split(data,data_y,
                                               test_size=0.30)
xgb = xgboost.XGBRegressor(n_estimators=100, learning_rate=0.08, gamma=0, subsample=0.75,
     colsample_bytree=1, max_depth=7)
xgb.fit(train_x,train_y)
predictions = xgb.predict(X_test)


texts:
I want to predict the probability that an individual credit_balance is more than value N is true

I need to compare 3 classification methods: Logistic regression, Least discriminant and quadratic discriminant.

For 1 sample prediction I get predict_proba for each of them with different values, how can I decide which model is the best for my prediction?

linear regression predict_proba result is [[0.93227393 0.06772607]]
LDA predict_proba is [[0.94144572 0.05855428]]
QDA precit_proba is [[9.99999999e-01 1.24419207e-09]]

What parameters should we look at to decide which classification is the best to predict the model?


texts: What is the difference between Dilated Convolution and Deconvolution?

texts:
These two convolution operations are very common in deep learning right now.

I read about dilated convolutional layer in this paper : WAVENET: A GENERATIVE MODEL FOR RAW AUDIO

and De-convolution is in this paper : Fully Convolutional Networks for Semantic Segmentation

Both seem to up-sample the image but what is the difference?


texts:
How do I find the hyperplane (line) that touches a cloud of 2D points from one side? I used $w^Tx+b=0$ based line definition and implemented a gradient-descent based routine that would start the support line from a certain side of points, away from them. The loss function is $w^Tx_i+b$ for all points, then I take log sum. The use of hyperplanes is similar as in SVM but in this case there are no -1 or +1 labels, there are only points I am trying to approach as close as possible. I am hoping to do this from 4 sides of a point cloud to get 4 approximate / defining lines of this point cloud.

My solution was fine for some carefully selected step sizes, and iteration numbers, but the procedure can overshoot, trying to find another minima.

I am curious if there is a better way of defining this loss function. My implementation based on automatic differentiation package autograd, seen below:

import pandas as pd

def plot_sep(w):
    Q = np.array([[0, -1],[1, 0]])
    x = np.linspace(-20,20,3000)
    w2 = np.dot(Q,w[:2])
    m = w2[1]/w2[0]
    y = m*x + (-w[2]/w[1])
    plt.plot(x,y,'.')

df = pd.read_csv('in.csv')
df['1'] = 1.
df2 = np.array(df)

#w1_init = np.array([0.,1.,-6.]);eta  = 1e-3;iters = 30 # good
w1_init = np.array([0.,1.,-30.]);eta  = 1e-2;iters = 40 # bad

plt.plot(df2[:,0],df2[:,1],'.')
plot_sep(w1_init)
plt.xlim(0,15);plt.ylim(-10,40)
plt.savefig('out1.png')

import autograd.numpy as np
from autograd import elementwise_grad
from autograd import grad

def compute_loss(w1):
    tmp = np.dot(df2,w1)
    tmp2 = np.log(np.dot(tmp,tmp))
    return tmp2

gradient = grad(compute_loss)

w1 = np.copy(w1_init)

for i in range(iters):
    loss = compute_loss(w1)
    print "iteration %d: loss %f" % (i, loss)
    dw1 = gradient(w1)
    w1 += -eta*dw1
print "iteration %d: loss %f" % (i, loss)
print w1

plt.figure()
plt.plot(df2[:,0],df2[:,1],'.')
plot_sep(w1)
plt.xlim(0,15);plt.ylim(-10,40)
plt.savefig('out2.png')

Data

x,y
6.85483870968,11.875
8.06451612903,12.3958333333
7.37903225806,12.34375
8.18548387097,12.34375
8.83064516129,12.6041666667
9.43548387097,12.96875
10.0,13.0729166667
10.5241935484,13.1770833333
11.0483870968,13.2291666667
6.97580645161,10.9895833333
6.97580645161,10.4166666667
8.46774193548,10.15625
7.98387096774,10.15625
9.1935483871,10.15625
9.79838709677,10.15625
10.6048387097,10.0
11.1290322581,10.1041666667
11.1290322581,10.5208333333
10.9274193548,11.0416666667
10.9274193548,11.40625
10.9274193548,11.7708333333
10.8870967742,12.4479166667
10.0,12.7083333333
9.07258064516,11.9270833333
8.75,11.9270833333
7.86290322581,11.8229166667
7.33870967742,11.09375
7.9435483871,11.3541666667
9.15322580645,11.5104166667
9.39516129032,11.5104166667
8.50806451613,10.8854166667
9.47580645161,10.78125
9.91935483871,10.78125
10.1612903226,10.8333333333
10.1612903226,11.9270833333
9.91935483871,12.03125
9.83870967742,12.03125
9.63709677419,11.9270833333
10.564516129,11.3020833333
10.564516129,10.6770833333
9.11290322581,10.5208333333
8.02419354839,10.625


texts: Check If Answer for a Question is Correct by Similarity

texts:
Let's say in an NLP problem, I have a question and some correct answers to that question (say, 10 correct answers).

Is there a way to get a new answer as input, and "calculate" whether it is correct?


texts: Getting stock data in a discipline manner from Yahoo finance

texts:
I used the below code for downloading stock data from yahoo finance:-

import yfinance as yf
import datetime

    stocks = ["AXISBANK.NS", "HDFCBANK.NS", "ICICIBANK.NS" ,"INDUSINDBK.NS",
    "KOTAKBANK.NS",
    "SBIN.NS",
    "YESBANK.NS"]
    start = datetime.datetime(2018,1,1)
    end = datetime.datetime(2019,7,17)
    data = yf.download(stocks, start=start, end=end)
    data

I get the data in a below manner:-

I saved the data using panda:-

import pandas as pd

df = pd.DataFrame(data)

# saving the dataframe
df.to_csv('BANKING STOCK.csv')

I got the data in this format:-

But I ant my data in this format:-

Because this format is more convenient for analyzing data through SQL. How can I change the format of the data?


texts: Obtaining column values for a dataframe from another dataframe based on a common column variable

texts:
I have 2 dataframes:
df1 = pd.DataFrame({"animal": [Cat, Dog, Rat, Bull, Dog, Bull, Bull, Dog, Cat, Rat, Dog], "lifeSpan": [2, 4, 6, 0, 4, 0, 0, 4, 2, 6, 4]})
df2 = pd.Dataframe({"animal":[Bull, Cat, Rat, Dog]})
The "LifeSpan" value is unique to each i in df1["animal"]. I need a third dataframe that mentions the "lifeSpan" value in a separate column against each j in df2["animal"] based on the 'lifespan' values in df1.


texts: How do I get the latitude and longitude information contained csv file?

texts: I have already done a project on "Segmenting and Clustering Neighborhoods in Toronto" using the latitude and longitude information contained CSV file link attached with the question[http://cocl.us/Geospatial_data]. How do I obtain the latitude and longitude information of a given territory?


texts: What is the difference between Pytorch's DataParallel and DistributedDataParallel?

texts:
I am going through this imagenet example.

And, in line 88, the module DistributedDataParallel is used. When I searched for the same in the docs, I haven’t found anything. However, I found the documentation for DataParallel.

So, would like to know what is the difference between the DataParallel and DistributedDataParallel modules.


texts:
I was using proc een in SAS, for time series forecasting. I want to include two variables in by statement.
My data contains four columns:

Shop_id
Item_id
Item_price
Date

The item_price column is the one which should be forecasted. The date is in months.
I want to include both shop_id and item_id in the by statement

Is it possible ?


texts:
Is there a way to feed tensorflow tensors into a sklearn model? I have the following model to set up data compression:

with tf.name_scope('model'):   with tf.name_scope('setup'):
    token_size = X_train[0][0].shape[1]
    batch_size = 64
    dropout_1 = tf.placeholder_with_default(1. , shape=(), name='dropout_1')
    dropout_2 = tf.placeholder_with_default(1. , shape=(), name='dropout_2')
    # [None, 5, 19, 400]
    inputs = tf.placeholder(tf.float32, shape=(None, FLAGS.eval_beam_size, largest, token_size), name='inputs')
    label = tf.placeholder(tf.float32, shape=(None, FLAGS.eval_beam_size), name='label')

  with tf.name_scope('layer1'):
    hid_size_1 = 20
    # [400, 5]
    w1 = tf.Variable(tf.random_normal([token_size ,hid_size_1], stddev=0.01), name='w1')
    b1 = tf.Variable(tf.constant(0.1, shape=([hid_size_1])), name='b1')
    y1 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.tensordot(inputs, w1, 1), b1)), dropout_1, name='dropout_1_op')

    tf.summary.histogram('layer1_weights', w1)
    tf.summary.histogram('layer1_bias', b1)

  with tf.name_scope('layer_2'):
    hid_size_2 = 5   # [400, 5]
    w2 = tf.Variable(tf.random_normal([hid_size_1, hid_size_2], stddev=0.01), name='w2')
    b2 = tf.Variable(tf.constant(0.1, shape=([hid_size_2])), name='b2')
    y2 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.tensordot(y1, w2, 1), b2)), dropout_2, name='dropout_2_op')

    tf.summary.histogram('layer2_weights', w2)
    tf.summary.histogram('layer2_bias', b2)

  with tf.name_scope('layer_3'):
    hid_size_3 = 1
    w3 = tf.Variable(tf.random_normal([hid_size_2, hid_size_3], stddev=0.01), name='w3')
    b3 = tf.Variable(tf.constant(0.1, shape=([hid_size_3])), name='b3')
    y3 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.tensordot(y2, w3, 1), b3)), dropout_2, name='dropout_3_op')

    tf.summary.histogram('layer3_weights', w3)
    tf.summary.histogram('layer3_bias', b3)

with tf.name_scope('support_ops'):   summed = tf.squeeze(tf.reduce_sum(y3, axis=2), axis=-1, name='summed')

From here I would like to be able to feed the outputs into a sklearn model and perhaps leave these layers to train based on the model output. Is this possible?


texts: Removing Categorial Features in Linear Regression

texts:
This is more of a design question regarding linear regression. Here is some info on our dataset:

Our dataset has 8 features; 3 of them being categorical. We are willing to perform linear regression to fit our target data.
We have tried including all of our 8 features (categorical ones being encoded in integer) and doing the linear regression.
We have also tried taking out the categorical features and running the linear regression algorithm for each possible combination of our 3 categorical features. That of course yields in a lot of regression runs (each category has 4 possible values; so 64 to be exact).

The latter approach gave us better results. So, fixing the categorical features and creating a new dataset for each combination turned out to give better estimations.

What does this tell us about our data?

Categorical features make the data non-linear, so they should indeed be taken out? For example, there is a complex relationship between categorical feature A and categorical feature B that can't be captured in linear regression?
If I am forced to step out of linear regression, which algorithm should I try to apply? I would prefer to have one dataset with 8 features instead of having 64 datasets with 5 features. There should be an algorithm that can capture this model.


texts: Function growing faster for negative inputs than for positives

texts:
I am working on a regression problem where I want to model the loss function in a way that it "punishes" to big errors much more than small errors (so I am in the realm of exponential functions) but also in a way that is punishes a negative error much more than a positive error.
So for example:

Prediction off by +4.0: is a problem, but still ok
Prediction off by +0.5: not a big deal
Prediction off by -0.5: is a problem, but still ok
Prediction off by -4.0: is a major problem

My problem is that I cant find a good function to describe this. x squared and so do not have the higher values for negative inputs that I am looking for.
My best workaround for now is to just move the whole function to the right (x-2)^2, but there must be something better?


texts: Should I use a decision tree or logistic regression for classification?

texts:
I am working on a classification problem. I have a dataset containing equal numbers of categorical variables and continuous variables. How do I decide which technique to use, between a decision tree and logistic regression?
Is it right to assume that logistic regression will be more suitable for continuous variables and that decision trees will be more suitable for both continuous and categorical variables?


texts: Cluster documents and identify the prominent document in the cluster?

texts:
I have a set of documents as given in the example below.

doc1 = {'Science': 0.7, 'History': 0.05, 'Politics': 0.15, 'Sports': 0.1}
doc2 = {'Science': 0.3, 'History': 0.5, 'Politics': 0.1, 'Sports': 0.1}

I want to cluster the documents and identify the most prominent document within the cluster.

e.g, cluster 1 includes = {doc1, doc4, doc5. doc8} and I want to get the most prominent document that represents this cluster (e.g., doc8). (or to identify the main theme of the cluster)

Please let me know a suitable approach to achieve this :)


texts: Information about study material for time series analysis and prediction?

texts: Can anyone help me out, I want to learn time series analysis and prediction in python. I am not able to find out proper study material and syllabus. I want to get deep knowledge in this domain.


texts: Graph Neural Network - Node and Global Properties

texts: I am trying to work with Graph neural networks. I am trying to use the 3d coordinates and some othe things related to those positions as features. However the whole system also has some global properties like the spread in x,y or z dimension and few others which by already know knowledge must have some dependence on target output variable. However. I don't know how to work together with node and global properties together for graph neural networks. Can anyone help.


texts: Machine Learning Techinques that Automate Fast Fourier Transform

texts:
I have a 40k Hz time-series data of vibration, which is used to predict equipment failure. The goal here is to make a system that predicts it automatically. I am thinking of a couple of ways but not sure what the best way is to frame the problem from machine learning perspective. I'll explain how an analyst usually does it manually:

Given a 1D vibration data, we want to break it down into smaller pieces (1 second in below figure, sometimes more). And then, using Fast Fourier Transform (FFT), we can bring this chunk into frequency domain and do some analysis there. A couple of things that are analyzed include: amplitude of spikes, how often spikes appear, vibration values near frequency of interest, etc. Then we can decide whether it is anomalous or not. Finally, we do it repeatedly on different chunk of data and bring it to the final conclusion.

How do we formulate this problem using machine/deep learning? Do we need to still do FFT or can we do it with raw time-series data? What are the features and target variable like? I'd really appreciate any ideas as to how to tackle this problem.

Thanks!


texts:
I'm new with keras with tensorflow backend and I'm trying to do transfer learning with pretrained net.
The problem is that the accuracy on validation set is very high, around the 90% , but on test set the accuracy is very bad, less that 1%.
I solved the problem using opencv to read and resize image, but I'd like to understand why with keras methods I've this problem. I paste my code below.

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.xception import preprocess_input
import keras

train_val_datagen = ImageDataGenerator( validation_split=0.25, preprocessing_function=preprocess_input)

train_val_generator = train_val_datagen.flow_from_directory( # subset di allenamento
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    subset = 'training',
    seed=17)
val_train_generator = train_val_datagen.flow_from_directory( # subset di validation
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    subset = 'validation',
    seed=17)
final_train_generator = train_val_datagen.flow_from_directory( # set finale di allenamento con tutti i dati
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    seed=17)

As you can see i used Xception as pretrained-net and I choose to resize my images to adapt them to the net.

After the training I created a new Iterator for test data as follow:

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_generator = test_datagen.flow_from_directory(
    directory="./TEST/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=1, # predico una alla volta
    shuffle = False,
    class_mode=None # non ce alcuna classe di riferimento
    )
test_generator.reset()

where the preprocess function is exactly the same.

The predictions are made with the following code:

predictions = model_xcpetion.predict_generator(test_generator, 6104, verbose = 1 )

where 6104 is the number of images in test folder.
After this i've generated a csv with images name with relative categorical probabilities:

import pandas as pd
import numpy as np
df = pd.DataFrame(predictions)
cols =[('probability of' + str(i)) for i in list(range(1, 30 )) ]
df.columns =  cols
df['images'] = imNames
df.to_csv('predictions_xception_all_data.csv', sep=',')

where the columns represent the labels (1 to 29) and imNames are obtained with filenames attribute of test_generator.
Finally I generated the csv with the labels with highest probabilities value and i compute the accuracy obtaining the value that I wrote before.

The code that I used to solve is the same but I read and resize images with the following code:

width = 299
height = 299
dim = (width, height)
images = []
# for each img
resized = cv2.resize(img, dim, interpolation = cv2.INTER_AREA)
images.append(resized)

where "img" are all the imageas read with "imread_collection" of skimage.io

Thanks in advance for your help.

EDIT1: the images resized with opencv have not been processed with the preprocessing function


texts: Hyperparameter tuning in multiclass classification problem: which scoring metric?

texts:
I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, RandomForestClassifier and a GradientBoostingClassifier using a randomized search and a bayesian search.

For now, I used just accuracy for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for parameter tuning?

I found that for example recall_micro and recall_weighted yield the same results as accuracy. This should be the same for other metrics like f1_micro.

So my question is: Is the scoring relevant for tuning? I see that recall_macro leads to lower results since it doesn't take the number of samples per class into account. So which metric should I use?


texts: XGBoost:Logistic poor performance with Scaled and PCA data

texts:
Working on a data set similar to fraud but not monetary transactions. Here are the steps that I have taken on the modeling side:
Convert Some of the categorical into numerical (One hot encode)
Over Sampled using SMOTE

Scaled X_train

PCA X_train - fit_transform

on the prediction side (done the same scaled X_test, PCA transform
pca = PCA(n_components=0.95)
X_train = pca.fit_transform(X_train)

I have done GridSearch to pick optimal parameters and picked the parameter when AUC is around 0.95.
My error rate is around 70%. My understanding was PCA and Scale should help tree based algorithms.
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=0.6, eval_metric='auc',
              gamma=0.5, gpu_id=-1, importance_type='gain',
              interaction_constraints='', learning_rate=0.1, max_delta_step=0,
              max_depth=4, min_child_weight=100, missing=nan,
              monotone_constraints='()', n_estimators=100, n_jobs=16,
              num_parallel_tree=1, random_state=0, reg_alpha=0, reg_lambda=1,
              scale_pos_weight=1, subsample=1.0, tree_method='exact',
              use_label_encoder=False, validate_parameters=1, verbosity=None)
Parameters:  {'colsample_bytree': 0.6, 'gamma': 0.5, 'learning_rate': 0.1, 'max_depth': 4, 'min_child_weight': 100, 'subsample': 1.0}
Highest AUC: 0.95


texts: How are weights calculated in a feed-forward neural network before they are summed up with bias?

texts:
I have read a lot of papers and watched different videos, it seems like they explain how they are summed up with bias before entering the activation function.

What I am trying to understand is the whole flow process of data from the matrix all the way through to the output layer. What I am struggling with is how do they gain their initial weights?

I have drawn a diagram hoping to explain what I understand so far. Please let me know if I am in the right track and help me understand how weights are calculated. Many thanks in advance.

Many thanks in advance.


texts: Is there a way to use UMAP on a weighted digraph?

texts:
I know If I have a weighted graph $G$(not a directed graph), I can define a metric on its vertices:
$$d(i,j) = \text{weight of the minimum path between vertex i and j}$$
So I think It makes sense to use UMAP on distance matrix $A$ of graph $G$ that $A_{i,j} = d(i,j)$. Am I correct?
If $G$ is a directed weighted graph, How can I use UMAP?


texts: Determining first name and surname from a name pair?

texts:
Suppose I'm given a list of name pairs:

[
    [ 'John', 'Smith'],
    [ 'Alex', 'Gordon'],
    ...
]

I wish to know (or be given reasonable certainty) which column is the surname, and which is the first name, assuming that all columns are consistent, and each name pair consists of exactly one first name and one surname.

An obvious challenge would be checking name pairs where both names can be used as first names, e.g. ['Graham', 'Tyler'].

Is this model something that is reasonable to achieve?


texts: What are the best tools for manually annotating textual topic segmentation?

texts:
I checked this old question on tools for text corpus annotation and there are many good tools there, however the tools there seem not to cover my problem.

I have a linear text segmentation problem, in which a text needs to be segmented in different sections (topics). I'm building a model for it, but I have no training data. Therefore, the team decided to manually label some text, by annotating blocks in the text that represent each section. I tried some NER or POS labelling tools, but they are not very convenient for selecting several lines and paragraphs to annotate a label.

Is there a good tool for human annotation of text segmentation?


texts: is multicollinearity a problem also in classification?

texts: This might be a dumb question, but from what I'm grasping, multicollinearity seems to be uniquely a problem in regression problems, and therefore models.


texts:
I'll try my best to be understood, thank you for reading.

So this is the problem :

I'm asked to predict cancer on X-ray as early as possible. The dataset is composed of images in this form  img_patient1_week1,img_patient1_week2, img_patient1_week3
and that for a lot of different patient.
Some of them did develop cancer and we can see the tumor growing with the weeks, and some of them didn't.

I usually train a CNN for classification with images but here it's quite different since I want to predict if something is about to happen for a new patient as early as possible (week 1 or week 2) by detecting patterns while training.

Maybe I should go for a CNN and a Recurrent Neural Network to create a link between the images of the same patient ?

So I don't know how to do this or if it's the right way of handling it.

Thank you for your help,

b.


texts: Combining outputs of ridge regression models?

texts:
I am facing an issue where I have 7 sets of different variables/columns/predictors.

I am trying to predict same target variable and I want to observe the importance/effect of all the sets according to their importance in an ordered manner. (I am trying to use ridge regression models for each of the 7 individual set as I want to keep all the variables and I want to combine the output of these 7 models, each set has more than 20 variables)


texts: Does a larger action space take longer to train an RL agent?

texts:
I am playing around with the openai gym to try and better understand reinforcement learning. One agent parameter you can modify is the action space i.e. the specific actions an agent can take in an environment at each state e.g. "Left", "Right", "Up" or "Down" if the environment is a game with 4 discrete actions.

In my research, I have not found anywhere that explicitly states that an RL model, specifically PPO2, will take longer to train if the action space is larger.

All else being held the same; same data, same environment, same hyperparameters, same hardware, will a model with a larger action space (more possible actions) take longer to train (1 episode) than a model with a smaller action space?

(e.g. will an agent with 100 possible actions take longer to train 1 episode than an agent with 2 possible actions?)

Intuitively, I would have thought that the more actions an agent has, the more "choice" it has at each state, and therefore it would take longer in choosing in one of those actions. But again, I haven't found anything proving or disproving this.


texts: I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost almost always performs better than GBM. What makes XGBOOST so fast?


texts: Does gradient boosting algorithm error always decrease faster and lower on training data?

texts:
I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which results in the following loss plot:

I'd say this is pretty standard plot with boosting algorithms as XGBoost. My reasoning is that my point of interest is mostly the performance on the test set and until the XGBoost stopped training around 600th epoch due to early stopping the test-set loss was still decreasing. On the other way, overfitting is sometimes defined as a situation when train error decreases faster than test error and this is exactly what happens here. But my intuition is that decision-tree based techniques always drill down the training data (for example, trees in random forest kind of deliberately overfit the training set and it's a role of bagging to reduce the variance). I believe gradient boosting techniques are also known from drilling down the training dataset pretty deep and even with low learning rate we can't help it.
But I may be wrong. Therefore, I'd like to confirm that this situation is not really something to worry about and I do not overfit the data. I'd also like to ask what the perfect learning curves plot would look like with gradient boosting techniques?


texts: Variable selection involving mixture of numerical, high cardinal,low cardinal features

texts:
Consider a dummy dataframe:

A B C  D …. Z
1 2 as we   2
2 4 qq rr   5
4 5 tz rc   9

This dataframe has 25 independent variables and one target variable ,the independent variables are mixture of high cardinal features, numerical features and low cardinal features, and the target variable is numerical. Now I first want to select or filter variables which are helpful in predicting the target variable. Any suggestions or tips towards achieving this goal is appreciated. Hope my question is clear, if the form of question is unclear I welcome the suggestions to make correction.

What I tried so far?
I applied target mean encoding (smooth mean) on the categorical features w.r.t target variable. Then I applied random forest to know variable importance. And the weird thing is that the random forest is selecting only one feature all the time, I expected at least 3-4 meaningful variables. I tried neural networks but the result is no different , what would be reason for this? What does that mean if the algorithms only using one variable? And the test predictions are not very accurate. The RMSE is about 2.4 where the target feature usually range from 20-40 in value. Thank you for your patience on reading this.
P.S: I am using SKlearn and in python.
 I also tried Anova(on categorical features) and P value stats. This seems to be helpful. But I will be keen on to hear further options from community users


texts: Naive Bayes Predict type = 'raw' returning NA

texts:
I have build a naive bayes model for text classification.It is predicting correctly.But it is returning 'NA' in prediction results if i put 'type = raw'.i have seen some results in stackoverflow to add some noise.when i do that i am getting all A category as 0's and all B category as 1's.How can i get correct probabilities in naive bayes?

library('tm');
library('e1071');
library('SparseM');
Sample_data <- read.csv("products.csv");
traindata <- as.data.frame(Sample_data[1:60,c(1,2)]);
testdata <- as.data.frame(Sample_data[61:80,c(1,2)]);
trainvector <- as.vector(traindata$Description);
testvector <- as.vector(testdata$Description);
trainsource <- VectorSource(trainvector);
testsource <- VectorSource(testvector);
traincorpus <- Corpus(trainsource);
testcorpus <- Corpus(testsource);
traincorpus <- tm_map(traincorpus,stripWhitespace);
 traincorpus <- tm_map(traincorpus,tolower);
 traincorpus <- tm_map(traincorpus, removeWords,stopwords("english"));
traincorpus<- tm_map(traincorpus,removePunctuation);
 testcorpus <- tm_map(testcorpus,stripWhitespace);
 testcorpus <- tm_map(testcorpus,tolower);
 testcorpus <- tm_map(testcorpus, removeWords,stopwords("english"));
 testcorpus<- tm_map(testcorpus,removePunctuation);
trainmatrix <- t(TermDocumentMatrix(traincorpus));
testmatrix <- t(TermDocumentMatrix(testcorpus));
model <- naiveBayes(as.matrix(trainmatrix),as.factor(traindata$Group));
results <- predict(model,as.matrix(testmatrix))


texts: Linear and logistic regression output with same neural network

texts:
it seems like this should be a very common task, but I have not found anything useful in my research: How can I do linear and logistic regression with the same neural network?
By example, what I mean is the following: Let's say I want to train a network on pictures of faces and I want a two-dimensional output: The gender and the age of the person on the photo. So for the gender, I would want to do logistic regression, while linear regression for the age.
Should there be two different activation functions in the output layer (sigmoid and linear)? How to deal with the fact that, for convexity reasons, one usually uses MSE loss for linear and logistic loss for logistic regression?
Thanks!


texts: Why my results have time delay when I use LSTM?

texts:
I am trying to fit and test LSTM on a numeric series(like stock prices). But it seems that I always get a lag in predicted graph(Blue) with respect to real graph(red). Does anyone know why this happened? (I searched and realized this is a problem for others too).

I use Keras.sequential.LSTM library.