texts:
I have a function in which I want to represent my data like this:

Input is a column:[ 123 125 11 122 ...]
Output:  123  125
         125  11
         11   122

The function in python is like that:

def create_dataset(dataset, look_back=1):
dataX, dataY = [], []
for i in range(len(dataset)-look_back-1):
    a = dataset[i:(i+look_back), 0]
    dataX.append(a)
    dataY.append(dataset[i + look_back, 0])
return np.array(dataX), np.array(dataY)

My dataset is represented as 1-grams of integers in a csv file, so I was obliged to use the transpose () of the dataframe to use this function. The problem is that I find the dataframe size null, and the train and test data (after spliting) also empty.

the code is:

dataframe = pd.read_csv("train2.csv")

print(dataframe.shape)  # (0,150)
print("--------\n")
#dataframe= np.asarray(dataframe)
dataframe = dataframe.transpose()
print(dataframe.shape)   #(150,0)
print(dataframe.size)  # 0

dataset = dataframe.values
dataset = dataset.astype('float32')

# split into train and test sets
train_size = int(len(dataset) * 0.67)
print(len(dataset))
#print (train_size)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]
print(len(train), len(test))

print(dataset[0:train_size,:])  #[]
print(train)                    # []
print(test)                     # []

# reshape into X=t and Y=t+1
look_back = 1
trainX, trainY = create_dataset(train, look_back)
testX, testY = create_dataset(test, look_back)

Any solution please?


texts:
I have a number of features X and a target which is the revenue generated by this interaction. More often than not no revenue is generated, but if there is it can differ considerably, it has a big right tail. I've made a model to estimate the expected revenue generated by a new x. Due to the weird distribution of the revenues I have split the model up in two parts, first estimating the probability of revenue generation and then conditionally on the fact that somebody will book, predict the amount of $log($revenue$)$. The first model is trained on a dummy variable which indicates revenue and the second one is trained on a subset of the data which contains all interactions that generated revenue.

After I have these two models P (probability of generation) and R (expected revenue generated conditioned on > 0) we can calculate the expected revenue for a new x using:

$$f(x) = P(x)e^{R(x)}$$

I've done a reasonable amount of cross validation on both the models individually and they work decent, however when they are combined they severly underestimate the amount of revenue generated. I have a good enough number of rows for the dimensionality of my data and I'm using Logistic Regression for the probability (linearity assumption is somewhat reasonable, however unbiased probabilities are very important) and Gradient Boosting for revenue estimation (got the best results for the second portion).

What should I do about the underestimation of the revenue? What are alternative unbiased probablity estimators, should I train a neural network with a hidden layer for the log-loss function? Why does the combined model underestimate? Is there a good alternative to do it in one model? Which would not increase small errors into huge errors due to the multiplication and exponent?


texts:
Background: I am using CNN to predict forces acting on a circular particle in a granular medium. Based on the magnitude of the forces, particle exhibits different patterns on its surface. The images are greyscale, 64-by-64 pixels. You can see different pictures with the magnitude of the corresponding force on the x-axis attached below.

My attempt at solution: I am relatively new to deep learning and data science and decided to use a simple conv net to run a regression. My code is provided below. I tried to fit the model using adam optimizer and mse as a loss function, but it takes forever and sometimes aborts execution by itself. What could be the problem? I am running it on a PC with 8GB RAM, 1TB SSD, Intel i7 CPU, and GTX 1080 GPU.

def build_model():

model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3), strides = (1,1),
             padding = 'valid', activation='relu',
             input_shape=input_shape))

model.add(Conv2D(64, kernel_size = (3, 3), strides = (1,1),
                 padding = 'valid', activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2), strides = (2,2)))

model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(53824, activation='relu'))
model.add(Dense(53824, activation='relu'))
model.add(Dense(53824, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='linear'))

return model


texts: Standardizing features by one specific feature

texts:
I am working on a project with a dataset that looks something like the following:

      velocity   accel_amp         f_vert   tau_vert      f_pitch_filt   tau
0    3.778595      -5.796777     2.400000  32.753227      1.600000   27.844535
1    1.970611      -6.087134     2.272727  32.638705      1.704545   30.639998
2    3.581163      -6.241817     2.400000  32.850969      1.600000   30.449256
3    4.735210      -6.109532     1.400000  28.809865      1.000000  127.749313
4    5.340568      -6.614317     1.400000  20.249699      1.000000  124.549628

I was suggested to standardize the last 5 features by velocity in order to improve my PCA.
Does this simply mean to take each element in these last 5 columns, subtract the mean of the velocity column, and then divide by the standard deviation of the velocity column?

That is how I interpreted this suggestion. Is there a functionality in Python for doing this? Any suggestions or clarification would be appreciated.

Thanks.


texts: How to best determine trim values for a trimmed mean calculation

texts:
I am designing a model for a processing facility. This facility runs certain processing steps on products. Depending on certain details of the product/step, these steps are grouped together into step groups. Basically I need to create some kind of aggregate model number for these step groups to use for predictive analysis. I've chosen trimmed mean, as mean and median tended to under or overestimate the predictive model.
My question is, how can I best determine trim values for these trimmed mean calculations, using statistical methods such a mean, median, mode, n, SD, skewness, ect.? The histograms  of the step groups tend to skew right, so I'm guessing the trim would be more heavily favor the right side (like bottom 5%/top 10% for example).
Any and all information/articles about this topic would be helpful. Thanks


texts: recommendations for self-study in ML/Deep Learning/the underlying math

texts:
I am a beginner in ML, though I have completed Andrew Ng's Machine Learning course on Coursera and am soon to complete his Deep Learning specialization, also on Coursera. I'm looking for recommendations for things to study after that.

They need to be on the easier side since I am still learning. Also in addition to resources on ML or Deep Learning, I'm also interested in any resources on the underlying math. I have a book in probability and another book in calculus, though both are fairly dense. Anything lighter might be a plus.

Are there any resources you would recommend? I'm particularly interested in other courses on Coursera since that's what my company currently sponsors, though I'm interested in whatever suggestions you have to offer.


texts: Val Accuracy not increasing at all even through training loss is decreasing

texts:
I am training a model for image classification, my training accuracy is increasing and training loss is also decreasing but validation accuracy remains constant.

Here is my code:

from keras.applications.vgg19 import VGG19 model= VGG19(include_top=False, weights='imagenet',
        input_tensor=None, input_shape=(224,224,3), pooling=None, classes=1000) x=model.output

x=Conv2D(filters=1024,kernel_size=2)(x)
x=MaxPooling2D()(x)
x=Flatten()(x)
x=Dense(1024,activation='relu')(x)
x=BatchNormalization(axis=1)(x)
x=Dropout(0.8)(x)
x=Dense(64,activation='relu')(x)
x=Dense(4,activation='softmax')(x)

model = Model(inputs=model.input,outputs=x)

for layer in model.layers[:12]:
   layer.trainable = False

for layer in model.layers[12:]:
   layer.trainable=True

opt = Adam(lr=0.0001, decay=1e-6)
model.compile(loss='sparse_categorical_crossentropy',
           optimizer=opt, metrics=['accuracy'])

checkpoint_path="/content/drive/My Drive/Model/model_vgg19_6.h5"

checkpoint = ModelCheckpoint(checkpoint_path, monitor="val_acc", mode="max",
   save_best_only = True,verbose=1)

reduce_lr = ReduceLROnPlateau(monitor = 'val_acc', mode="max", factor = 0.7,
     patience = 5, verbose = 1, min_delta =0.00001)

earlystop = EarlyStopping(monitor = 'val_acc', mode="max", min_delta = 0, patience = 30, verbose = 1,
            restore_best_weights = True)

callbacks = [reduce_lr,checkpoint]

model.fit_generator(aug_train, steps_per_epoch=int((len(data_x)/128)+1), validation_data=
                (val_x,val_y), validation_steps=int((len(val_x)/128)+1), workers=-1,
                use_multiprocessing=True, shuffle=True, epochs=300, callbacks=callbacks )

I am getting a constant val_acc of 0.24541


texts: Pandas: Calculating the best seller to incorporate number of houses sold

texts:
I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller

for seller,seller_df in g:

df=df.append({'Seller':seller,'AveragePrice':seller_df.Price.mean() }, ignore_index=True)
print(seller)
    df.sort_values(by='AveragePrice',ascending=False,inplace=True)
plt.figure(figsize=(15,30))
sns.barplot(data=df.dropna()[:100],y='Seller',x='AveragePrice')

The bar plot is only for the top 100 sellers and shows who has the max average selling price.

I believe this method is inaccurate as I also should consider the amount of properties that he has sold in order to get the best seller. For e.g.: if A sold 20 houses with average 5 and B sold 200 with average 5, B should be the winner.So I have two questions:

How can I incorporate this in my code?
Can I optimise the code snippet?

I am a newbie and any help is appreciated. Please refer to this link for my Kaggle notebook. It does not contain the code posted here as I am doing the analysis locally but you can have a look at the data.

Dataset


texts: What is the meaning of Face Recognition in wild and in static?

texts: What is meant by when someone says face recognition on wild dataset and on static dataset?


texts: Am I supposed to be using Mixed Effects?

texts:
I ran a GLS random effects regression on some NBA data in Stata, and I was told that it was wrong because I didn't use mixed effects model. This may every well be the case, but I was quite confused by the explanation. Here is what I did using Stata:

Dependent variable is team win's and independent variables is different types of opponent shooting data.
Paneled data using xtset to account for the different NBA seasons
Ran robust xtreg for both fixed and random effects
Hausman test had a high chi^2, so I stayed with the GLS random effects regression.

If this is incorrect, please let me know so I can fixed this. I always thought that the mixed effects model was for logistic regressions when my dependent variable was categorical. I was told that I did not take into account this team grouping structure within your analysis, but the Group Variable was the teams.


texts: Finding a subset of predictable data within randomness

texts:
I am using the random Forrest algorithm to classify data points based upon 10 features. The accuracy score from these forests has been low (5-15% above random). I have a theory that there is a subset of highly predictable data within the dataset that shares similar features, but that the rest is essentially random - giving low accuracy meterics.  How can I test this theory?
One approach I have thought of involves clustering the data and seeing if any clusters co-localize with predictions. For example, individuals within cluster X have a 95% chance of being class Y. What do you think? Thanks so much for the advice!


texts: What does it mean: "Everything looks OK but loss won't decreases!"

texts: I have written a LSTM network. It seems all the things are OK but when I train the network, I get the same loss amount about 4.9e-4 for every iterations! What is the problem? Why my network can't decrease the loss amount?


texts: I'm trying to build a ResNet 18 model for Cifar 10 dataset, but I'm not able to fit the data dimension

texts:
At avergae pooling after the ConvNet, the error is displayed as the dimensions cannot be negative because the shape the previous output layer is 1,1,512 and on this the maxpooling cannot be done. Is it something that i did wrong in the architecture design?
def identity_block2(X,f,filters):
  f1,f2 = filters
  X_init = X

  X = Conv2D(f1,(3,3),strides=(1,1),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X = Conv2D(f1,(3,3),strides=(1,1),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X = Add()([X,X_init])
  X = Activation('relu')(X)
  return X

def conv_block2(X,f,filters,s=2):
  f1,f2 = filters
  X_init = X

  X = Conv2D(f1,(3,3),strides=(s,s),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X = Conv2D(f2,(3,3),strides=(1,1),padding='same')(X)
  X = BatchNormalization()(X)
  X = Activation('relu')(X)

  X_init = Conv2D(f1,(3,3),strides=(s,s),padding='same')(X_init)
  X_init = BatchNormalization()(X_init)

  X = Add()([X,X_init])
  X = Activation('relu')(X)]
  return X

def resnet18(input_shape,classes):
  X_input = Input(shape=input_shape)

  X = ZeroPadding2D((3,3))(X_input)
  X = Conv2D(64,(7,7),strides=(2,2))(X)
  X = MaxPooling2D((3,3),strides=(2,2))(X)

  X = conv_block2(X,3,[64,64],s=1)
  X = identity_block2(X,3,[64,64])
  X = Dropout(0.4)(X)

  X = conv_block2(X,3,[128,128],s=2)
  X = identity_block2(X,3,[128,128])
  X = Dropout(0.4)(X)

  X = conv_block2(X,3,[256,256],s=2)
  X = identity_block2(X,3,[256,256])
  X = Dropout(0.4)(X)

  X = conv_block2(X,3,[512,512],s=2)
  X = identity_block2(X,3,[512,512])
  X = Dropout(0.4)(X)

  # It shows error at this point as the dimension of the previous output layer is (1,1,512)
  # X = AveragePooling2D((2,2))(X)
  X = Flatten()(X)
  X = Dense(classes,activation='softmax')(X)

  model = Model(inputs=X_input,outputs=X,name='ResNet18')

  return model


texts: Implementation of LDA (Latent Dirichlet Allocation) for classification tasks

texts: Until now I have used LDA only for topic modelling. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.


texts: Grouping with non-sequential index (datetime) [Pandas] [Python]


texts: How to create a parquet file from a query to a mysql table

texts:
Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:

res = sh.sed(
    sh.mysql(
       '-u',
       settings_dict['USER'],
       '--password={0}'.format(settings_dict['PASSWORD']),
       '-D', settings_dict['NAME'],
       '-h', settings_dict['HOST'],
       '--port={0}'.format(settings_dict['PORT']),
       '--batch',
       '--quick',
       '--max_allowed_packet=512M',
       '-e', '{0}'.format(query)
    ),
    r's/"/\\"/g;s/\t/","/g;s/^/"/;s/$/"/;s/\n//g',
    _out=filename
)

the mid term solution with more traction is AWS Glue, but if I could have a similar function to generate parquet files instead of csv files there would be much needed big short term gains


texts: GRU - why gamma is close to 0 or 1 resolves the vanishing gradient problem?

texts:
Background
It is explained that in GRU (Gated Recurrent Unit), the gate value $\Gamma$ close to zero mitigates the vanishing gradient problem.

Coursera Sequence Model by Andrew Ng - Week 1 Gated Recurrent Unit (GRU)

because gamma can be so close to zero,  it doesn't suffer from much of a vanishing gradient problem.  Because when you say gamma so close to zero  this becomes essentially $c^{<t>}$ equals $c^{<t-1>}$ and the value of $c^{<t>}$ is maintained pretty much  exactly even across many time-steps.  So this can help significantly with the vanishing gradient problem  and therefore allow a neural network to go on even very long-range dependencies

Question
If the $\Gamma \approx 1$ and  $c^{<t>} = \tilde{c}^{<t>}$, would the back-propagation go through $tanh \left( W_c \right[c^{<t-1>}, X^{t} \left] \right)$ in which $W_c$ would be multiplied multiple times through the time? This part seems to me still the same in RNN.
If $y = x$ then the gradient will be $\frac {\partial y}{\partial x} = 1$. However, why $ W_c \left[ c^{<t-1>}, X^{t} \right] $ would not cause the vanishing or exploding gradient at back-propagation?
The GRU fixes the issue so I miss something. Please help understand.


texts:
I am trying to create a Choropleth map using the Chorolpeth widget from the Geo Add-on in Orange. However, this widget is not appearing? Any ideas?
My current version: Orange version 3.24.1 for Windows


texts: How to represent ROC curve when using Cross-Validation

texts:
I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold.  My desired output is one ROC curve with a corresponding AUC value.

One method (taken from here) is to take the mean false positivity rates (fpr) and true positivity rates (tpr) over all folds and plot the overall ROC curve using the mean tpr and fpr values. Then compute the AUC using the mean-ROC curve. However, this method does not work well when the dataset is small. Without a long explanation, my classification is a diagnosis that uses many samples for one diagnosis and thus reduces the predictions per fold to around 3-5.

The alternative method is to save the probabilities of each prediction in every fold and then construct a ROC curve after k-Fold CV and compute the AUC using this ROC curve. However, this would mean that various models, trained on different datasets are combined into one ROC curve. I don't know if this is an issue?

What is the industry standard for model evaluation reporting when using ROC and AUC combined with k-Fold Cross validation?

-feel free to edit my question.


texts:
I run a heterosexual matching making service.  I have my male clients and my female clients.  I need to pair each of my clients with their "soul mate" based on several attributes (age, interests, personality types, race, height,horoscope, etc.)

After I create all my pairings, there will be some sort of score to grade the quality of my matches.

I can't match a man with multiple women or vice versa.  I also want to minimize the number of unmatched clients.

What's the best algorithm to come up with a way to match my clients based on their attributes?

Again, this is a toy example.  My actual use case is completely different.

Edit:

The score is computed at the pair level and then summed.  I can calculate how the score changes when I swap partners by looking at the new scores of two pairs.  I do have access to the internals of the metric, but it's complicated.  I don't have any constraints, other than I'd prefer it to be fast and simple for my own sanity.


texts:
In the 2000 paper that originally described the modern LSTM (Gers et al.), the training algorithm is described as follows:

  Output units use BPTT; output gates use slightly, modified, truncated BPTT. Weights to cells, input gates, and the novel forget gates, however, use a truncated version of RTRL.

However this choice is not justified in the text.

Is there an intuitive explanation for the decision of training output gates with BPTT and the other gates with RTRL?


texts: How to find splits in data so that each split has equal weighting according to function f

texts:
I have a weight function f that outputs a numeric weighting for a sample s. I also have an ordered set of samples S where the weight of each sample s in set S varies greatly.

How can I create n splitting points so that each split is weighted approximately the same? What kind of methodology, algorithms or models could I use to achieve this?


texts: MLPClassifier threshold factor to eliminate test samples that are not in match with train data

texts:
I am using MLPClassifer example from
scikit-learn

The code for training:

from sklearn.neural_network import MLPClassifier
X = [[0., 0.], [1., 1.]]
y = [0, 1]
clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                    hidden_layer_sizes=(5, 2), random_state=1)

clf.fit(X, y)

At the predict step, we use test data [2., 2.], [-1., -2.] in
clf.predict([[2., 2.], [-1., -2.]]). The output of this function is
array([1, 0])

As we observe, the test data [2.,2.] is not in the train dataset we passed. Still, we got the closest match as label 1.

What i am trying to find is if the test data i supplied is not in the train dataset, i should print a message to user that data is not valid instead of telling him the wrong label as 1.

For instance, in knn classification, i have kneighbours function which tells the distance of my closest neighbours to the test data i supplied in a 0 to 1 scale. So, i could easily eliminate the test data samples which are highly distant from my train data samples by keeping threshold at 0.6 or 0.7.

Is there any criteria/threshold like this i could do with MLPClassifier or with any one of Incremental Classifiers mentioned here which can restrict my test samples if not present in train dataset ?

Question migrated from SO


texts: Training Error decreasing with each epoch

texts:
I am trying to train a VGG-19 neural network on STL-10 dataset containing 5000 images(500 for each class). And the number of output classes is 10.

I have made no changes to the architecture except that I reduced the size of fully-connected layer from 4096 to 2048, keeping the dropout(0.5) same

[The reason behind doing this is that, since the number of training images is less, so to avoid over-fitting, I decreased the size of fully-connected layer, but I don't know if this is the right thing to do].

I have also used Adam optimiser(learning rate = 0.001) instead of SGD as mentioned in the paper.

I have only run the code for only 4 epochs. And I have observed that although the cost is decreasing by very small amount, but the training accuracy is decreasing.

After the 1st epoch, cost:: 2.304091 and training accuracy:: 11.99%

After the 2nd epoch, cost:: 2.303365 and training accuracy:: 11.249%

After the 3rd epoch, cost:: 2.301936 and training accuracy:: 10.5625%

After the 4th epoch, cost:: 2.30415 and training accuracy:: 8.1249%

I want to know, that is this behavior natural or is it due to some fault in the changes I made in the architecture?
Or using less number of layers would have been better? (For example VGG-16)

(This is my first hands-on neural network experience)


texts: How to limit a Julia notebook to a single core?

texts:
I'm running a Julia 1.4 notebook, via Jupyter lab. I need to limit the notebook to a single logical CPU core, on average.
In Python I use the following incantation to restrict Numpy's CPU usage:
from threadpoolctl import threadpool_info,threadpool_limits
with threadpool_limits(limits=1, user_api='blas'):
    >> code here <<

Would anyone happen to know whether its possible to similarly limit Julia's CPU usage from within a notebook?


texts: Class asks me to give self for Naive Bayes Model python

texts:
I try to use the following code but when I try to use fit function with my X_train and y_train,
I get the following error:

fit() missing 1 required positional argument: 'self'

I do not know much about classes but I know it should not ask for self. I found somethings about instantiation but could not figure it out.

class BernoulliNB(object):
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def fit(self, X, y):
        count_sample = X.shape[0]
        # group by class
        separated = [[x for x, t in zip(X, y) if t == c] for c in np.unique(y)]
        # class prior
        self.class_log_prior_ = [np.log(len(i) / count_sample) for i in separated]
        # count of each word
        count = np.array([np.array(i).sum(axis=0) for i in separated]) + self.alpha

        smoothing = 2 * self.alpha
        # number of documents in each class + smoothing
        n_doc = np.array([len(i) + smoothing for i in separated])
        print(n_doc)

    def predict(self, X):
        return np.argmax(self.predict_log_proba(X), axis=1)

when I try

b = BernoulliNB()
b.fit(b, X_train,y_train)

this time, I receive

fit() takes 3 positional arguments but 4 were given

Then I changed it to

BernoulliNB().fit(X_train,y_train)

but this time, this error occurs:

The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().


texts: Where to find height dataset, or datasets in General

texts:
Hi there smart people,

I am new to data  Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest.

For example, I wanted to build a simple program that takes a person's height and predicts the likely height of his/her child.
(greetings from Galton)

But even for this relatively famous topic I am unable to find data.

Are there any good websited or something that list a lot of datasets for specific topics?  A database of datasets, so to speak.

I mean, I am on the Internet, it should not be that difficult to find some data ;)


texts:
I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs.
Here is my data table

games<-10
game_results<-as.data.frame(matrix(NA,games,3))
install.packages("data.table")
library(data.table)
setnames(game_results,c("V1","V2","V3"),c("winner","round","winnerscore"))
game_results


texts: Increase Specificity of a model using SMOTE arguments from DMwR package in R when training data is unbalanced

texts:
I'm working on a binary classification problem and training data which I'm using is unbalanced.

I used SMOTE function from DMwR package in R to balance the data and then used gbm to train on the data.

I used different values of arguments perc.over and perc.under and got following results:

      Sensitivity Specificity perc.over perc.under
 1:   0.5778939 0.759237187       100         90
 2:   0.8529533 0.425506555       200         90
 3:   0.7081825 0.632896305       100        100
 4:   0.9794145 0.103694875       400        100
 5:   0.9840695 0.070321812       600        100
 6:   0.9920348 0.050059595       800        100
 7:   0.9993276 0.004767580      1000        100
 8:   0.9933278 0.028605483      1200        100
 9:   0.9129513 0.270560191       200        200
10:   0.9997931 0.001191895      1000        300

So, values in first row gave best Specificity.
I'm more interested in increasing Specificity of a model. Is there a general thumb rule to get best possible values of perc.over and perc.under for a specific data set?
So that model would be just trained once using best values of  perc.over and perc.under.


texts: what make lightGBM run faster than XGBoost?

texts: I am curious on what differences in implementation allow speed up of lightGBM over XGBoost, some times up to magnitude of orders.


texts: Is multi-device trajectory prediction possible using only classical machine learning?

texts:
So I have a dataset of trajectories of multiple devices. Each device has trajectory IDs (multiple trajectories), where each trajectory consists of start point (x, y) and end point (x, y), start time, and end time. Basically, each device is said to stop when it doesn't move for some time. When they move again, it's another trajectory.
The task is to predict the END end point (the x and y for the last trajectory) for each device, so it's a multi-label regression problem. The problem is each device will travel inherently different path, so I think I have to train a model for each device. I don't think it'll work since I'll have to predict entirely different set of devices, but let's put that aside for a moment. The main problem I'm having is that in the training data, some devices only have ONE trajectory and almost half of the devices have a count of 5 trajectories, so I think my solution is out.
How would you suggest to approach this problem using classical machine learning, assuming it's possible? If it's not, what NN architecture(s) should I consider? I'm not yet familiar with NNs but if it's the only way to do it, I'll try to quickly learn about the necessary architectures.


texts:
In the lecture notes of CS231n, it says (emphasis mine)

  ... There are three major sources of memory to keep track of:

  From the intermediate volume sizes: These are the raw number of activations at every layer of the ConvNet, and also their gradients (of equal size). Usually, most of the activations are on the earlier layers of a ConvNet (i.e. first Conv Layers). These are kept around because they are needed for backpropagation, but a clever implementation that runs a ConvNet only at test time could in principle reduce this by a huge amount, by only storing the current activations at any layer and discarding the previous activations on layers below.
  ......

Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?

I guess output is the top layer. This is because we only need the gradients of the next (closer to output) layer to compute those of the current layer. The exact activations of the next layer are not required, so we can discard them after the corresponding gradients have been computed. This is possible because we compute gradients backwards (last layers first, first layers last).


texts: Python, Tkinter, Application packaging and distributing with custom packages

texts:
I have built an application on Tkinter in Python 3 and I want to package that application with all the dependent packages.

I would want to build the .exe application of my python script that installs python 3, some of the packages/dependencies, and install my python script as a .exe. I have heard of py2exe, but is that suggested?

How shall I have this and which software is recommended for it?

I do not have experience in packing and distributing software.


texts: How should I construct a binary classifier for thousands of positive data and millions of unlabeled data?

texts:
So far, I have stumbled upon many advices and papers on PU Learning and Unary classification.

TLDR: Does anyone have suggestions for specific algorithm or implementation for labeled data of only one class and unlabeled data that can be from either classes? And I'm unsure what is the proportion of Class A to B that exists within the unlabeled data.

The simplest answer I have found has been one-class SVM (Binary semi-supervised classification with positive only and unlabeled data set), but I have so many unlabeled examples compared to how many labeled ones I can find. And I am unsure if either the positive class or negative class are rare enough for anomaly detection.

One of the other suggested methods is the two-step process where I can figure out a set of reliable negative class data, but I cannot really identify a set of data as reliably negative (https://www.cs.uic.edu/~liub/publications/ICDM-03.pdf).

And another method suggests a weighted SVM (http://users.csc.tntech.edu/~weberle/Spring2011/CSC6910/Papers/posonly.pdf), but I am unsure if I can make the same assumption as the authors in that my positive data is a random subset of all the positive data, as I used a criteria to figure out which ones were positive, so I assume there is bias in the labeled data.

Overall, I have a lot of labeled data of positive class, that is to say the data of what I am looking for, but then I have many more unlabeled data. (Though in a way, the labeled data could also be considered data of a negative class.) And I am unsure what proportion of positive data and negative data exists within the unlabeled data, as there could be an equal distribution between the two classes. Or who knows, maybe data of the positive class could be rarer than the data of the negative class.


texts: Is Neural Style Transfer considered supervised or unsupervised learning?

texts:
As far as I understand it, the Neural Style Transfer uses a content image and a style image, and generate a new image based on the two images.

It tries to find a set of pixel values such that the cost function J(C, S) is minimized. It does not have any labels associated in advance, but it has an output (generated image) that should be the target of the learning.

However, I'm not sure if this is considered supervised or unsupervised learning. Which does it belong to?


texts: Conversion of Interval data to ordinal data

texts:
I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both.
Lets say the three participants finished the race in 45 seconds,35 seconds and 30 seconds, this is a interval data since there is fixed order and 0 has no meaning and if we convert it to ordinal 30s becomes first 35s second and 45 second third,according to this option a perfectly fits. What am i missing here?


texts: Is time series forecasting possible with a transformer?

texts:
For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then predict another univariate time series M steps into the future.
I started out by following the "Attention is all you need" paper but since this paper is meant for NLP I had to make some changes. Instead of embedding each word to a random point in d_model-dimensional space I use a linear layer embed the data. I've also tried using a nn.Conv1d layer with a kernel size of 1 to embed, but these approaches fail to make a non-linear prediction of the data and instead only predict a straight line through the average of the data.
First I though that the problem was my implementation of the transformer, but even when I use Pytorch' build in nn.Transformer module I get the same results. I then tried different types of positional encoding like the "Time2Vec" paper that approximates the data by using different sinus functions.
I feel like I've tried a lot of different things to make this transformer work but to no avail. So my question is, do transformers alone work for multistep forecasting with univariate data. And if so are there any articles, papers, repositories etc. that forecasts time series data with succes? If not which approach should I take going forwards to see if I can get my transformer to work.


texts:
I create a neural network with multi-label classification using MSE.
Now I want to use Mixup. Do I need two loss functions (for each target one) or is the result the same if I just combine the two targets like this?
 target = t * target1 + (1-t)* target2


texts: is there big difference between data Science , big Data and database? i am confused in these three can anyone help me to out of this confusion?


texts: Is there an interface for bootstrap sampling in rattle in R?

texts: I am using rattle in R for predictive models and am trying to see whether there is a difference in different sampling methods. The split function at the start of rattle for splitting into training and testing (optional validation) I take it is split validation. Is there a way to do bootstrap validation and cross validation directly in rattle?


texts: How is the target_f updated in the Keras solution to the Deep Q-learning Cartpole/Gym algorithm?

texts:
There's a popular solution to the CartPole game using Keras and Deep Q-Learning:
https://keon.github.io/deep-q-learning/

But there's a line of code that's confusing, this same question has been asked in the same article and many people are confused but there's no a complete answer.

They are basically creating a main network but also a target network to try to approximate the Q function.

In this part of the code they are replaying from the buffer to train the target network:

# Sample minibatch from the memory
minibatch = random.sample(self.memory, batch_size)

# Extract informations from each memory
for state, action, reward, next_state, done in minibatch:

# if done, make our target reward
target = reward

if not done:
  # predict the future discounted reward
  target = reward + self.gamma * \
           np.amax(self.model.predict(next_state)[0])

# make the agent to approximately map
# the current state to future discounted reward
# We'll call that target_f
target_f = self.model.predict(state)
target_f[0][action] = target

# Train the Neural Net with the state and target_f
self.model.fit(state, target_f, epochs=1, verbose=0)

What I can't understand is this line:

target_f[0][action] = target

In terms of code, the predict function is returning a numpy array of arrays, like this one for example:

[[-0.2635497   0.03837822]]

Writing target_f[0] to access the first predicted action is understandable, but why are they using the [action]?

Thank you very much for the help!


texts: How regularization parameter in SVM affects hyperplane parameters

texts:
While learning the SVM classification I came across the regularization parameter $\lambda$:
$F(w,b) = \left\lVert w\right\lVert_2^2 +\lambda \sum_{i=1}^n max(0,1-y_i(w^Tx_i +b)).$
So from what I understand, the main point of SVM is to find the hyperplane $\{w^Tx+b=0\}$separating 2 classes. How does $\lambda$ affects the hyperplane parameters w and b? I tried to plot the 2D case, but still can't get it. Can anyone give the interpretation in this case? Thanks.


texts: Unequal length observation sequences when training Hidden Markov model

texts:
I want to train a sequence classifier with Hidden Markov Model. The length of observation sequences is not fixed. I tried some HMM packages such as Matlab's HMM toolbox and Kevin Murphy's library. All of them seem to require the user to specify the size of transition probability matrix and emission probability matrix.

I understand that for a Hidden Markov Model (HMM), the sizes of the transition probability matrix and emission probability matrix are dependent on the number of hidden states and the length of observation sequence.

For example if:

states = ('Rainy','Sunny')
observations = ('walk', 'shop', 'clean')

The number of states is 2, the length of observation is 3. Then,
transition probability would be a 2x2 matrix. Whereas,
emission probability would be a 2x3 matrix.

What if the length of observation sequence is not fixed?

For example:

observation 1 = ('walk', 'shop', 'clean')
observation 2 = ('walk', 'shop', 'clean','eat pizza')
observation 3 = ('walk', 'shop', 'clean','drink beer','eat pizza')
...so on

What's the size of emission probability matrix in this case? Or can I just make the observation sequence the same length by padding with zeros?


texts:
We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data involved, ideally I'd want to run the computation in the same location as the data rather than shifting the data around, but this doesn't seem possible with Redshift. I've looked at MADlib, but this is not an option as Redshift does not support UDFs (which MADlib requires). I'm currently looking at shifting the data over to EMR and processing it with the Apache Spark machine learning library (or maybe H20, or Mahout, or whatever). So my questions are:

is there a better way?
if not, how should I make the data accessible to Spark? The options I've identified so far include: use Sqoop to load it into HDFS, use DBInputFormat, do a Redshift export to S3 and have Spark grab it from there. What are the pros/cons for these different approaches (and any others) when using Spark?

Note that this is off-line batch learning, but we'd like to be able to do this as quickly as possible so that we can iterate experiments quickly.


texts: I have a list of sentences and a 10 dimensional embedding for each of the sentence.   I am trying to visualize these embeddings so that i can see if several sentence embeddings form a cluster such that these have similar embeddings which in turn mean similar context's. How can i do this in python? Also, i am wondering if there's any good metric or method using which i can form cluster's or calculate similarity between sentences that can tell me how close two sentences are?


texts:
I have a set of unstructured data consisting command output logs for different operating systems like Unix, Windows, etc.

For example:

  Releasing version 0.0.1 for Stackoverflow, on 01/01/2019. The coverage is 99% and build is passed.

(This is just an example, and not related to actual use case). This output is different for different operating systems. I want to perform tagging on this data

For the case above, the output should be:

  Releasing version VERSION_NUMBER for PRODUCT_NAME, on releaseDate. The TEST_TYPE is TEST_VALUE and TEST_TYPE is TEST_VALUE

Here, some words are replaced with their corresponding sample tags.

I have studied techniques like POS tagging, NER, LSTM, but I don't know which one is suitable for this particular problem. How can I gather data from raw output and how to apply those techniques here.

Thanks to everyone who is willing to help me with this.


texts: How to define quadratic weighted kappa as eval_metric in catboost classifier

texts:
I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric.

Catboost already has WKappa as an eval_metric but it is linearly weighted variant, not quadratic.

clf = CatBoostClassifier(objective='MultiClass',
eval_metric=q_weighted_kappa(),
learning_rate=0.01,
n_estimators=750,
od_type="Iter",
depth=35,
early_stopping_rounds=500,
random_seed=42,

)

I need help to define the function q_weighted_kappa() that I can use as specified above.


texts: Applying and Visualizing k means clustering on a data set that has 9 features

texts:
I had a data set of images that I have extracted 9 numerical features that I want to apply k means clustering or hierarchical clustering to. I'm just not sure how to go about it. The tutorials I have read all only have 2 or 3 features to them, so it's easy to apply and graph. Any help would be appreciated

Here's the code I have so far:

numpyArr = df.values

ms = MeanShift()
ms.fit(numpyArr)
labels = ms.labels_
cluster_centers = ms.cluster_centers_
n_clusters_ = len(np.unique(labels))
print("Number of estimated clusters:", n_clusters_)


texts: How to apply Stacking cross validation for time-series data?

texts:
Normally stacking algorithm uses K-fold cross validation technique to predict oof validation  that used for level 2 prediction.

In case of time-series data (say stock movement prediction), K-fold cross validation can't be used and time-series validation (one suggested on sklearn lib) is suitable to evaluate the model performance. In this case no prediction shall be made on first fold and no training shall be made on last fold. How do we use stacking algorithm cross validation technique for time-series data?


texts: In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

texts:
When learning Add-1 smoothing, I found that somehow we're adding 1 to each word in our vocabulary but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me throw an example to explain.
Example:
Assume we've a corpus of three sentences: "John read Moby Dick", "Mary read a different book", and "She read a book by Cher".
After training our bi-gram model on this corpus of three sentences, we need to evaluate the probability of a sentence "John read a book", i.e. to find $P(John\; read\; a\; book)$
To differentiate John appearing anywhere in a sentence from it's appearance at the beginning, and likewise for book appearing at the end, we rather try to find $P(<s>John\; read\; a\; book<\backslash s>)$ after introducing two more words $<s>$ and $<\backslash s>$,  indicating start of a sentence, and end of a sentence respectively.
Finally, we arrive at the

$P(<s>John\; read\; a\; book<\backslash s>)$ as
$P(John|<s>)P(read|John)P(a|read)P(book|a)P(<\backslash s>|book)=\frac{1}{3}\frac{1}{1}\frac{2}{3}\frac{1}{2}\frac{1}{2}$

My Question:
Now, to find $P(Cher\; read\; a\; book)$, using Add-1 smoothing (Laplace smoothing) shouldn't we add the word 'Cher' that appears first in a sentence? And to that, we must add $<s>$  and $<\backslash s>$ in our vocabulary. With this, our calculation becomes

$P(Cher|<s>)P(read|Cher)P(a|read)P(book|a)P(<\backslash s>|book)=\frac{0+1}{3+13}\frac{0+1}{1+13}\frac{2+1}{3+13}\frac{1+1}{2+13}\frac{1+1}{2+13}$

The 13 added to each numerator is due to the unique word count of the vocabulary which has 11 English words from our 3-sentence corpus plus 2 tokens - start and end of a sentence. In few places, I see 11 is added instead of 13 to the numerator, wondering what I'm missing here!


texts: What is the purpose of Logit function? At what stage of model building process this logit function is used?

texts:
We have two prominent functions (or we can say equations) in logistic regression algorithm: 1. Logistic regression function. 2. Logit function.

I would like to know: a. Which of these equation(s) is/are used in the logistic regression model building process? b. At what stage of model building process which of these equation(s) is/are used?
I know that logit function is used to transform probability values (which range b/w 0 and 1) to real number values (which range b/w -Inf to +Inf). I would like to know the real purpose of logit function in logistic regression modeling process.

Here are few queries which are directly related to the purpose of logit function in Logistic regression modeling:

Has Logit function (i.e. Logit equation LN(P/1-P)) being derived from Logistic Regression equation or its the other way around?
What is the purpose of Logit equation in logistic regression equation? How logit function is used in Logistic regression algorithm? Reason for asking this question will get clear after going through point no. 3 & 4.
Upon building a logistic regression model, we get model coefficients. When we substitute these model coefficients and respective predictor values into the logistic regression equation, we get probability value of being default class (same as the values returned by predict()). a) Does this mean that estimated model coefficient values are determined based on the probability values (computed using logistic regression equation not logit equation) which will be inputed to the likelihood function to determine if it maximizes it or not? If this understanding is correct then, where the logit function is used in the entire process of model building.
Assume that - "Neither logit function is used during model building not during predicting the values". If this is the case then why do we give importance to logit function which is used to map probability values to real number values (ranging between -Inf to +Inf). a) Where exactly the logit function is used in the entire logistic regression model buidling process? Is it while estimating the model coefficients?
The model coefficient estimates that we see upon running summary(lr_model) are determined using linear form of logistic regression equation (logit equation) or the actual logistic regression equation?

It would be great if someone clarifies my doubts.


texts:
Given a matrix A, I want to complete Multidimensional Scaling by hand, instead of using any given R functions.

As such, I have calculated the centered matrix B with the following code:

 n<-nrow(A)
 id<-diag(n)
 e<-diag(id)
 H <- id - (1/n)*e %*% etranspose
 B <- (-1/2)* H %*% A %*% H

My question is: how can I use my B matrix to complete multidimensional scaling on my A matrix, without the cmdscale function or anything along those lines?


texts: Finding TN,FN, TP, and FN for arrays using confusion matrix


texts: How to deal with string labels in multi-class classification with keras?

texts:
I am newbie on machine learning and keras and now working a multi-class image classification problem using keras. The input is tagged image. After some pre-processing, the training data is represented in Python list as:
[["dog", "path/to/dog/imageX.jpg"],["cat", "path/to/cat/imageX.jpg"],
 ["bird", "path/to/cat/imageX.jpg"]]

the "dog", "cat", and "bird" are the class labels. I think one-hot encoding should be used for this problem but I am not very clear on how to deal it with these string labels. I've tried sklearn's LabelEncoder() in this way:
encoder = LabelEncoder()
trafomed_label = encoder.fit_transform(["dog", "cat", "bird"])
print(trafomed_label)

And the output is [2 1 0], which is different that my expectation output of somthing like [[1,0,0],[0,1,0],[0,0,1]]. It can be done with some coding, but I'd like to know if there is some "standard" or "traditional" way to deal with it?


texts: Dealing with categorical variables in Isolation Forest

texts:
Isolation Forest is widely used when dealing with outlier/anomaly detection when we have no labels. The theory behind is that making random split at random points and counting how many splits you do to isolate a feature will help you determine if an instance is or not an outlier.

I have categorical features and I am not sure how to deal with them:

Label Encoding: Will misrepresent the data in euclidean space.
One Hot Encoding: Will give me more features and since the source code first selects the columns and then the values, it will give a non-realistic probability for my algorithm to select the one hot encoded
Target Encoding wont work since we have no target

How to properly encode categorical features in Isolation Forest?
Could we encode categorical features  in a space that suits the algorithm


texts: What does anneal mean in the context of machine learning?

texts:
An article released by Open AI gives an overview of how Open AI Five works. There is a paragraph in the article stating:

  Our agent is trained to maximize the exponentially decayed sum of future rewards, weighted by an exponential decay factor called γ. During the latest training run of OpenAI Five, we annealed γ from 0.998 (valuing future rewards with a half-life of 46 seconds) to 0.9997 (valuing future rewards with a half-life of five minutes).

Does annealing in this context mean the network found through training that γ was better as 0.9997? How would this be determined?

My limited understanding of the topic led me to the following assumption on how γ was annealed: Different versions of the network were trained for a given amount of time using different versions of γ. Then those different versions of the network played against each other or their true skill scored were compared to determine the ideal value of γ.


texts: No loss decay while learning Neural Network for X-OR operation in torch

texts:
I have implemented a simple 1 hidden layer feed forward neural network in torch to learn X-OR operation. Below is my code:

require 'torch'
require 'nn'

m = nn.Sequential()
m:add(nn.Linear(2,2))
m:add(nn.Linear(2,1))
m:add(nn.Sigmoid())

torch.manualSeed(1)

m.modules[1].weights = torch.rand(2,2)
m.modules[2].weights = torch.rand(2,1)

--print(parax_m)

criterion = nn.BCECriterion()

inputs = torch.Tensor(4,2)
inputs[1][1] = 0
inputs[1][2] = 0

inputs[2][1] = 0
inputs[2][2] = 1

inputs[3][1] = 1
inputs[3][2] = 0

inputs[4][1] = 1
inputs[4][2] = 1

targets = torch.Tensor(4,1)
targets[1][1] = 0
targets[2][1] = 1
targets[3][1] = 1
targets[4][1] = 0

function trainEpoch(m,criterion,inputs,targets)
    for i=1,inputs:size(1) do
        local input = inputs[i]
        local target = targets[i]
        local output = m:forward(input)
        --print(output)
        local loss = criterion:forward(output,target)
        print(loss)

            -- backward
        local gradOutput = criterion:backward(output,target)
        m:zeroGradParameters()
        local gradInput = m:backward(input,gradOutput)
        --update
        --module:updateGradParameters(0.9) -- momentum (require dpnn)
        m:updateParameters(0.01) -- W = W -0.1*dL/dW
    end
end

for i=1,10000 do
    trainEpoch(m,criterion,inputs,targets)
end

-- prediciton
testinput = torch.Tensor(4,2)
testinput[1][1] = 0
testinput[1][2] = 0

testinput[2][1] = 0
testinput[2][2] = 1

testinput[3][1] = 1
testinput[3][2] = 0

testinput[4][1] = 1
testinput[4][2] = 1

for i=1,testinput:size(1) do
    local output = m:forward(testinput[i])
    print(output)
end

When I run the above code, there is no decay in loss(almost same in all iterations) so it does not predict the correct output. Can anyone help me to find the mistakes what I am doing wrong here?

I have also tried with different manual seed value, different initialisation of weights but still loss remains same in all iterations.


texts: Train object detection without annotated data/bounding boxes

texts:
From what I can see most object detection NNs (Fast(er) R-CNN, YOLO etc) are trained on data including bounding boxes indicating where in the picture the objects are localised.

Are there algos that simply take the full picture + label annotations, and then on top of determining whether an image contain certain object(s) also indirectly

learn to understand the appropriate bounding box(es) for objects?


texts: Implementing Batch normalisation in Neural network

texts:
I have implemented my own mini neural network program1.
Currently, it does not have batch updates, it only updates the parameters by simple backpropagation using SGD after each forward pass. I was trying to implement batch updates and batch normalisation2.

1) For simple batch updates, instead of updating parameters each time, for each image of the batch size of 'n' I should backpropagate and accumulate the deltas for all the parameters and finally update them once after the end of the batch.

2)For batch normalisation (BN), I went through the paper and I am sort of the clear with the idea but I am confused regarding how to implement it. Generally, I would multiply the matrices in the net one after the other for a single image to get final input, but with BN, do I need to feed forward for all the images in the batch till the first layer, then normalise the values, then fwd pass these values till second layer, then normalise again, and so on?
Once I reach the final layer, should I backpropagate the error for the corresponding input-output pair and update the parameters immediately as fwd pass for all the images in the batch has been done already?

Going by the way I have described, it seems to require a lot of parameter tracking throughout the batch.
It will be helpful if you can point out a better way to do it or anything that I have misunderstood so far.


texts:
I got the following warning messages when calling the 'model.fit_generator' function on keras:

Warning_1: Anaconda3\lib\site-packages\PIL\Image.py:968: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
Warning_2: Anaconda3\lib\site-packages\PIL\Image.py:2685: UserWarning: image file could not be identified because WEBP support not installed

What these messages mean and how to get rid of them?


texts: How can i use Hellinger Distance on array of different length?

texts:
I have to use Hellinger distance to compare arrays that are not the same length.

How do you do this correctly? Putting a zero in the missing fields for the shorter array does not sound like the best method to me.

Some more info on my data:

Most array dimensions are (1,58), but a some others are (1,28). Arrays contain numbers from 1 to 3.

Example:

Array1=[1 1 3 2 3]

Array2=[2 3 1 1]

One possible solution: newArray2=[2 3 1 1 0]

Is possible to use Hellinger Distance in this case? Is there any other distance function that could solve my problem?

I'm using Helliger in K-means because it's what the author of a paper I'm reading used. So, I would like to solve this issue using Hellinger.

Thanks.


texts: An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

texts:
I would like to use ANNs for my problem, but the issue is my inputs and outputs node numbers are not fixed.
I did some google searches before asking my question and found that the RNN may help me with my problem. But, all examples which I've found somehow have a defined number of input and output nodes.
So, I'm looking for a strategy, how to make it real or at least some examples, preferable in Keras or PyTorch.
More details about my issue:
I have two inputs lists, where the length of the first one is fixed and equals two, e.g.:
in_1 = [2,2]

but the length of the second list is flexible, the length can be from three to inf, e.g.:
in_2 = [1,1,2,2]

or
in_2 = [1,1,1,2,2,2,3,3,3]

Also, input lists depend on each other. The first list shows the dimension of the output list. So if in_1 = [2,2], means the output must have a possibility to be reshaped to [2,2] form.
Currently, I'm thinking to combine two input list into one:
in = in_1 + in_2 = [2, 2, 1, 1, 2, 2]

Moreover, the output has the same length as the in_2 list, e.g.:
if input lists are:
in_1 = [2, 2]
in_2 = [1, 1, 2, 2]

Output should be:
out = [1, 2, 1, 2]

Any ideas are welcome!


texts: How does the Backpropagation through time work?

texts:
I have to write a paper on LSTMs and I want to explain why LSTMs exist in the first place. According to some papers and books because usual RNNs had problems with vanishing gradients and the LSTM has not.
I would like to understand that in detail, but the backpropagation through time gradient descent is really hard to understand for me.

If I have a time series with the length 3 (as an simple example), I can calculate 3 outputs. Therefore I can calculate 3 Losses: L1, L2, L3

Let L = L1 + L2 + L3

Now, where do I start my backpropagation? Do I apply it to all outputs or only the last one? Do I always apply the accumulated loss L?

I hope you understand my question!


texts:
How to use machine learning algorithms to predict class of single values/words ?

 Use-case : I have a list of different labels and their values.

 eg :
 Labels [Colour, Height, Weight, Location, Width etc]
 Values [red, green, 10 cm, 80 kg, usa, uk etc]

Now given a sentence like "color red height 10 usa 80 kg", what is the best way to tokenize them as Labels and Values?

Input : color red height 10 usa 80 kg"
Output:
     Labels [color, height]
     Values [red, 10, usa, 80kg]

Note : there is no delimiters in the given sentence,
       values and labels are not fixed,
       values can occur without label in a sentence.

Once Values and Labels are detected,need to map them as pairs.

{
  color  : red
  height : 10
  weight : 80 kg
  location : usa,uk
}

Currently I am using a lookup list for mapping labels and values and this list need to be updated whenever a new label or value is found. Is this the best way to solve ?

Can this problem be solved using ML ?


texts: Time Decay Formula for Conversion Attribution

texts:
Question:

What is the Time Decay formula that web analytics packages use to distribute credit across the multiple sessions associated with a user before a conversion?

Context:

All web analytics packages like Google Analytics rely on the concept of:

USERS visitors, ie unique cookied browsers. Users have many...
SESSIONS visits, ie sets of pageviews within close time proximity.
CONVERSIONS  a success event such as a sign-up which can occur during a session.

These analytics packages allow you to distribute the credit of a single conversion across all the preceding sessions because each played a part in leading the user to conversion. One such model is Time Decay, which gives a diminishing share of credit to sessions further from the conversion.

Time Decay Visual: (1st session) ▁ ▂ ▃  ▅ ▆ (conversion)

more info on attribution models.

Sample Data:

I am assuming the data required would be:

n - the total number of sessions a user has before a conversion
t - the total duration of time from the user's first session to conversion
tbc - the individual time before conversion for each session

So for three different sample users, data may look like this:

UserID  n   t   Session#    Timestamp   TBC Conversion in session?
User1   4   12  1           2015-10-01  12  No
User1   4   12  2           2015-10-06  7   No
User1   4   12  3           2015-10-11  2   No
User1   4   12  4           2015-10-13  0   YES
User2   3   4   1           2015-10-09  4   No
User2   3   4   2           2015-10-09  4   No
User2   3   4   3           2015-10-13  0   YES
User3   1   0   1           2015-10-14  0   YES


texts: ARI (Adjusted Rand Index) result meaning

texts:
I am using K-Means for a binary classification problem in labelled data.
I think that K-Means used opposite labels to mine for the output variable.
I calculated the ARI to better understand if the model actually uses the labels in a different way from the data.
The ARI return a score of 0.13 and the K-Means has an efficiency of 40%.
What can I assume from that ? Can I reverse the model labels althought the low ARI score ?
It makes sense to use clustering for a dataset with such a low ARI score ?


texts: Why isn't local averaging (including KNN) used often for regression?

texts:
My professor said that the "holy grail of regression" is the function E(Y|X=x) i.e. the conditional expectation of Y on X. In practice, you'd take a small window of X and take the average value of Y for all observations that lie in the window.

The professor said that this is basically the best prediction you can make, but we don't usually do it because the curse of dimensionality reduces its effectiveness in when # of predictors is large. So it seems that local averaging (KNN regression is a type of this) is good with few predictors. However, in most articles and stats classes, I always see linear regression being used even in low dimensions. Why isn't local averaging used more often?


texts: I'm working on a project in which my data set is in xarray and I need to write it to IMMA format. I converted it from xarray to pandas and am totally stumped on how to go from pandas to IMMA. Any advice? Would it be easier to go straight from xarray to IMMA? I am working in python by the way. Thanks!


texts: PyMC3: how to efficiently regress on many variables?

texts:
I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to define each on their own, like alpha=pm.Normal('alpha',mu=0,st=1), and manually add them all with each other. However, what are the standard approaches when one is dealing with dozens/hundreds of variables, each needing a prior?

I see that the shape argument is helpful in defining many priors at once, but after that how do you then add them all together in the mean equation (if your model specifies a Normal distribution), or what if different groups of variables need different priors?

I would heavily appreciate any advice on this matter, as I'm switching over to PyMC3 from regular machine learning and regression.


texts: For calculating gradient penalty, why we need to consider data point that lies on the straight lines between actual and generator data pairs?

texts:
I am trying to understand the gradient penalty which was introduced in the following famous paper:
Improved Training of Wasserstein GANs
Introduced in section 4, equation 3
For calculating gradient penalty,
they first step is to calculate gradients of Critic outputs W.R.T inputs, which gives us an approximation of how large the weights of Critic. (for larger weights, larger Critic output and therefore larget gradients)
But, here instead of only actual and generator data as input, they also used random data points that lie between actual and generator data pairs. As a reason, they wrote,
"This is motivated by the fact that the optimal critic contains straight lines with gradient norm 1 connecting coupled points from Pr and Pg (see Proposition 1). "
Here,
Pr=Actual data distribution
Pg=Generator data distribution
In Proposition 1, they mathematically proved the fact, which is less intuitive and hard for me to grasp.
Can anybody explain a bit more intuitively (possible without math)?


texts: Are there any learner-specific optimizers?

texts:
In reading about machine learning (ML), and working through some basic examples, it appears to me most learning algorithms use generic optimizers. I am using the word "optimizer" to describe the technique the learner uses to minimize the loss function. Gradient decent, and it's variants, seems to be the most common. But the general idea in ML seems to be to continually iterate a learning algorithm, each time adjusting various things to try to improve the loss.

Gradient decent, and similar techniques, are generic in that they work on a wide variety of learning models. This makes intuitive sense, all you need is the predicted value and the known target value.

But, often the trade-off with generic techniques is that while they work everywhere, they are slow. In other words, in theory you could construct a learner-specific optimizer that exploits characteristics of the learning algorithm to avoid some of the work that something like SGD might have to do. So if you know you're going to use learning algorithm XYZ, then maybe you can speed up computation considerably by using a loss-minimizing technique designed explicitly for learner XYZ.

As a rough analogy, since my background is in programming, consider sorting algorithms. Quicksort is often cited as the best general sorting algorithm. But if you know in advance that your data is constrained in certain ways, other algorithms will be faster, given a specific use-case (even good old bubblesort can be fast if conditions are right!).

So, that's the question: have learner-specific optimizers been developed? (And a related question, is "optimizer" the right word to use here?)

(FWIW, I posted this same question to Kaggle Q&A, but didn't get any responses there.)


texts: Why do the arrange of clusters change when change the unit

texts:
I did k-means clustering with 2d data. The x axis represent depth (km). And result was :.
But when I converted km to metr, I got it

Could you give explanation why do this happens?


texts:
I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and filtering?

The Data:

a huge amount of human text input of variable length (searchterms and whole sentences (up to 200 characters) ) over several years. The text contains a lot of spam (like machine input from bots, single words, stupid searches, product searches ... ) and only a few % of seems to be useful. I realised that sometimes (only very rarely) people search my side by asking really cool questions. These questions are so cool, that i think it is worth to have a deeper look into them to see how people search over time and what topics people have been interested in using my website.

My problem:

is that i am really struggling with the preprocessing (i.e. dropping the spam). I already tried some stopword list from the web (NLTK etc.), but these don't really help my needs regarding this dataset.

Thanks for your ideas and discussion folks!


texts: Applying random forest model to a dataframe with multiple types of data

texts:
I am trying to solve the following classification problem: one has a large (~2-10 GB) .csv file containing a data frame of various types. In particular, there are columns containing numeric floats as well as date/times, text strings and discrete parameters from a small set. There is also a binary parameter classifying each entry which one wants to predict in the future using the machine learning approach.

Now, what would be the most appropriate tool for this job? After some research I settled on scikit-learn as the main option, but if there is other software radically better suited for the task, I'd be happy to switch.

Secondly, is the random forest approach the best way to go here? I realise there are probably many factors influencing the answer to this question, but from playing around with a few tutorials and looking at the scikit-learn flowchart I am currently leaning towards random forests.

Assuming one uses random forests, how exactly one should prepare these data in order for the model to make sense of it? From what I understand, scikit-learn takes only numerical floats as inputs for the model. While I could discretise date/time and other discrete fields with something like OneHotEncoder, it is not immediately clear what to do with text fields since the processing, I suppose, is very different. (I have looked at the bag-of-words approach, but being someone new to the field I fell it a bit hard to grasp right away.) Is there a meaningful way of transforming text fields for use in the model?

Finally, what is the best way of dealing with large datasets for this task? While for the start their size could be limited to something fitting inside runtime memory, ideally there should be a way to extend the training set to be very large.

Apologies for the vagueness of some questions. Since I am quite new to data science in general, I would appreciate if you could include actual working examples in your answers. Thanks!


texts:
In Aurelien Geron's  book I found this line

This cost function makes sense because –log(t) grows very large when t approaches
 0, so the cost will be large if the model estimates a probability close to 0 for a positive instance, and it will also be very large if the model estimates a probability close to 1
for a negative instance. On the other hand, – log(t) is close to 0 when t is close to 1, so
the cost will be close to 0 if the estimated probability is close to 0 for a negative
instance or close to 1 for a positive instance, which is precisely what we want.

What I dont get is, How will  the cost will be large if the model estimates a probability close to 0 for a positive instance, and it will also be very large if the model estimates a probability close to 1
for a negative instance?


texts:
A posterior  $p(\theta\vert x) = \frac{p(x \vert \theta)p(\theta)}{p(x)} $

Many materials say that since the $p(x)$ is a constant, the $p(x)$ can be ignored. Thus, $p(\theta\vert x) \propto p(x \vert \theta)p(\theta)$

My question is why $p(x)$ is a constant and ignored. Is this because even though we don't know the distribution x, there is a corresponding true distribution for $x$? So, $p(x) $ is a constant (we don't know but already determined) and thus, can be ignored?


texts: Is it acceptable to append information to word embeddings?

texts:
Let's say I have my 300 dimensional word embedding trained with Word2Vec and it contains 10,000 word vectors.
I have additional data on the 10,000 words in the form of a vector (10,000x1), containing values between 0 and 1. Can I simply append the vector to the word embedding so that I have a 301 dimensional embedding?
I am looking to calculate similarities between word vectors using cosine similarity.


texts: What is the Time Complexity of Linear Regression?

texts:
I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) needs to be computed over iterations of the weight combinations (as a brute force approach). This makes computation time dependent on the number of weights and obviously on the number of training data.

If $n$ is the number of training data, $W$ is the number of weights and each resolution of the weight space is set to $m$ meaning that each weight will iterate through $m$ number of possible values. Then the time complexity of this linear regression is

$O(m^Wn)$

Is this correct?


texts:
I have a DataFrame with more than 2 million rows, that looks like this:

+-------------------+--------------+--------+---------------------------+-------------+
|   PartitionKey    |    RowKey    |  Type  |           Path            |    Name     |
+-------------------+--------------+--------+---------------------------+-------------+
| /                 | /People      | Folder | /People                   | People      |
| /People           | /index1.xlsx | File   | /People/index1.xlsx       | index1.xlsx |
| /People           | /index2.xlsx | File   | /People/index2.xlsx       | index2.xlsx |
| /People           | /index3.xlsx | File   | /People/index3.xlsx       | index3.xlsx |
| /People           | /Employees   | Folder | /People/Employees         | Employees   |
| /People/Employees | /cv1.pdf     | File   | /People/Employees/cv1.pdf | cv1.pdf     |
| /People/Employees | /cv2.pdf     | File   | /People/Employees/cv2.pdf | cv2.pdf     |
| /People/Employees | /cv3.pdf     | File   | /People/Employees/cv3.pdf | cv3.pdf     |
+-------------------+--------------+--------+---------------------------+-------------+

I want to add two new columns: DirectFileCount and RecursiveFileCount.

Those should indicate the number of files within the folder itself (folder identified as Type == Folder), and the number of files within itself and all sub folders recursively, as per the Path --> PartitionKey relationship from the folders to files.

It should make the DataFrame look like this:

+-------------------+--------------+--------+---------------------------+-------------+-----------------+--------------------+
|   PartitionKey    |    RowKey    |  Type  |           Path            |    Name     | DirectFileCount | RecursiveFileCount |
+-------------------+--------------+--------+---------------------------+-------------+-----------------+--------------------+
| /                 | /People      | Folder | /People                   | People      |               3 |                  6 |
| /People           | /index1.xlsx | File   | /People/index1.xlsx       | index1.xlsx |               0 |                  0 |
| /People           | /index2.xlsx | File   | /People/index2.xlsx       | index2.xlsx |               0 |                  0 |
| /People           | /index3.xlsx | File   | /People/index3.xlsx       | index3.xlsx |               0 |                  0 |
| /People           | /Employees   | Folder | /People/Employees         | Employees   |               3 |                  3 |
| /People/Employees | /cv1.pdf     | File   | /People/Employees/cv1.pdf | cv1.pdf     |               0 |                  0 |
| /People/Employees | /cv2.pdf     | File   | /People/Employees/cv2.pdf | cv2.pdf     |               0 |                  0 |
| /People/Employees | /cv3.pdf     | File   | /People/Employees/cv3.pdf | cv3.pdf     |               0 |                  0 |
+-------------------+--------------+--------+---------------------------+-------------+-----------------+--------------------+

I have something for direct count that works:

df_count = df.groupby(['.tag', 'PartitionKey']).size().reset_index(name='counts')
df_file_count = df_count[df_count['.tag'] == 'file'].set_index('PartitionKey')

def direct_count(row):
    if row['.tag'] == 'folder':
        try:
            return df_file_count.loc[row['path_lower']].counts
        except:
            pass

    return 0

df['DirectFileCount'] = df.apply(lambda row: direct_count(row), axis=1)

The above code takes care of DirectFileCount and completes in less than 2 minutes.

RecursiveFileCount I cannot get done in a way that doesn't take ages. It performs insanely bad, and since there's two million rows, if it just takes a second for each, it practically never completes.


texts: Visualize interval type data graphically and regression

texts:
Suppose I have data like this:

x
y

5
10

10
12-30

15
45

20
50

25
55-60

I want to visualize it graphically. How can I do it? I googled how to describe intervals graphically, but didn't find anything. And I also want to do regression analysis on the data.


texts: RStudio keyboard shortcut to run child documents in a knitr doc?

texts:
I have a knitr doc.  Actually, many.  I've started factoring out common code in child documents.  For example, in a doc foo.Rnw I might have:

<<setup, child='setup.Rnw'>>=
@

When I'm debugging, I run all the chunks from foo.Rnw by opening that file and typing option-command-n (on OS X) over and over.  However, it doesn't run the chunks in the child doc, so I have to open the child doc, run option-command-n in that doc, then switch back.  With several child docs, this is a pain.

Is there a simple way to run an entire child doc in interactive mode in RStudio?

This post asks the same thing (I think), and is unresolved.


texts:
I just want to know if my ipython is properly installed. I have both Python and Anaconda installed on my Windows. I entered the "pip3 install ipython" command on the Command Prompt and it downloaded and installed the ipython package. However, whenever I try to access it, I see this:

Notice the icon on the top left hand corner:

And whenever I execute a code and will like to continue working in that file, it will take forever for iPython to take me to the next line of code (or in some cases, may not even do so). For example, in the case below, I have not been able to enter any code after the "In [18]: plt.show()" since "In [19]:" would not just come up. Please see this:

I believe this is because the package is not properly installed.

I will be grateful if you guys can just show me what I can do about this?
Thanks.


texts: when can xgboost or catboost be better then Logistic regression?

texts:
I need to improve the prediction result of an algorithm that is already programmed based on logistic regression ( for binary classification).

I tried to use XGBoost and CatBoost (with default parameters). but it takes a long time to train the model (LR takes about 1min and boost takes about 20 min).
and if I want to apply tuning parameters it could take more time for fitting parameters.

I want to ask if there are any suggestions to apply fastly boosting methods.
And if there are other ways to get better performance I hope to mention them, please.

Ps: My data is about 280 000 simples and 247 (numerical) features;


texts: Training And Testing Error Curves caret package in r

texts:
I am Running the following models

Logistic regression
Decision Trees
SVM
Naive Bayes
Random Forest
On the same data set.

I am using Caret package in r.

Its My dream to plot Training error and Testing error curves, I know those plots

are pretty beautiful as well as other performance measure curves using caret

package, any help would be really appreciated.


texts: Can PyLearn do everything that Theano can?

texts: Since PyLearn2 is build upon Theano, is it possible to do anything I can do in Theano in Pylearn2? For example, if I have some snippets of Theano code, can I run them as-is in Pylearn2, or would this not work? If it wouldn't, why not?


texts: Understanding notation of Goodfellow's GAN objective function

texts:
What is the meaning of $V(D,G)$? How do we get these expectation parts?
I was trying to understand it following this article: Understanding Generative Adversarial Networks (D.Seita), but, after many tries, I still can't understand how he got from $\sum_{n=1}^{N} \log D(x)$ to $\mathbb{E}(\log(D(x))$.


texts:
The KNN algorithm is very handy and particularly suited to some of my problems, but I can't find any resources on how to implement it in production.
As a comparative example, when I use a neural network, I already have at my disposal high level tools allowing me to apply the neural network to examples (either libraries allowing me to smartly exploit the hardware of my devices when I want to do embedded, or infrastructures allowing me to use my neural networks at a lower cost if I want to run them on the cloud), and these tools are generally black boxes and are very well optimized, especially when you pass them batches of examples.
It seems that there are no such tools to use a KNN on a large scale for example on a cloud; and I guess one has to implement it oneself.
A naive implementation of KNN consisting in calculating the distances from the inferred example to all known and recorded examples makes the computational cost and thus the financial cost of using it explode (which strangely enough becomes much higher than the one I pay when I use a neural network, even though the algorithm seems simpler).
Do you know people who have shared efficient and cheap implementations of KNN? Or do you have any ideas on the subject ?
I started to imagine heuristics to quickly disqualify examples that are too far away without having to go through them one by one.
Concretely, this would consist in a set of n-dimensional nested grids such that searching in each sequence of nested grids is like searching in a tree or in a hash table and allows to locate the example in as many bins as sequences of nested grids. One then expects that the closest examples are either in the same bin or in a contiguous bin (because the example may be close to the bin boundary).
I suppose one could also imagine dividing the space into decision zones (so that a 1-KNN would generate a Voronoi diagram for example).
However, I am not a researcher and my ideas will certainly not be able to challenge the state of the art or already existing and explored ideas.


texts: Problem in constructing co-occurence matrix

texts:
I have a basic doubt in construction of co-occurence matrix from text. Coocurence matrix X is defined as :

Xij : number of times word i occurs in context of word j in a window of size w.

Now is this context limited to sentence i.e do we context word of a main word in a sentence only or across the whole document?

For ex:

Consider the corpus :

human interface computer
survey user computer system response time
eps user interface system
system human system eps
user response time
trees
graph trees
graph minors trees
graph minors survey
I like graph and stuff
I like trees and stuff
Sometimes I build a graph
Sometimes I build trees

Now for word graph and context window 10 do we look for only 4th last line or do we search for words across line also considering whole doc as one line?


texts: Statistical learning for data-limited systems

texts:
I'm currently conducting a review for quantitative methods being used for tropical inland fisheries. One of the major problems for modeling methods in tropical inland fisheries is the lack of data available. Fisheries assessments are difficult with widely distributed, small-scale fisheries. As many of the people living in tropical regions are subsistence fishers, they directly consume fish without any recordings of the catch.

I'm trying to find statistical/mathematical modeling methods that are able to deal with data-limited systems. I do not have a significant background in statistics, but I am curious about the concept of "statistical learning" and data science methods in general. My understanding of current "hot" topics in data science (machine learning, AI) is that they are useful for big data. I have not come across their usefulness in data-limited methods. Although it seems as though they could potentially deal with data limitation by using a range of other data available to make hypotheses about the state of the data that is limited. Do you know of any usefulness of statistical learning or data science methods in general for data-limited systems?
Thanks!


texts: How does class_weight work in Decision Tree

texts:
The scikit-learn implementation of DecisionTreeClassifier has a parameter as class_weight.
As per documentation:

  Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one.

and

  The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

My understanding is that it should be used in case of imbalanced classes.

Question: How does the DT (classification) algorithm use this parameter while determining the ideal split for a given node? Does it consider some kind of "weighted" majority class instead of simple majority class in a region in the prediction space?


texts: How to do give input to CNN when doing a text processing?

texts:
As a signal processing engineering and being new to NLP, I am confused with giving input to CNN network.

With my knowledge of CNN, I am trying to build a classifier for ethnicity with inputs as text (last name(LN), middle name(MN), first name(FN)).
I have a list of 8,000,000 samples with last, middle, first names and class information

array = [['person1_LN','person1_MN','person1_FN','Person1_class'],
         ['person1_LN','person1_MN','person1_FN','Person2_class'],
         ....]

I want to apply conv layer (CL) followed by Pooling Layer(PL) on LN,MN,FN respectively.

Text processing example demonstrates with the sentences to convert to word embedding. I am trying to understand this code snippet

W = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0), name='W')
self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

This tutorial says training to be done using word2vec.
Reading these two blogs I could understand nothing. In Either of the cases, my data is not a sentence, moreover in the second case on what things I have to train?

Will the CL operate if I directly give the input of words without word embeddings?
If not any example on how to embed the words in my case to give the input to CNN?


texts: Should we apply normalization to test data as well?

texts:
I am doing a project on author identification problem. I had applied the tf-idf normalization to train data and then trained a svm on that data.

Now when using the classifier should I normalize test data as well. I feel that the basic aim of normalization is to make the learning algo give more weight to more important features while learning. So once it has trained it already knows which features are important, which are not. So is there any need of applying normalization to test data as well?

I am new to this field. So please ignore if the question appears silly?


texts: Machine learning for predicting HTML Elements on a web page?

texts:
My goal is to implement an assistant for crawling web data for users that don't understand anything about HTML or DOM. I will show a web page to the user and the user has to select, what data he is interested on the page (or what data he is not interested in).

Example: If the user clicks on the cell inside a table, it is very likely he wants to extract all elements inside that column. He might only be interested in the table row, or he might be interested in only the one cell. So the algorithm proposes him three selectors for each possibility. The user might choose one of the proposed selector or he might click on another element to get a new proposition.

So far the use case. The component I want to create:

has as input data a list of elements inside a DOM (hierarchy of HTML elements), that the user wants to crawl
shall output a (probability) ranked list of selectors that will fit the users selection to 100% and possibly include other elements that the user is probably interested in.

The problem here is the high variability of websites, and the possibility to create similar looking results with very differently structured elements. Manually creating rules for prediction can therefore probably cover the most basic uses (like getting data from a table column), but will fail when it comes to finding article parts on a news page.

So finally my questions. I am thinking about the following processing:

first generate selectors that will fit the users selection
sort out the selectors that do not fit the users choice of elements by 100%
use a (machine learned?) model to predict which selector the user might be interested in

Do you think the above processing will be productive for this problem?

What algorithm could be most suitable for ranking the selectors in 3.?

Is there an idea on how to auto-magically come up with a list of selectors for the selected elements?

I hope I am making sense - I do not have a very strong background in data analysis and machine learning so I am hoping for some directions where to look into.


texts: What is the order of elements in an image in python?

texts:
I have a set of images which are loaded from an h5 file. I checked their dimensions and I get (209, 64, 64, 3).

read = h5py.File('datasets/train_catvnoncat.h5', 'r')
read['train_set_x'].shape

  (209, 64, 64, 3)

it means that there are 209 images but the point that I cannot understand is that, what is (64, 64, 3)?
I have used the following code for plotting:

import matplotlib.pyplot as plt

plt.imshow(read['train_set_x'][1])
plt.show()

and I get a colored image which is 64 by 64. before this, I thought for (., ., .) shapes, the second number specifies the number of lines and the third one specifies the number of rows. also the first one specifies the number of the mentioned (row and column) arrays.
My question is that in numpy if you have a three dimensional array, for accessing rows and columns you have to change the second and third entries in the indexing operator; Why this is different in images and rows and columns are arranged differently in images. Shouldn't it be (3, 64, 64)?


texts: a neural network can be used as an optimizer?

texts:
I was wondering if a neural network can be used as an optimizer.
In other words, a network2 used as an optimizer takes the loss value of network1 and based on that it predicts the best weights for network 1.
as a discriminator in architecture, gan imitates a function of loss.


texts: Undersampling for credit card fraud detection before or after Train/Test Split

texts:
I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and precision on the test set.
When I do the undersampling only on training set and test on the independent set I get a very poor precision but the same recall!
My question is :

Should I undersample before splitting into train and test , will this mess with the distribution of the dataset and not be representative of the real world?

Or does the above logic only apply when oversampling?


texts:
Considering that individual trees in random forests use the cart algorithm(or can be configured to) , if cart fails to predict anything(empty tree) ,why should random forest perform better on the same dataset.
Edit:
I am using R (rpart vs randomForest) ; the data is daily financial data of a stock. CART gave empty trees but random forest achieved decent accuracy even on unseen data.


texts: How can I reduce the number of dimensions using a Clustering algorithm in a mixed dataset?

texts:
I am working with a mixed data set, corresponding to TV consumption data, with the aim of reducing the number of features to only those relevant to detect TV consumption patterns (or consumption groups) using clustering.
The dataset is composed of about 20 dimensions and 2.000.000 samples for 1 day of consumption. I have access to the data of up to 3 years of consumption, so I can exploit up to ~1 billion data. My idea is to start working with only a few days of consumption (and therefore a few million data).
3 dimensions are of continuous/numerical type (the date-time of consumption, the duration...) and the remaining dimensions are of discrete/categorical type, with features with binary options (e.g. whether the programme is live or not) or with multiple and even hundreds of options (e.g. name of the programme, theme, type of device, etc). For this reason, I am trying to implement a clustering algorithm with Python that can deal with mixed data.
Since I suspect that there are many dimensions that might be irrelevant to my study, I would like to reduce the number of features. To do so, I have thought of applying some clustering algorithm and check that, by removing a given feature, the clustering results are not affected.
However, I don't know what metric I should calculate or use to evaluate the clustering results and the effect of removing one of the attributes.
At the moment I have applied the K-prototypes algorithm which is based on K-means but for mixed data. It is easy to implement in Python (https://antonsruberts.github.io/kproto-audience/). However, it only returns the cluster labels, the centroid coordinates and the cost (defined as the sum of the distance of all points to their respective centroids). Therefore, I do not know how to interpret the results or how to study the effect of the features.
I would like to know if my approach to the problem is correct and what metric I should use to evaluate the clustering results and reduce the number of dimensions, as well as if there are other easily implemented algorithms in Python (for clustering or unsupervised feature selection) capable of dealing with the type of data I have.


texts: How do deep learning model store the mean and standard deviation and how to run sampling?

texts:
GPT-3 can generate code
But distribution learning is just learnt the repeated things and no ordering of code if sampling
Is deep learning model a storage of mean and standard deviation?
How can mean and standard deviation stored as the format of deep learning model?
How can python model.predict do like sampling from distribution?
Update
How GPT-3 code generation make learning has ordering of code in reasonable way when using model.predict to output code?
Is deep learning of GPT-3 no statistics meaning?


texts:
I am working with an algorithm studying the response of a system to an external factor (Phenomenon A). The system can either respond perfectly, or can be completely disrupted.

I am plotting the system's response in the following graph, which shows kind of an oscillating trend. High y values imply poor responses (upper limit: 100, total collapse), whereas low y values mean very good responses (lower limit: 0, no influence whatsoever).

I am running this experiment based on an arbitrary number of simulations - in this case, 1000.

My question: is there a metric which can help me identify the most appropriate number of simulations? I was thinking of something which could associate an optimal number of runs with a reduction in oscillation (convergence). What metric/formula could be useful here?


texts: Word based perplexity from char-rnn model

texts: I'm training a character based RNN model for text prediction and want to compare it to similar models. Since most literature uses word based perplexity as a performance metric, what would be the "proper" way to calculate word based perplexity from a character based model?


texts: CNN model contains several images that are null

texts:
I'm using a deep CNN with the ReLU activation function. When visualizing the layers (each with 32 filters), several of the filtered images are zeros.
I am trying to reason why this may be happening? What would cause the entire image to be zero? I can understand having a sparse image (given the ReLU), but what would cause all zeros?
I hope this is clear and has sufficient detail to answer appropriately.


texts: Imputing missing value based on filtering result of another column

texts:
C1
C2

A
x

A
y

A
z

A
x

A
NaN

A
x

A
x

A
x

B
y

B
y

B
z

B
y

B
NaN

B
y

B
x

B
x

I have to impute missing values in C2 , the imputation should be such that if the missing values corresponding C1 column is A, then filter the datset by A and find the mode of C2 and replace the missing value, similarly for B.
In the above example it should be x for first missing value and y for second missing value

C1
C2

A
x

A
y

A
z

A
x

A
x

A
x

A
x

A
x

B
y

B
y

B
z

B
y

B
y

B
y

B
x

B
x

Tried this, not sure if this is the best way
for i in df['c1'].unique():
  df[(df['c1']==i) & (df['c2'].isnull())]['c2'] = df[df['c1']==i]['c2'].mode()[0]


texts: Why is the kernel of a Convolutional layer a 4D-tensor and not a 3D one?

texts:
I am doing my final degree project on Convolutional Networks and trying to understand the explanation shown in Deep Learning book by Ian Goodfellow et al.
When defining convolution for 2D images, the expression is:
$$S(i,j) = (K*I)(i,j) = \sum_{m}\sum_{n}I(i+m,j+n)K(m,n)$$
where $I$ is the image input (two dimensions for widht and height) and $K$ is the kernel.
Later it states that the input is usually a 3D tensor. This is because usually the input image has multiple channels (say, red, green and blue channels). Furthermore, it says that it can also be a 4D-tensor when the input is seen as a batch of images, where the last dimension represents a different example, but that they will omit this last dimension for the sake of simplicity.  I understand this.
Let now $I_{i,j,k}$ be the input element with row $j$, column $k$ (height and width) and channel $i$. The output of the convolution is a similarly-structured 3D-tensor $S_{i,j,k}$.
Then, the generalized convolution expression for this 3D-tensor input is
$$S_{i,j,k} = \sum_{m,n,l} I_{l,j+m-1,k+n-1}K_{i,l,m,n}$$
where "the 4-D kernel tensor $K$ with element $K_{i,j,k,l}$ giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input, with an oﬀset of $k$ rows and $l$ columns between the output unit and the input unit".
I am completely lost in this definition. Why is the Kernel 4-D and not 3-D (which is a logical generalization of the first formula)? What is the analogous to the sentence "giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input" in the initial 2D Kernel tensor? I think that is the main things that has to be understood to understand the 4D-kernel.