texts: Why does not log transformation make the data normalized?

texts:
Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following codes

  import numpy as np

  import pandas as pd

  from sklearn.preprocessing import MinMaxScaler

  df=pd.read_csv("Data.csv")

  scaler=MinMaxScaler(feature_range=(0,1))

  df['vBMD (mgHA/cm3)']=scaler.fit_transform(np.array(df['vBMD
  (mgHA/cm3)']).reshape(-1,1))

  df['vBMD (mgHA/cm3)']=np.log(np.array(df['vBMD (mgHA/cm3)']))

After the transfromation, I have got the following result.

While I am waiting that the feature will be normalized, its skewness increased. Thus, what am I doing wrong?


texts: What is the best way of combining audio and visual data to make predictions?

texts: I am trying to predict the probability of a disease by using audio and images, the audio and the images do not come from the same source. I am thinking of combining the outputs (maybe average them) of two models one utilizing image data to calculate the probability the other using audio data. I am feeling unsure of this approach as I will not be able to verify the final combined output. Will the outcomes be reliable, or should I use a different approach?


texts: Why does Jim Gray call "data-driven science" a new paradigm?

texts:
Wikipedia it says about data science:

Turing award winner Jim Gray imagined data science as a "fourth paradigm" of science (empirical, theoretical, computational and now data-driven) and asserted that "everything about science is changing because of the impact of information technology" and the data deluge.[4][5]

The term "theoretical" links to "basic research", which I believe to be false, since basic research can be also empirical. However, "theoretical" makes sense in contrast to empirical: In theoretical research you try to find solution "simply" by thinking about the problem, in empirical research you gather data on the problem, and use it to either build models/theories regarding reality or test these models/theories.
I do not see what computational science is supposed to be in in contrast, because I believe that I can use computer for both empirical science (e.g. using statistics and machine learning models) or in theoretical science (e.g. building simulations that are based on my assumptions and see if they result in the expected outcome - no data involved). So I think computational science would be a complement to empirical science and/or theoretical science, not a contrasting paradigm.
Now, comes in "data-driven science". How is this supposed to be different? Either you could say data-driven is just empirical (as I would say), so it is not a new paradigm. Or you might interpret it as "computational", then it is not even a paradigm.
Two lines of argumentation I came up with are invalid after all:

"While empirical science uses data, data-driven uses more data." But having more apples doesn't mean you didn't have apples before. This is quantitative, not qualitative. Paradigms are by definition qualitative different.
"We have different methods." We do have new machine learning methods, we did not have in statistics, but that does not change the general paradigm to collect data and use it to build models.

Questions:

What does Jim Gray (allegedly) mean with "fourth paradigm"?
Is he justified to call it a new paradigm akin to "empirical science" and "theoretical science"? (Please make an argument why!)


texts: How to make Training Data for Meta-Learning

texts:
If I am doing 5-way-1-shot meta learning on dataset with 200 classes, what would be the sample space for my labels?
I know there would be 5 labels in each meta-set, but will I be chosing labels from digits 1-5 or from 1-200?


texts: What are named entity advantages over strings?

texts:
I am unclear about the practical benefit of named entity recognition. Specifically, I don't get how this entity is any better than a string representation of the same object. Let's take the popular example of movies. I can create a model where each row is a movie. Any attribute you can have in a recommendation system can just as easily be a column in my movie table: release date, actors, genre, etc. If a user does a search for 'animation' - or even if they don't, but you know from that user's history that they saw The Lion King, any decent search engine could pull up Moana as a recommendation, so how is a named entity any better?

Or you pull a named entity out of a text document. Great, but now what? Isn't it still just a string, which can be assigned as an object with it's own unique dict of values, like genre and release date? I assume there are benefits over strings, or it wouldn't be such a big deal to so many people. I just don't know what those benefits are.


texts: Comparing two sets of exponential data T(t)=−e^(−kt)

texts:
I have two sets of exponential data (temperature measurements) of the form:

$T(t)=−e^{(−kt)}$.

$k$ is a constant that determines the rate of temperature change. 1 temperature measurement was taken every 10 seconds to obtain 200 temperature readings for each data set. Under normal circumstances, I could use ANOVA to compare the data and determine its variability and error. This assumes a normal distribution, however, which my exponential data is not. I believe it is of Poisson distribution. What is the best way to compare these two data sets to determine variability? I cannot compare datapoint to datapoint since I can't guarantee that I measured the temperatures starting at the exact same initial point (say set 1 started at $t=0$, set 2 could have started at $t=.01$)


texts: Analysis of Alternating Decision Tree on Weka

texts:
I am applying the AD Tree algorithm & this is the tree visualization of the output:

I can't understand the values in the decision nodes (-0.4,0.541,-0.882...), How are these calculated? & how did we calculate the root node's score?

Are predicate conditions (<127.5..) formed by entropy splitting mechanism?

This is an image of the output:

Any help is appreciated, cannot find any AD Tree output analysis document!!


texts: I don't see a package for doing Convolutional Neural Networks in R. Has anyone implemented this kind of algorithm in R?


texts:
How would you see the relation of a monthly variable categorical with a yearly target value float for differents years in order to predict.

In the farming industry the relation is not so simple for categorical values as the drought level , like pre-harvest period has more weight for some months on the yearly result because some months are more relevant.

What kind of null hypothesis test would you do for with both variables to conclude
I was thinking of anova
Any other suggestions.


texts: Gradient derivation reference for Phased LSTM

texts:
I've worked-through the back-propagation for the Phased LSTM (Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu 2016) and would like to show the notes. It was a relatively difficult task, so I post it here, to help if anyone struggles with derivation.

I assume you know how to do the back-prop for the usual LSTM, and so will show only the most critical gradient-pieces.

Keep in mind the gradient flows in to opposite direction of the arrow.
The Phased LSTM uses Peepholes.

Also, notice, the authors skip some timesteps by referring to the "most recent update time" as $t_j$  but I am using just $t$ for everything. That's because during training we need to consider every timestep - there is a "leak" in the time gate. However, as the paper advises, the leak is turned off at runtime, and you can then use $t_j$ and skip any 'non-j' timesteps to save performance.

Note - there still might be an error in the notes, although I've checked it thoroughly, and also via Gradient Check procedure. Still, please let me know if you spot one.

Here is a link regarding partial derivatives through the modulo operator

Tools to be used:

The quotient rule says:

$$\frac{\partial}{\partial{a_i}} \left( \frac{g(a_i)}{h(a_i)} \right) = \frac{\frac{\partial{g(a_i)}}{\partial{a_i}}h(a_i) - g(a_i)\frac{\partial{h(a_i)}}{\partial{a_i}}}{h(a_i)^2}$$

and the the Chain rule says:

$$\frac{\partial}{\partial{a_i}} [ g(h(a_i))] = \frac{\partial{g(h(a_i))}}{\partial{h(a_i)}}\frac{\partial{h(a_i)}}{\partial{a_i}}$$

as a bonus,product rule says:

$$\frac{\partial}{\partial{a_i}}g(a_i)h(a_i) = \frac{\partial{g(a_i)}}{\partial{a_i}} + \frac{\partial{h(a_i)}}{\partial{a_i}}$$

Part #2:


texts: How to know when to treat a problem as a classification task or a regression task?

texts:
Facing this dataset: https://archive.ics.uci.edu/ml/datasets/wine+quality I treated it both as a classification task and as a regression task.

Due to the fact that classifying the quality of a wine as 8 while it is a 3 is a huge difference while classifying a 5 as 6 is not that bad

This has two subquestions:

Is there an easy way to change the weight of the classification in a (scikit-learn) classifier?
Following the regression approach and having the output being rounded to the nearest valid wine quality value (converting the outputs to a classifier-like outputs) hides any pitfalls?

Many thanks


texts: Strangeness in validation loss between CPU vs GPU when training CNN

texts:
I've been training an implementation of Mask R-CNN and it was training very successfully on my CPU but I've just set up my GPU and it is giving some strange results when looking at my validation loss. It is as if the learning rate got turned up when I moved to using my GPU. Has anyone ever experienced this or know what could cause this difference? I didn't think there was any difference in the actual computation on CPU vs GPU, is there?

Training on CPU:

Training on GPU:


texts: Order of normalization / augmentation for image classification

texts:
I'm currently working on a common image classification with CNN.

I would like to use both normalization (substract mean / divide by std per channel) and data augmentation (rotation, color, blur, ...) but I don't know how to use them together.

Which order should I use ?

First normalize with parameters based only on original images and then augment it (augment a normalized image is relevant ? should I ban some type of augmentation like color ?)
Augment data and apply normalization based on all image (compute mean/ std with augmented images) which seems to be counterintuitive.
Augment data and apply normalization based on only original image which means that data are not really normalized
Or don't use both methods


texts: What's the difference between fit and fit_transform in scikit-learn models?

texts:
I'm a newbie to data science, and I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we might need to transform data?
What does it mean, fitting a model on training data and transforming to test data? Does it mean, for example, converting categorical variables into numbers in training and transforming the new feature set onto test data?


texts: replacing last duplicate values in pandas

texts:
I have a dataframe ,that looks like this
  | **site**  |  **Active**|

0     |: deals|  Active:|
1     |: deals | InActive:|
2     | : deals | Active:|
3     |:discount | InActive:|
4     |:discount|  Active:|
i don't want to drop the duplicate items, but i want to change the Active columns value based on Site column,for example Active has to change inactive based on duplicate item in site column,Inactive also have to change based on number of duplicate items present,last duplicate item has to Active, other than that Inactive, If it is already inactive it also have to change
Expected
   site    Active

0     deals  InActive
1     deals  InActive
2     deals    Active
3  discount  InActive
4  discount    Active


texts: Basic doubt regarding "training" of a YOLO model

texts:
So I have just recently started exploring machine learning, and for a project I was required to train the YOLO v5 model. I first tried it on the coco128 dataset:https://www.kaggle.com/ultralytics/coco128..
repository of the yolo v5:
https://github.com/ultralytics/yolov5
I followed this tutorial: https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data step by step and managed to train the model succesfully, it is detecting objects as I intended it  to do.
I wanted to have a very basic overview an what is basically happening when we are "training" the model.
The command used for training was:
python train.py --img 640 --batch 16 --epochs 5 --data coco128.yaml --weights yolov5s.pt

From what I understand, this tells the command line to run the file "train.py" (which uses argeparse library) with the following arguments: 640 (image size), 16 (batch size for batch gradient descent), 5 (no. of epochs), coco128.yaml (the config file), and yolov5s.pt (the weights file).
Doubts

The tutorial mentions that the weights are pretrained. This is confusing me because I thought the entire objective of "training" a model is to find the correct weights for the parameters. If the weights are already supplied, what exactly is the training achieving? It has been fed a bunch of images along with the labels for the bounding boxes for that images. What does it do with these images and labels to become capable of predicting objects?
The coco128.yaml mentions a path for the "validation" folder. What exactly is a "validation" folder used for?


texts: Should I only use scalar labels with Keras LSTM?

texts:
I have an array X_train = (1110,25,2) and a y_train = (1110,5,2). It means I use arrays with length of 25 for inputs and length of 5 for labels. But when I use:

model = Sequential()
model.add(LSTM(units = 25, return_sequences = True, input_shape = (25, 2)))
model.add(Dropout(0.2))
model.add(Dense(units = 2))
model.compile(optimizer = 'adam', loss = 'mean_squared_error')
model.fit(X_train, y_train, epochs = 100 , batch_size = 25)

It gives me this error in the last line of the code:

  ValueError: Error when checking target: expected dense_1 to have 2
  dimensions, but got array with shape (1110, 5, 2) [Finished in 5.1s
  with exit code 1]

The code works if I change the length of y_train to the 1, but I like to test longer y labels to train. What is the problem and how can I fix it?

EDIT:
I create X_train and y_train arrays with this code:

for i in range((len(training_set)%30) + 30 , len(training_set) - days ):
    X_train.append(training_set_scaled[i-30:i-5])
    y_train.append(training_set_scaled[i-5:i])
X_train, y_train = np.array(X_train), np.array(y_train)

This is the result of model.summary()

model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_45 (LSTM)               (None, 25, 25)            2800
_________________________________________________________________
dropout_45 (Dropout)         (None, 25, 25)            0
_________________________________________________________________
lstm_46 (LSTM)               (None, 25, 25)            5100
_________________________________________________________________
dropout_46 (Dropout)         (None, 25, 25)            0
_________________________________________________________________
lstm_47 (LSTM)               (None, 25, 25)            5100
_________________________________________________________________
dropout_47 (Dropout)         (None, 25, 25)            0
_________________________________________________________________
lstm_48 (LSTM)               (None, 25)                5100
_________________________________________________________________
dropout_48 (Dropout)         (None, 25)                0
_________________________________________________________________
dense_12 (Dense)             (None, 2)                 52
=================================================================
Total params: 18,152
Trainable params: 18,152
Non-trainable params: 0
_________________________________________________________________

EDIT2:
I have tried to solve my problem with RepeatVector() function in encoder-decoder approach with the following code:

model = Sequential()
model.add(LSTM(units = 25, return_sequences = True, input_shape = (25, 2)))
model.add(Dropout(0.2))
model.add(LSTM(units = 25, return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(units = 25)) #, return_sequences = True))
model.add(Dropout(0.2))
model.add(RepeatVector(5))
model.add(LSTM(units = 5 ,return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(units = 5 ,return_sequences = True ))
model.add(Dropout(0.2))
model.add(Dense(units = 2))

But I get this stupid result:


texts: Is the binary matrix of transaction database low-rank in the problem of association rules mining?

texts:
In the problem of association rules mining, the transaction database with $N$ transactions and $M$ items can be represented by a binary matrix $D$ with dimension $N \times M$. My questions are:

Is $D$ low-rank or approximately low-rank? Equivalently, are there only few large singular values dominate the overall singular values of $D$?
Intuitively, only few items on average are contained in each transaction because a customer generally buy only few items. However, is that true?

It will be much appreciated if someone familiar this field could answer my questions or even only provide any clues.


texts: How to predict unknown(hidden) value by incomplete value or partly recorded value

texts:
Let me make it clear by make an example:

Suppose I knew a person's cost each month for 3 years like:

2016Jan : $2500

2016Feb : $4000

2016Mar : $3500
...

Just according to this, can I predict how much he earned each month in these years? If I could, what model should use? If I couldn't, what info I need to make this possible. If I couldn't guess in 100 percent but I could guess by some probability, what model should I use?

Another example:
Need to estimate how much each store sells in each city, however I could only got the sales when they record it in electric system, if they didn't record it in electric system(may use paper or something else); then I don't know, how can I get all sales, not just the sales record in electric system?

Any suggestion is welcomed


texts:
I'm building a decision tree in R using the rpart function, available in the library of the same name, but am experiencing some serious performance issues when partitioning data using more than one feature (there is roughly 5000 rows in my data set, and the features in question have around 30 unique values each). Running rpart with the one feature completes in a couple hours (running single thread on a CentOS server with about 250GB ram), but, after 6 days, a model with a second feature added has yet to complete!

I've created an approach to extracting the path through the tree new data takes (see my question/solution here), which is my main motivation here, but the method requires an object that can be type-cast as a party object. A colleague of mine told me that he's seen performance boosts using the ctree function, but the output of this is an inscrutable S4 object that I can't figure out how to convert. My questions are two:

Does anyone know a way to speed up rpart beyond what I've already done here:clf <- rpart(Y ~ X$var1 + X$var2, rpart.control(maxcompete=1, maxsurrogate=0))?
Does anyone know how to convert a ctree object to a party object? Alternatively, does anyone have concurring or contradicting observations on ctree performance?


texts: Dense layer: how to transform hgih dim vector to one dim? (sigmoid function)

texts:
I have a dense layer who transforms a 100 dimensional vector to a 1 dimensional object using a sigmoid function. I understand how a sigmoid function transforms a number to a number between 0 and 1.

But I don't understand how it transforms the dimensionality of the vectors?


texts:
I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the user input)
n-result are retrieved but the closest expressions are not necessarily the most relevant one.

For example, by typing : hospital machine
The top results will be "dialysis machine", "medical machine", ... but I'll also find expressions like "building machine", "office machine"
A user will most likely choose medicine related machine.

Is there a way to optimize my ranking system based on the user input while keeping this similarity between vectors of the expression ?


texts: Is splitting the data set into train and validation applicable in unsupervised learning?

texts:
I am having a tough time implementing all the steps of setting up support vector machine (SVM) for unsupervised learning. My data set is labelled but for educational purposes I am learning unsupervised methods especially how to use SVM in an unsupervised way.

Basically, in supervised learning we have a data set X that is split into Xtrain Xval and an independent Xtest which is never seen by the SVM during training. During training, the hyperparameters are learned using cross-validation (CV approach). My question is that for unsupervised learning should I need to have Xtrain, Xval and CV for learning the hyperparameters? The reason I ask is that theoretically in unsupervised learning we don't have any labelled dataset. Does this mean that we don't have Xtrain and Xval?

In the Matlab link, cross validation is done but it is unclear to me if that is done on part of the training data Xval

CVSVMModel = crossval(SVMModel);
[~,scorePred] = kfoldPredict(CVSVMModel);

QUESTIONS:

1) Do we estimate the hyperparameters in SVM for unsupervised learning?

2) If yes, then this implies that we have access to labels when theoretically unsupervised learning is without knowing the labelled data set. This is contradictory to me.

How do I estimate the hyperparameters in unsupervised SVM without Xval and splitting the dataset using c = cvpartition(n,'KFold',k)

Can sombebody please suggest how to go about unsupervised learning. Thank you.


texts: Reasons why training error can go up after more training?

texts:
In the context of deep-learning, I understand why the test or validation error can go up with more training: this is the result of overfitting.

I also can think of one reason why even the training error would go up after more training: if the learning rate is too big, then the parameter will "jump" too far on the loss function: Even though the gradient is in direction $x$, the parameter update will be too big so that it "goes over" the minimum.

But is this the only reason that training error can go down after more training?

What are all the reasons that training error can go down after more training?


texts: What technique to use in order to identify what position an audio sample is at in a longer audio sample?

texts:
I am interested in what techniques and algorithms could be used in order to tackle the following problem:

I have a database of audio samples, specifically live performances of various songs. I have about a dozen songs and for each of them I have again about a dozen samples of that song's performance.

I am hoping that by having more than one sample for each song, it will be possible to better "lock down" the song's general features and filter out noise and differences between performances. These being live performances, each sample is a bit different, some are captured at better quality than others (directly from the sound guy versus a phone recording in the crowd), some songs have interludes, false starts (guitarist forgot to turn on the amp), start too late, end too early…

Now the next thing that I aside from this database is a live feed of a currently playing song and I am interested in using ML to find out which song is the live feed most likely to be. The way I see this, it could either be that the live feed continues to be captured so the changes of matching it up with features of the existing sample database grow or it might be more practical to periodically chop of fixed sized chunks if it is not possible to use a live feed like this.

I am interested in finding out what is the most common / most reliable approach to find what song the live audio most likely is and on top of that what position in the song is the live feed currently at.


texts: Algorithm or function or model that encourages clustered classfication?

texts:
I have a soft classfication problem, i.e., the correct label for a certain instance is not just one class with 100% probability, but rather bunch of classes with probabilities that sum up to one.

What I know as apriori information (I know it because I understand the underlying physical phenomenon) that the classes are close together, like a cluster. E.g, class one and five and eight always come together cause they're adjacent (PS: I can create adjacency matrix if required).

What do I want?

I want some way to tell the NN about this fact. Currently it is vanilla classfication neural net. Any suggestions or readings or guidannce are appreciated.


texts: How pre-trained BERT model generates word embeddings for out of vocabulary words?

texts: Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary words? Models like ELMo process inputs at character-level and can generate word embeddings for out of vocabulary words. Can BERT do something similar?


texts: Remedies to CNN-LSTM overfitting on relatively small image dataset

texts:
Notes

Using a pretrained model, trying data augmentation (not possible knowing nature of images, lowering number of parameters in the network, all didn't help)

Context

I have a sequence of images. Target is a multivariate continuous time series. I am trying LSTM on CNN without using a pretrained model. Training a CNN model didn't got me a satisfying results. A very sure reason is that train is only on one year. While predicting with test images present is on several months.

Any image augmentation is nearly impossible from nature of images, satellite images on a fixed geo-location, tracking passing clouds.

Along with images, I have time features, and trend, seasonality of target which is known for test set, as it can be scientifically calculated (it's about GHI, estimated by the Ineichen and Perez model).

Problem

The problem is with over-fitting, tracking best model on validation set is done by early stopping.

Validation set is a small fraction from train, .9 for train, so train set is furthermore made fewer.

Train is a set of 8804 images, and target variable. Timed Model layers take series of 31 sequences. For example train takes (255,31) and validation takes (29,31).

The model, I came up with is the following:

losses_weights=[[1, .4]];
main_input__ = Input(shape=(31, 120, 120, 1), name='main_input__')extraction

x__ = TimeDistributed(
    Conv2D(8, kernel_size=(3, 3), strides=(1, 1)  , activation='relu')
)(main_input__)
x__ = TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2)))(x__)
x__ = (TimeDistributed(BatchNormalization()))(x__)
x__ = TimeDistributed(Conv2D(8, (2,2), strides=(1, 1), activation='relu'))(x__)
x__ = TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2)))(x__)

# extract features and dropout
x__ = TimeDistributed(Flatten())(x__)
x__ = (TimeDistributed(Dense(8, activation='relu')))(x__)
x__ = Bidirectional(LSTM(lstm, return_sequences=True, dropout=0.3))(x__)
lstm_out__ = Bidirectional(LSTM(lstm, return_sequences=True, dropout=0.3))(x__)
auxiliary_output__ = Dense(8, name='aux_output')(lstm_out__)

auxiliary_input__ = Input(shape=(31, 10), name='aux_input')
z__ = keras.layers.concatenate([lstm_out__, auxiliary_input__])

# We stack a deep densely-connected network on top
# z__ = (LSTM(lstm, return_sequences=True, dropout=0.4))(z__)
z__ = Bidirectional(LSTM(lstm, return_sequences=True, dropout=0.3))(z__)
main_output__ = Dense(8, name='main_output')(z__)
################################################################################

loss=[loss_mse_warmup, loss_mse_warmup];
 #;
model__ = Model(inputs=[main_input__, auxiliary_input__], outputs=[main_output__, auxiliary_output__])
model__.compile(loss=loss_mse_warmup, optimizer='adam', loss_weights=loss_weights)

history__ = model__.fit(x=[x_train, aux_train],y=[y_train, y_train], epochs=100, batch_size=2, validation_split=.9, callbacks=callbacks)

loss_mse_warmup is just a mean_squared_error that ignores 5 first training input signals.

Tries

Several Batch-size lengths: [32, 16, 8, 2].
loss weights ranging in [[1, .4], [1, .3], [1, .2]].
Different variants of number of nodes in CNN: [8,16,32].
variants of strides: (2, 2), (1, 1), (3, 3).
LSTM layer number of nodes: 20 seems to be far better from other
tries.
Stacking two layers of LSTM gives nearly same result as one layer for main input and auxiliary input.
Validation and train loss of auxiliary output is less than main output, so auxiliary data is useful.
Time sequences tried: 62, 31 and 1. 31 Is slightly better. it represents half a day.
Tested pretrained mobilenet model wrapped in TimeDistributed layer. But it didn't show better results.

All tries didn't achieve validation loss better than .2 knowing that learning can be improved, knowing the challenge platform.

This is a visualization of the model:


texts: What do the parameters used in crop mean?

texts:
When we have an image to be used as an input to a CNN and we want to classify only part of the image, we usually feed the classifier with a crop of the image.
Lets say my image is called frame and x, y, w and h are xmin, ymin, xmax and ymax, respectively:
frame = frame[y:y + h, x:x + w] #Crop a part of the image

What does y:y or x:x mean and why do we sum them to h and w, respectively?
I've been seeing some people performing the crop in the following way:
frame = frame[y:h, x:w] #Crop a part of the image without adding to `w` and `h`

I saw the second approach being used in some places like in the following line: https://github.com/balajisrinivas/Face-Mask-Detection/blob/master/detect_mask_video.py#L51
What's the difference?


texts: Using softmax for multilabel classification (as per Facebook paper)

texts:
I came across this paper by some Facebook researchers where they found that using a softmax and CE loss function during training led to improved results over sigmoid + BCE. They do this by changing the one-hot label vector such that each '1' is divided by the number of labels for the given image (e.g. from [0, 1, 1, 0] to [0, 0.5, 0.5, 0]).
However, they do not mention how this could then be used in the inference stage, because the required threshold for selecting the correct labels is not clear.
Does anyone know how this would work?


texts: Quantifying the Reproducibility of LDA Models

texts:
I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my client. My biggest concern at this point is whether my models are reproducible or not. I understand that LDA models are probabilistic, so separate models run on the same data under the same parameters are not guaranteed to be identical.

However, for a model to be considered reliable you would expect duplicate models to come out looking roughly the same. A more reproducible model would be a more accurate representation of the text it is modeling. I wrote a script to find the average similarity between two distinct models (found here: https://github.com/HarryBaker/gensim/tree/topic2topic_seperate_models). The idea behind it is that I try to force a one to one relationship between two models' topics, and then find the average similarity of each match. The assumption is that a reproducible model would have a clear bijection between a duplicate.

From my early research it seems like training a model for longer increases the similarity of duplicate models. (Models trained under 500 iterations were more similar than those trained under 150 passes).

What I'm wondering is if there's been any papers or studies done on the reproducibility of LDA models, or if anyone has any ideas. I do not want to fix a seed. I don't want to be able to output an identical model--I want to be confident that for any model I create, a model trained under the same data and parameters will be almost the same.


texts: Ingestion of periodic REST API Calls into Hadoop

texts:
We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).

I've already done Twitter ingestion using Flume, but I don't think using Flume would suit my current use-case because I am not using a continuous data firehose like this one in Twitter, but rather discrete regular time-bound invocations.

Please I would like to hear suggestions / alternatives (if there's easier than what I'm thinking right now) about design and which Hadoop-based component(s) to use for this use-case. If you feel I can stick to Flume, then kindly give me also an idea how to do this.


texts:
I'm using Faster RCNN to digitalize a document. It is able to recognize zones of the document (name, name of enterprise, name of director, etc) but cannot classify them accurately.
So, my problem is that I have some rectangles predicted from Faster RCNN of well recognized elements but with bad labels.
How can I classify those rectangles accurately? Should I use a downstream model?


texts: Some confusions on Model selection using cross-validation approach

texts:
https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation explains the procedure and the importance of doing cross-validation to assess the performance of the method/ classifier. I have few concerns which I could not clearly understand from that answer. It will be immensely helpful if these are clarified.

Consider that I am using the Matlab's fisheriris dataset. The variable meas contains 150 examples and 4 features. The varaible species contains the labels. I have put the data and labels into a variable: Data = [meas species]According to the procedure outlined above,

I have split the data set Data using cvpartition into 60/40 where 60% is the Xtrain and 40% is a separate Xtest data subsets.
Using Xtrain I perform k fold cross-valiation and inside each fold I validate the model using the indices from Xtrain. This loop is used to tune the hyperparameters of the model. I never use Xtest in selecting the hyperparameters. Is my understanding correct?

Confusion 1) The answer in the link says

You build the final model by using cross-validation on the whole set to choose the hyper-parameters and then build the classifier on the whole dataset using the optimized hyper-parameters.

"use the full dataset to produce your final model as the more data you use the more likely it is to generalise well"

I am a bit confused on what  dataset  and  whole set  are we referring to and how is building the final model by using cross-validation on the whole set using the selected hyper-parameters different from building the classifier on the whole dataset using the hyper-parameters?

I wanted to verify if my understanding of this part is correct or not. Does this statement mean that using  the cross-validated hyper-parameters obtained using the Xtrain, should the classifier be build by re-training on the Xtrain subset or on  Data ?

Should my final model be the one from Data?

Confusion 2) What is the role of the unseen Xtest data set? In papers is the performance reported on the Data or on the untouched Xtest ?


texts: Factorization Machine - prevent over fitting

texts: I was recently asked this question in an interview and wondered what the answer would be - "How do Factorization Machines get around the overfitting problem when using second-order interactions?"


texts: Can BERT/ELMo be used (or retrained) to generate a text in both directions?

texts: Text generation is perhaps one of the fun things to do with old NGram or new BERT/ELMo models. I am wondering can BERT be used to generate text from the end of a sentence, or better in both directions. That is instead of giving some starting words, we give some ending words.


texts: why always predict same class for all input

texts:
import numpy as np
from sklearn import datasets as ds

iris = ds.load_iris()
x = iris.data
y1 = iris.target

x = x / x.max()
y1 = np.matrix(y1)
np.random.seed(1)
y = np.zeros((y1.size, y1.max() + 1))
y[np.arange(y1.size), y1] = 1

class NeuralNetwork(object):

    def __init__(self):
        self.inputSize = 4
        self.outputSize = 3
        self.hiddenSize = 5

        self.W1 = np.random.randn(self.inputSize, self.hiddenSize) * 0.01
        self.W2 = np.random.randn(self.hiddenSize, self.outputSize) * 0.01
        self.b1 = np.random.randn(1, self.hiddenSize)
        self.b2 = np.random.randn(1, self.outputSize)

    def forward(self, x):
        self.z = np.dot(x, self.W1) + self.b1
        self.z2 = self.sigmoid(self.z)
        self.z3 = np.dot(self.z2, self.W2) + self.b2
        o = self.sigmoid(self.z3)
        return o

    def sigmoid(self, s):

        return 1 / (1 + np.exp(-s))

    def sigmoidPrime(self, s):

        return s * (1 - s)

    def backward(self, x, y, o):
        l = 0.2
        self.dz2 = y - o
        self.dw2 = (1 / 150 * self.dz2.T.dot(self.z)).T

        self.db2 = 1 / 150 * np.sum(self.dz2, axis=0).reshape(1, 3)

        self.dz1 = 1 / 150 * self.W2.dot(self.dz2.T).T * self.sigmoidPrime(self.z)

        self.dw1 = (1 / 150 * self.dz1.T.dot(x)).T

        self.db1 = 1 / 150 * np.sum(self.dz1, axis=0).reshape(1, 5)

        self.W1 = self.W1 - l * self.dw1

        self.W2 = self.W2 - l * self.dw2

        self.b1 = self.b1 - l * self.db1

        self.b2 = self.b2 - l * self.db2

    def train(self, x, y):
        o = self.forward(x)
        self.backward(x, y, o)

when I run this code it predicts always same class
but when I change backward function to this one:

def backward(self, x, y, o):
    self.o_error = y - o
    self.o_delta = self.o_error * self.sigmoidPrime(o)
    self.z2_error = self.o_delta.dot(self.W2.T)
    self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)
    self.W1 += x.T.dot(self.z2_delta)
    self.W2 += self.z2.T.dot(self.o_delta)

It predicts correctly. Why first function doesn't work correctly (backward)?


texts: How to use boolean data in DecisionTreeClassifier in sklearn?

texts:
I am trying to build a decision tree using python and sklearn DecisionTreeClassifier.

One of the data_type used for splitting the tree  is Boolean(let it be x).However the tree that is generated contains comparisons like x<=0.5 .
This does not make sense.

Can anybody suggest how to use boolean values in Decision trees.


texts: What is the difference between dynamic programming and Q-learning?

texts: What is the difference between the DP-based algorithm and Q-learning?


texts: Using Predictions: Data does not have the same target as predictors

texts:
As per the Orange video tutorial: youtu.be/D6zd7m2aYqU?list=PLmNPvQr9Tf-ZSDLwOzxpvY-HrE0yv-8Fy

I'm trying to make predictions on a new dataset based upon one that's already been 'coded' - both data sets are formatted exactly the same, where the 'training' set has all entries marked as either 1 or 0, see below:

whereas the second set contains no values, as I hope the 'predictions' feature will try to guess the status:

Trouble is, when I look at the predictions widget, it does nothing, displaying 'Data does not have the same target as predictors'

I've only been using Orange for two days, so I imagine it's a really simple error on my part.

Any help would be greatly appreciated.

John


texts: How to generate image using deep learning

texts:
There are lots of examples of classifier using deep learning techniques with CIFAR-10 datasets. The way it works is that, train thousands of images of cat, dog, plane etc and then classify an image as dog, plane or cat. But I want to do the reverse thing. I want to train dog, cat, planes and it should output images. Here is my idea

Group similar/clear images of cat as value 1, a less similar/blur images as 1.1 and so on.
Similarly, group similar images of dog as value 2 and less similar images as 2.1 and so on ...
Do the same for all types of images.

Generated dataset should look like this

input    output
 1      pixels(24*24) of a cat images(clear)
 1      pixels(24*24) of a cat images(clear)
 .
 .
 1.1      pixels(24*24) of another cat images(blur)
 1.1      pixels(24*24) of another cat images(blur)
  .
  .
  and so on

Now train values of input and label of images as output. Input will be just 1 dimensional, may be I will think of some other data. Output layer will have 24*24 i.e 576 units or neurons.
At the end of training, I want something like this, if I give a input for example 1.15 it should output a new image since we trained with values 1.1 and 1.2 but we didn't train with 1.11 or 1.115.

Please give me some idea how can I do this? Any link to example or papers would be nice.


texts: embedding layers in neural network when 2 values have same information value

texts:
One of the techniques to reduce cardinality of categorical features is to use embedding layers in neural network. Another method is weight of evidence. When weight of evidence is calculated, it looks at one feature at a time.
For simplicity, say I am building a model to predict if a person in the US likes football. For the sake of my question, say state A and state B have exact same population and exactly the same number of people like football in each state but fan bases are very different. Maybe younger people like football in the state A but older people like football in the state B. If you use weight of evidence for the feature "State", the both states would end up having the same values.
In neural network model, if I use an embedding layer for State, it is technically connected and interacted with "Age". Would the N-dimensional representation of the 2 states be very very similar? Or Would they not because other features such as age are very different?
Either way, can we 'prove' that mathematically?


texts: Text extraction from large pool of documents of different formats

texts:
I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, xlsx, docx, and other Microsoft Office file types.

Some documents contain digital text (90%), other documents are scanned copies (10%). Some documents are in english (5%), some are in russian (5%) and rest of them is mix of english and russian (containing translations with similar meaning). Also some documents are packed into zip or rar archives.

What is the best approach to parse all this documents for text data?

What is the best approach to store extracted data?

How to make this data searchable? E.g. if I want to list top 30 documents that contain text similar to my search paragraph.

PS: I am interested to do this task in the shortest time possible, so I guess possible solution would involve big data techniques and distributed computing.


texts: Build a binary classifier with only positive and unlabeled data

texts:
I have 2 datasets, one with positive instances of what I would like to detect, and one with unlabeled instances. What methods can I use ?

As an example, suppose we want to understand detect spam email based on a few structured email characteristics. We have one dataset of 10000 spam emails, and one dataset of 100000 emails for which we don't know whether they are spam or not.

How can we tackle this problem (without labeling manually any of the unlabeled data) ?

What can we do if we have additional information about the proportion of spam in the unlabeled data (i.e. what if we estimate that between 20-40% of the 100000 unlabeled emails are spam) ?


texts:
I'm new to regression and we are doing a very simple exercise in a course. I'm taking to get a basic understanding of GD and SGD for a linear regression.
From my understanding, the only difference between GD and SGD is that instead of performing the algorithm on dataset size m as is processing in GD, SGD performs the operation on subsets of m.
My question is, for SGD does one simply perform the algorithm on the mini-batch, or is there some sort of summation of the results to come out with a final answer? Apologies if I'm not asking in the correct terms, I'm newer to some of the mathematical concepts involved.


texts: Tensorflow RNN: Batching data of different length

texts:
I'm starting to experiment with neural networks in Google's tensorflow framework, and in particular am working on a model that has an LSTM layer. I am using the RNN tutorial to get me started.

To train my model, I have sequences of data that are several thousand data points long. In the tutorial, the author is processing small sequences of data (short sentences of paragraphs) and concatenates those sequences together, separated by <end of stream>, divides the entire data set equally into batch_size number of chunks and processes those chunks in parallel. I would like to process each of my sequences of data separately from start to finish, but would like to be able to process several of them in parallel, as the rnn_cell objects are set up to do (they process batch_size sequences at onces).

This would be straight forward to do if the sequences were of the same length, but they are not. An rnn_cell.LSTMCell expects the data to come in with a fixed dimensionaltiy, batch_size x input_size, so there's a problem once some of the sequences reach their end and my batch_size wants to decrease.

How can I work around this? I don't think I can 'left' or 'right' justify my data so that they line up without affecting the training.


texts: Interpretation of the data through scatter plot

texts:
I was exploring the data and had observed the data points are forming a triangle on the lower side.
x-axis: Total items
y-axis: Cancelled items

Can someone help me in interpreting this data? And help me how to proceed further in analyzing and building a model?


texts:
I have a dataset of the type:

Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests x and z at time -3. OR it is the minimum time between last test and today if diagnosis was negative.

How would I create a new feature that accounts for both the time of different tests and the different types of events and their numeric values for each patient?


texts: K fold cross validation reduces accuracy

texts:
I am working on a machine learning classifier and when I arrive at the moment of dividing my data into training set and test set Iwant to confron two different approches. In one approch I just split the dataset into training set and test set, while with the other approch I use k fold cross validation.

The strange thing is that with the cross validation the accuracy decreases, so if I have 0.87 with the first approch, with cross validation I have 0.86.

Shouldn't cross validation increase my accuracy? Thank's in advance.


texts: Which recommender system approach allows for inclusion of user profile?

texts:
I wanted to enhance a recommendation engine with information relying not only on past purchases or ratings but also on behavioral and demographical variables like sex, age, location, service usage frequency or hours. This information may be sparse (e.g. the user may not have provided me with his age).
Can you suggest an approach that would allow for it?

My first approach:
After screening the general approaches to recommender systems, I think the one that is able to implement my idea is custom(?) item-based collaborative filtering.
To be more precise: I would include user profile information in the same way item ratings are introduced, probably with two alterations to raw data:

Normalize scale to be compliant with the rating scale.
Add a parameter in my algorithm that would put a weight (e.g. 20% or 50%) on user profile rows, as there may be 10 user-profile
variables and a million product items.


texts: How to train ML model with multiple variables?

texts:
I am trying to learn Machine Learning concepts these days. I understand in a traditional ML data, we will have features and labels. I have following toy data in my mind where I have features like 'units_sold' and 'num_employees' and a label of 'cost_$'. I would like to train the model to learn these features and label for a particular 'city' and 'store'. For example if I perform Linear Regression, the model learn intercept and coefficient for the city and store it relates to. When I input units_sold and num_employees for next year, I get the prediction.

city        store   units_sold  num_employees   cost_$
New York    A       10          4               11000
New York    B       12          4               11890
New York    C       14          5               15260
New York    D       17          6               17340
London      A       23          5               22770
London      B       27          6               25650
London      C       22          3               21450
Paris       A       4           2               5200
Paris       B       7           3               9590

I'm trying to brainstorm about it and would like to know how to approach this problem?


texts: Why some ML models can't take advantage of text ordering information?

texts:
On this google tutorial (https://developers.google.com/machine-learning/guides/text-classification/step-4) it is said:

  >
  Build n-gram model [Option A]
  We refer to models that process the tokens independently (not taking into account word order) as n-gram models. Simple multi-layer perceptrons (including logistic regression), gradient boosting machines and support vector machines models all fall under this category; they cannot leverage any information about text ordering.
  >

Then

  >
  Build sequence model [Option B]
  We refer to models that can learn from the adjacency of tokens as sequence models. This includes CNN and RNN classes of models. Data is pre-processed as sequence vectors for these models
  >

Why some ML model (Simple multi-layer perceptrons, gradient boosting machines and support vector machines models) cannot leverage any information about text ordering?

Is it because they can't accept floats on their input? (Not sure if those ML models accept floats on their input. I am guessing they can't.)


texts: One-hot & interaction one-hot on multiple categorical

texts:
I was wondering if there is any value to creating combined features out of multiple categorical variables when the individual categorical variables are already one-hot encoded?
Simple example: there is a variable P with categories {X,Y} and and a variable Q with categories {Z,W}. After one-hot we would have 4 variables, P.X, P.Y, Q.Z and Q.W.
In this scenario I'm wondering if the algorithm (Xgboost or a deep neural network) would sufficiently learn interaction effects between these or is there further value to creating variables: X.Z, X.W, Y.Z, Y.W which would be the unique combinations of P and Q.
The reason I am asking is to try to asses whether to embark on creating these interaction variables in my real-world scenario, where I have 7 such categorical features and 6-15 categories each which would mean thousands of new variables to account for all possible levels of permutations.


texts: Calculate cosine similarity in Apache Spark

texts:
I have a DataFrame with IDF of certain words computed.
For example

(10,[0,1,2,3,4,5],[0.413734499590671,0.4244680552337798,0.4761400657781007, 1.4004620708967006,0.37876590175292424,0.48374466516332])

 .... and so on

Now give a query Q, I can calculate the TF-IDF of this query.
How do I calculate the cosine similarity of the query with all documents in the dataframe (there are close to million documents)

I could do it manually in a map-reduce job by using the vector multiplication

  Cosine Similarity (Q, document) =  Dot product(Q, dodcument) / ||Q|| * ||document||

but surely Spark ML must natively support calculating cosine similarity of a text?

In other words given a search Query how do I find the closest cosines of document TF-IDF from the  DataFrame?


texts:
Could you please assist me with to following question?

I have a customer activity dataframe that looks like this:

It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and zeroes represent customer activity. If a customer was active during a particular month then there will be a 1, if not - 0. I need determine those customers that most likely (+ probability) will not be active during the next 6 months (2018 July-December).

Could you please direct me what approach/models should i use in order to predict this? I use Python.

Thanks in advance!


texts: Trainer Component Error in Tensorflow Extension : AttributeError: module 'user_module' has no attribute 'trainer_fn'

texts:
I want to build a tfx pipeline on own dataset.
I could use some advice on the transform code that I wrote and understand better.

I've done the ExampleGen, StatisticsGen, SchemaGen, ExampleValidator, Transform, and having an error in Trainer components.

How I could fix these issues?

ERROR:

c:\lib\site-packages\tfx\components\trainer\executor.py in _GetFn(self, exec_properties, fn_name)
    128     if has_module_file:
    129       return import_utils.import_func_from_source(
--> 130           exec_properties['module_file'], fn_name)
    131
    132     fn_path_split = exec_properties[fn_name].split('.')

c:\lib\site-packages\tfx\utils\import_utils.py in import_func_from_source(source_path, fn_name)
     66       user_module = types.ModuleType(loader.name)
     67       loader.exec_module(user_module)
---> 68       return getattr(user_module, fn_name)
     69
     70   except IOError:

AttributeError: module 'user_module' has no attribute 'trainer_fn'

CODE:

def get_model(show_summary=True):

#one-hot categorical features
num_A = 4,
num_B = 3,
num_C = 2,
num_D = 8,
num_E = 12,
num_F = 4,
num_G = 16,
num_H = 26

input_A = tf.keras.Input(shape=(num_A,), name="A_xf")
input_B = tf.keras.Input(shape=(num_B,), name="B_xf")
input_C = tf.keras.Input(shape=(num_C,), name="C_xf")
input_D = tf.keras.Input(shape=(num_D,), name="D_xf")
input_E = tf.keras.Input(shape=(num_E,), name="E_xf")
input_F = tf.keras.Input(shape=(num_F,), name="F_xf")
input_G = tf.keras.Input(shape=(num_G,), name="G_xf")
input_H = tf.keras.Input(shape=(num_H,), name="H_xf")

fl = keras.Input(shape=(75,))
dense = layers.Dense(35, activation = "relu")
x = dense(fl)
x = layers.Dense(15, activation="relu")(x)
outputs = layers.Dense(1, activation="sigmoid")(x)

_inputs = [input_A, input_B, input_C, input_D, input_E, input_F, input_G, input_H]

model = keras.Model(inputs=inputs, outputs=outputs)

model.compile(optimizer='rmsprop',
          loss='binary_crossentropy',
          metrics=['accuracy'])

if show_summary:
    model.summary()

return model


texts:
I have a hybrid data that contains 15 categorical data and 4 continuous data.
I need to implement a prediction on the data. So as I don't have any labeled data, I need to implement the unsupervised algorithm.

Now Using PCA my dimensions reduced to 5 but still it has 3 categorical and 2 continuous data. Now to create a prediction algorithm, I can use KMeans for continuous data or KModes for categorical data, But I couldn't find any algorithm that works on both types of data.

Can anyone know any algorithm that works on both the data types for unsupervised prediction?


texts: Dummies Variables and Scaling in Regression Problems

texts:
I was wondering if having dummies variable and scaling other variables could joke my model. In particular, I have implemented a Random Forest Regressor by using scikit-learn, but my data model is composed by a set of dummies varibles and 2 numerical variables. I approached in this way:

Convert categorical in  dummies variables
Separate the numerical variables
Scale with Standard Scaler from scikit-learn the numerical variables (at point 2)
Join the dummies and numerical
Split train, test
train the model

Would this approach create an inappropriate bias considering the different scale from dummies and the scaler numerical?
Or, at least, is it correct?


texts:
I was trying to categorical variable engineering following this paper. The code is the following:

import random
import pandas
import numpy as np
import tensorflow as tf

from tensorflow.contrib import layers
from tensorflow.contrib import learn
from __future__ import print_function

from sklearn.preprocessing import LabelEncoder

My dataset looks like the following. It's has 2 independent variable ('X1' & 'X2')and 1 dependent variable ('lable'). 'X2' is the categorical variable. I want to create an embedding vector for this variable and run the simple linear regression to predict 'label'using Tensorflow.  I could use any other method. But since linear regression is easiest to understand, I'm trying that.

df = pd.DataFrame({'X1': np.array(["A","A","B","C","B","C","B","C","C","B",
                         "A","B","A","C","A","A","C"]),'X2': np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1]),
                       'label': np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])})

For variable 'X1', I'm creating levels.

encoder = LabelEncoder()
encoder.fit(df.X1.values)
X = encoder.transform(df.X1.values)

Recreating dependent variable list.

y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])

Setting Hyper-parameters

training_epochs = 5
learning_rate = 1e-3
cardinality = len(np.unique(X))
embedding_size = 2
input_X_size = 1
n_hidden = 10

Setting up variables:

embeddings = tf.Variable(tf.random_uniform([cardinality, embedding_size], -1.0, 1.0))

h = tf.Variable(tf.truncated_normal((embedding_size + len(df.X1), n_hidden), stddev=0.1))

W_out = tf.get_variable(name='out_w', shape=[n_hidden],
                            initializer=tf.contrib.layers.xavier_initializer())

Embedding:

embedded_chars = tf.nn.embedding_lookup(embeddings, x)
embedded_chars = tf.reshape(embedded_chars, [-1])
embedded_chars= embedded_chars + np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])

Multiplying with Hidden Layers:

layer_1 = tf.matmul(embedded_chars,h)
layer_1 = tf.nn.relu(layer_1)
out_layer = tf.matmul(layer_1, W_out)

# Define loss and optimizer

cost = tf.reduce_sum(tf.pow(out_layer-y, 2))/(2*n_samples)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

Run the graph

init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    for epoch in range(training_epochs):
        avg_cost = 0.

        _, c = sess.run([optimizer, cost],
                        feed_dict={x: X, y: Y})
print("Ran without Error")

While running the code, I'm getting the following error.

  ValueError: Shape must be rank 2 but is rank 1 for 'MatMul_1' (op:
  'MatMul') with input shapes: [17], [19,10].

I'm not able to add the continuous variable with embedding variable.

Can anyone please guide me how to do it?

Thank you!


texts:
I'm using the MLPRegressor in sklearn to train a network with approx 1000 inputs and a continuous output variable.  Essentially, the issue is one of image classification (1000 pixels) with the distribution of the pixels related to a continuous variable output.

I can generate an arbitrarily large training set, but run into memory problems running this on a single machine when loading arrays into numpy.  I've selected hyper parameters that give a reasonable result with a limited (5k) training set, but am trying to work out the best way to train the network with two orders of magnitude more data.  Is there a way to pass mini-batches to sklearn myself so that I can manage the memory and IO?  Is training and retraining the network helpful if I can setup a means to ensure I read batches for each training step in a randomised fashion?

I'm guessing this is a common problem, but I'm struggling to find a sensible solution.


texts: How can I plot/display a dataset or an image distribution?

texts:
I want to view a specific image or a dataset's distribution, and see if they are different.
Does simply writing something like :

# mydataset.shape = (50k,32,32,3)
plt.hist(mydataset.reshape(-1))

do the trick? or should I be doing something else?
for example doing so on cifar10 gave me this plot :

However, it doesn't look right, there are 50K training images, and  I don't know how to interpret this or even if it is the correct way of doing it in first place!

if I do :

#dataset shape is (50k, 3072)
plt.hist(mydataset.reshape(-1,32*32*3))
#and testset which is (10K,3072)
plt.hist(mytestset.reshape(-1,32*32*3))

This is what I get :

So this is very puzzling to me, and I have no idea how to go about this!
Since there are 10 bins, is it giving me the distribution of each class(cifar10 as 10 classes) ? if it is so why do I get the same shape when I try to get the distribution of a single image?
I mean when I try to do :

#image is of shape (1,3072)
plt.hist(mytestset[0])

this is what I get :

and interestingly, the histogram for the whole test set looks like this :

#dataset shape is (10k,32,32,3)
plt.hist(mytestset.reshape(-1,32*32*3))

Why do I get ten bins for the single image as well?
what does each axis mean then?
What should I be looking for when it comes to image/dataset distribution?
Is it the raw values for the whole dataset only?
or Is it the raw values for each class? or even each image?


texts: Domain specific language to describe ensemble model

texts:
I'm looking for some tool/library/widely used approach to describe Hierarchical Model structure like ensemble:

It's absolutely straightforward how to do it with simple ensemble like

It can be explained even in yml format like this:

code: "cats"
models:
  - code: "siamese"
  - code: "persian"
  - code: "sphynx"

It's obvious from code that execute simple ensemble model that leaves should go first and parent model aggregates their results.

But it's looks difficult to describe in the same way hierarchical model because it's executed in inverse direction. Probably, someone can suggest approach how to do it?


texts: What is the scientific term/keyword for "big data time series"?

texts: There are business articles from 2014 and 2015 that the time series analysis of the big data is the next big thing. However Google returns almost no scientific articles under the query "big data time series". I am sure the the research happens in this direction but what terms should I use to find this research? Big Dependent data? Large time series? Temporal big data? Streaming big data?


texts: suggest ingredients based on recipe title

texts:
I would like to construct system that would suggest user ingredients once he/she inputs title of the recipe.

I think that this is the task of machine learning or AI, but on the other hand I am pretty new to ML and generally in AI development and I feel kinda lost, since I don't know where to start and with what to start.

So far, I've managed to crawl data from web page with recipe title and ingredients which are used for this recipes.

My thoughts are that, this problem may be solved with suggester system, but on the other hand I think classification / clustering algorithms may be used to divide recipes in clusters / categories and once input title is associated to cluster ingredients may be generated from that cluster, but I don't know which is best solution and I wonder if there may be other ones.

Thanks in advance!

I'm posting my temporary solution which may help others solve similar problems and I think this is pretty basic, but works nice:

import all data to database (postgres in my case), where Recipe table has only name and Ingredient table has name and ForeignKey to recipe
once user inputs recipe name (rname), I run query using postgres' Trigram Similarity module to detect similarity between rname and recipe table names. Then I filter out recipes which similarity is greater than .1 (for example)
After that, I get all ingredients which are linked to filtered recipes, annotate ingredient usage for individual ingredient (calculate ingredient usage ratio). And order by usage ratio DESC
Finally, I return ingredients (10 mostly used ones)

I think, looking through proposed solutions and also posting them.


texts:
I tried everything and I am not sure how to resolve the following error:

  ValueError: bad input shape (5634, 2)

This is my first machine learning example so please bear with me. This is the python code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
from pylab import rcParams
from IPython import get_ipython

ipy = get_ipython()
if ipy is not None:
    ipy.run_line_magic('matplotlib', 'inline')
# Loading the CSV with pandas
data = pd.read_csv('...WA_Fn-UseC_-Telco-Customer-Churn.csv')

# Data to plot
sizes = data['Churn'].value_counts(sort = True)
colors = ["grey","purple"]
rcParams['figure.figsize'] = 5,5
explode = (0.1, 0)  # explode 1st slice
labels = "Yes","No"
# Plot
plt.pie(sizes, explode=explode, labels=labels, colors=colors,
        autopct='%1.1f%%', shadow=True, startangle=270,)
plt.title('Percentage of Churn in Dataset')
plt.show()

data.drop(['customerID'], axis=1, inplace=True)
tc = (data['TotalCharges'].str.strip())
#print(tc)
data['TotalCharges'] = pd.to_numeric(tc)

data["Churn"] = data["Churn"].eq('Yes').astype(int)
Y = pd.get_dummies(data["Churn"].values).fillna(0)
X = pd.get_dummies(data.drop(labels = ["Churn"],axis = 1)).fillna(0)

print(X.shape) #(7043, 45)
print(Y.shape) #(7043, 2)

# Create Train & Test Data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=101)

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
result = model.fit(X_train, y_train)

Why does the error

  bad input shape (5634, 2)" when X.shape is (7043, 45) Y.shape is
  (7043, 2)?


texts: What is the most used format to save data with type information

texts:
I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R.

Currently, I am using csv files to save the data. Everybody supports csv. But csv does not support type information. This makes it sometimes cumbersome to load a csv file because I must check all the column types. This seems unnecessary because the SQL database already specifies the types of the columns.

I want to know if there is a broadly accepted file format to save data that also specifies the type of the columns.

Currently I am working with SQL databases, FME ETL'ing and R but I think this is an issue for every data tranfer.


texts: Supervised learning approach - creating my own labels

texts:
Scenario - I have data that does not have labels but I can create a function to label the data based on behavior and deploy the model so I don't have to keep labeling the data. Is this considered machine learning ?

Objective: Classify accounts with Volume spikes based on high, medium or low labels to deploy on big data (trillions of lines of data) .

Data: The data I have includes the following attributes:
Account, Time, Date, Volume amount.

Method:

Create a new feature column called spike and create a pandas function to ID a spike greater than 5.  Is this feature engineering ?
Next I create my label column and  classify it as low medium or high spike.
Next I train a machine learning classifier and deploy it to label future accounts with similar patterns in big data.

Thoughts on this process ? Is this approach correct for machine learning ?


texts: Evaluating recommendations quality and accuracy

texts:
I'm developing a recommendation system, that should provide my clients what actions they should take in order to hit certain targets.
The underlying mechanics of the process is physical - where both actions and outcomes can be measured physically.

My current algorithm is based on a predictive model that can predict expected result for a given action.
This result can be scored, and thus scoring* many outcomes can lead to finding the best action - the action that will be recommended.

*the scoring is based on distance between outcomes and targets.

My predictive model performance is good(low MAE). Its performance estimation is based on predicting already taken(historic) action and comparing its prediction to actual(historic) outcomes.

My problem arise when I try estimating the recommendations quality, since many of the recommendations are actions that were not taken, I have little ways evaluating their quality.

Ive tried using actual outcomes as 'targets' to my recommendation system, and compare the recommended action to actually taken action.
I am not sure this is the right way to go, and any advice will be appreciated.


texts: (Serious) Dataset of paedophilic Youtube comments (or similar)?

texts:
I think it would be useful to create a model that tries to predict whether a youtube comment is paedophilic - maybe the model should also take into account the channel name/description/front image.
It's not an easy task but at the moment I'm just looking for data.
I know it's a sensitive topic - but does anyone know of a dataset out there with the characteristics I need?
Thanks


texts:
What are bottleneck values and how are they generated?
How does the next-to-last layer of Inception use these bottlenecks to generate the accuracy?
How is the final accuracy calculated?


texts: Which (naive) NLP method for correlating human messages in chatrooms?

texts:
Suppose an online chatroom is filled with many "alt" accounts - i.e, multiple accounts are being controlled by one user, a troll. This user leverages multiple accounts in order to steer the conversation in particular directions to suit their needs. This user may also be altering their mode of speech per account, to avoid detection.
Suppose I want to detect these alt accounts by using some kind of NLP classifier. What would the best method be? Where "best" means effective, but also relatively simple to set up - as in something you could do in python with relatively basic sklearn modules.
What I've tried so far is to collect the 100 most frequent words used by each user, and just throw that corpus into a sklearn.feature_extraction.text.TfidfVectorizer, and then looking at the pairwise similarity. These are mostly stop words, words which most NLP articles online are telling me to ignore. But I think these basic (almost unconsciously-used) words are less likely to be subject to obfuscation. The user may, for example, be applying varying spellings of less-common words (realize vs realise etc) - but the user could not plausibly have alternative spellings for ["the", "of", "you", "one"] etc.
Like I said, that isn't an "advanced" approach, but it's an example of the general level of complexity that I'm willing to undertake. Is there are any validity to that approach? If not, can you suggest a better one?


texts: Finding optimal time series using convolution

texts:
we logged sensor data while milling a workpiece. At several points, the workpiece was damaged and this induced a certain sensor data time series. Due to noise and since its a real world measurement, this certain sensor data time series is somewhat different for each workpiece damage that occurs. In my dataframe, i have the sensor data and the points where a damage occured.
I want to detect workpiece damages using convolution on the sensor data. My challenge now is: how to identify a certain time series which gives me the best performance when convoluting on the sensor data.
I thought about using historic data with workpiece damages and finding the optimal time series (maybe using simulated annealing) which gives me the highest value when summing up the convolution.
Maybe there is a smarter way to do this or somebody can tell me whether there is an algorithm to do this.
Thanks for all your help


texts: Is there a clustering algorithm that works with only pairwise distances as input?

texts:
My data are places for which I know all pairwise travel times (='distances'), and I want to cluster those places minimising the total pairwise travel time inside a cluster.

K-means can't be used because it's centroid-based and the 'distance' is a duration that is provided, not computed thanks to coordinates
DBSCAN can't be used because it excludes outliners and I want to include each places in a cluster (not 100% sure for this one)

Bonus: a Java library would be much appreciated


texts: Using Pandas to_numeric() in Azure Machine Learning Studio

texts:
I am facing an issue that Azure Machine Learning Studio fails to find the to_numeric method in pandas.

After reading a .csv in AMLS I try to process it in a python script. The line that is throwing me an error is:

dataframe1['Monthly Debt'] = pd.to_numeric(dataframe1['Monthly Debt'])

pd of course is pandas, dataframe1 is my working dataframe. The error thrown is:

AttributeError: 'module' object has no attribute 'to_numeric'

Of course everything works on my local python. Do you have any idea what AMLS might be talking about?


texts: In pandas I can set the date as index, and then run df.plot() to see a line chart. How do I make that line chart stacked as in the picture below?


texts: Does anyone use Julia programming language?

texts:
Is anyone using Julia (http://julialang.org/) for professional jobs?

Or using it instead of R, Matlab, or Mathematica?

Is it a good language?

If you have to predict next 5-10 years: Do you think it grow up enough to became such a standard in data science like R or similar?


texts: Numpy: how to add zeros while reshaping the array

texts:
I have a scenario where I have three numpy arrays -
indices = np.array([1,2,3], [4,2,3])
distances = np.array([0.2, 0.3, 0.4], [0.3, 0.4, 0.5])
weights = np.array([0.2, 0.8])
union_set = np.array([1, 2, 3, 4])

The goal is to perform np.dot operation between the corresponding distances of the indices and weights.

I am trying to avoid n(square) solution where you iterate over each index in the union_set and get the distance as

for index in union_set:
    loc = np.where(indices==index)
    dist = distances[loc]
    weight_array = np.zeros(len(indices))
    dist_array = np.zeros(len(indices))
    np.put(dist_array, loc[0], dist)
    np.put(weight_array, loc[0], weights[loc[0]])
    np.dot(weight_array, dist_array)

The explaination of above will be -

>>> indices = np.array([[3,6,9], [4, 3, 8], [7, 3, 3]])
>>> distances = np.array([[0.1, 0.2, 0.3], [0.2, 0.5, 0.8], [0.3, 0.4, 0.5]])
>>> weights = np.array([0.6, 0.4, 0.9])
>>> weight_array = np.zeros(len(indices))
>>> dist_array = np.zeros(len(indices))
>>> loc = np.where(indices==7)
>>> loc
(array([2]), array([0]))
>>> dist = distances[loc]
>>> dist
array([0.3])
>>> np.put(dist_array, loc[0], dist)
>>> dist_array
array([0. , 0. , 0.3])
>>> np.put(weight_array, loc[0], weights[loc[0]])
>>> weight_array
array([0. , 0. , 0.9])

The problem here is, the operation is n(square).
I an trying to solve it between n(square) and n.

One solution I could think of is

>>> indices = np.array([[1,2,3], [2, 5, 9]])
>>> distances = np.array([[0.1, 0.2, 0.3], [0.2, 0.4, 0.6]])
>>> np.where(np.in1d(indices, 1))
(array([0]),)
>>> np.where(np.in1d(indices, 2))
(array([1, 3]),)
>>> np.where(np.in1d(indices, 9))
(array([5]),)
>>> distances.ravel()[np.where(np.in1d(indices, 9))]
array([0.6])
>>> distances.ravel()[np.where(np.in1d(indices, 2))]
array([0.2, 0.2])

Now the distances I have are corresponding to indices in union_set.
But I can't perform np.dot operation here because,

>>> union_set = np.array(list(set().union(*indices)), dtype=np.uint32)
>>> union_set
array([1, 2, 3, 5, 9], dtype=uint32)
>>> distances.ravel()[np.where(np.in1d(indices, union_set))]
array([0.1, 0.2, 0.3, 0.2, 0.4, 0.6])

Any help is appreciated


texts:
import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.read_csv('Downloads/breast-cancer-wisconsin.data.txt',skiprows=1)
df.replace('?', -99999, inplace=True)
df.drop('id', 1, inplace=True )

X= np.array(df.drop(['class'],1))
y= np.array(df['class'])

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y,test_size=0.2)

#clf = neighbors.KNeighborsClassifier()
clf = LinearRegression(normalize=True)
clf.fit(X_train, y_train)

accuracy= clf.score(X_test, y_test)
print(accuracy)

example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,2,2,2,3,2,1]])
example_measures = example_measures.reshape(1,-1)

prediction = clf.predict(example_measures)     ##(example_measures)

print(prediction)

Problem arises when  I run the above command line at Ubuntu or Anaconda:

  ValueError: query data dimension must match training data dimension

How to solve that problem ? I am sure that by method of isolating individual commandline-- and find it appears Error at :

prediction = clf.predict(example_measures)

I try to use :

prediction = clf.predict(X_test).

It is ok.
I really want to predict the example I create. How can I change the code?


texts: How do I control sensor data readings against (measured) outside influences?

texts:
I need some hints in this problem I have.
I have a dataset of electro-dermal activity readings that are, by nature, influenced by the movements of the person that is being surveyed. In the same dataset, however, I also have accelerometer data that I could use to "control" the data. Meaning that, I hope, I could somehow factor in the person's accelerometer readings when looking at EDA readings and filtering them/correcting against posture changes made obvious in the accelerometer values.
As an example, fig. 1 shows EDA (also called galvanic skin response, or gsr) readings with higher peak prominence at the beginning of the experiment:

This second figure shows accelerometer readings in x, y and z directions, appearing to show more activity during these phases of higher peak prominence:

How do I control my EDA values against these accelerometer readings? Any hints to literature or math will be greatly appreciated.
Thanks in advance!


texts: What can be done to increase the accuracy of a biological dataset?

texts: I have a biological unbalanced dataset on which I have applied deep learning, Support Vector Machine (all the kernel functions) and Artificial Neural network for multiclass classification (size: 139 samples , 5 attributes) in python. Unfortunately the accuracy is not exceeding 55%. What can be done to increase the accuracy? If a dataset cannot ever go beyond such an average accuracy, what is the solution?


texts: Over/Under Sampling for Multi-classification

texts:
I'm trying to apply xgboost and random forest for over and under sampling

For imbalanced data:

train shape -> (199991, 23)

However, reverse my expectation. The accuracy went down for both cases.

Questions:

Is under and over sampling always not good?
What else should I consider when applying over and under sampling?


texts: Mean Absolute Error in Random Forest Regression

texts:
I am new to the whole ML scene and am trying to resolve the Allstate Kaggle challenge to get a better feeling for the Random Forest Regression technique.

The challenge is evaluated based on the MAE for each row.

I've run the sklearn RandomForrestRegressor on my validation set, using the criterion=mae attribute. To my understanding this will run the Forest algorithm calculating the mae instead of the mse for each node.

After that I've used this: metrics.mean_absolute_error(Y_valid, m.predict(X_valid)) in order to calculate the MAE for each row of data.

What I would like to know is if the logic I'm following is sound. Am I making a fundamental mistake or missing something here? Should I have used the default MSE based Regressor and then calculate the MAE of each row using the mean_absolute_error function?


texts: Free/open interactive softwares/plugins for end-users' high-dimensional data visualization

texts:
Aside from questions about How to visualize data of a multidimensional dataset (TIMIT), the Purpose of visualizing high dimensional data? and High-dimensional data: What are useful techniques to know?, I am looking at tools for high-dimensional data (numerical only) visualization, possibly embedding some dimension reduction techniques (Sammon maps, t-SNE), that could be used by end-users. I have found for instance:

HiSee (version 1.0.0, 2004-07-05),
GGobi (version 2.1.10a, 2010-03-12),
Data Projector (2014-07-15).

They do not seem to be heavily maintained. Are there more active or efficient developments to follow?


texts: Technical name for this data wrangling process? Multiple columns into multi-factor single column

texts:
What is the technical name for the following data wrangling process? I want to collapse Table A into Table B. (To make the data suitable for ANOVA.)

Table A:

ArmyVet_ID  Served_WW2  Served_KoreanWar    Served_VietnamWar
110001          1              0                    0
110002          1              0                    0
110004          0              1                    0
110005          0              1                    0
110009          0              0                    1
110010          0              0                    1

Table B:

ArmyVet_ID    Served
110001          WW2
110002          WW2
110004          KoreanWar
110005          KoreanWar
110009          VietnamWar
110010          VietnamWar

Also, the question of how to do the above conversion using R has been asked to death on SO. However, there seem to be way too many ways to do it. If anyone's figured out the absolutely best way to do it (quickest, easiest), I'd appreciate pointers.

Update after correct answer marked below: It turns out that Table A is called "wide format" and B is called "long format".


texts:
Recently I have been trying different Scikit-Learn anomaly detection clustering methods, like

DBSCAN
Isolation Forest.

Based on how many training data I use, how I tweak on the algorithms

Example in DBSCAN I play around this min_samples and eps distance.

My problem
Now I am getting different results, however this is where the problem is coming. I don't know how many anomalies I am supposed to get, so this leads to me not being sure what is working best.

I have read a it into adding anomalies yourself into the dataset. Is this a thing?
Meaning that I would know if the found anomalies from the algorithm are the same as the one that have been added to the dataset.

I might have gotten this wrong.

The other problem is that the column I want to check for anomalis has numbers that really differ in magnitude so I also wouldn't what numbers I should add as anomalies.

Would appreciate some help.


texts: What do graphs of signal vs background neural network outputs represent?

texts:
For example, this image shows the output of a neural network I assume but I am not sure how the output is not symmetric. So if the neural network gives a .6 for signal it should give a .4 for background.

This is my neural network output that is symmetric. However, I have not seen anyone else get a symmetric output like this. Does this mean that I'm doing something wrong or does it mean that the first graph is not the output of the neural network.


texts: Fraud detection using auto-encoders and Keras

texts:
I am following this example to learn a bit about the use of auto-encoders in fraud detection.
Now that I reached the end of the article, two questions rose in mind:

Can we train the network in an unsupervised way, freeze the layers weights, add a sigmoid layer as an output then finally use this model as a binary classifier?
Are auto-encoders considered as the state-of-the-art method in fraud detection? What other alternatives can be used in this case?


texts: SeaBorn: Custom axes scaling & spines using PairPlot?

texts:
{ Figured this is the most germane place for this question, but definitely let me know if not! }

Two Asks:

How would I uniformly standardize the size and scale for each of the bivariate (i.e., scatter plots) "subplots" when using the PairPlot functionality in Seaborn (if the data is already standardized)? By default it creates the plots which are sized the same, but scaled differently.
Once the scale is uniform across all plots, I want all of the axes to be formatted such that each plot looks like a coordinate plane (i.e.,4 quadrants; both axes spines centered).

Here's some added context on my specific use case:

I have z-scores for the World Series winning teams for a set of variables (ERA, R, Batting Avg., etc.)
I want to plot all pairwise comparisons for this selected set of variables, and because all the data are already standardized, the axes should range roughly from [-3.5, 3.5].
I'm less interested in regressions, and more interested in "archetypes" (e.g. 1995 Atlanta Braves had the best pitching in baseball but weren't exactly stellar hitters -- they still won the World Series! How many other WS winning teams would be in Quadrant III with them?)

Here's code I've used to make one such plot (not using Seaborn), but I want to take advantage of the power of doing as many pairwise comparison's as possible:

x = zscores_wswinners["teamBA"]
y = zscores_wswinners["ERA"]

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(x, y)
plot_axes_range = [-3, 3, -3, 3]
ax.axis(plot_axes_range, 'equal')

ax.spines['left'].set_position('center')
ax.spines['right'].set_color('none')
ax.spines['bottom'].set_position('zero')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')

ax.axhline(linewidth=1, color='black')
ax.axvline(linewidth=1, color='black')
plt.show()

(This seems pretty tedious, and PairPlot gets me 90%+ of the way to what I desire!)

Anyways, thanks, in advance!


texts: How to interpret a trained Decision Tree

texts:
I built my first decision tree, to predict if students will pass or not - the data set - depending on 30 variables. Now I need to know how to read the decision tree, since many variables were strings and now are floats made me confused.

The 30 variables are:

school sex age address famsize Pstatus Medu Fedu Mjob Fjob reason guardian traveltime studytime failures schoolsup famsup paid activities nursery higher internet romantic famrel freetime goout Dalc Walc health absences

edit: to sum up I used the below code to generate this tree. How can i interpret it and understand the method the program is using, I want to understand every word and number written in the tree.

import pandas as pd
import numpy as np
from sklearn import tree
import graphviz
from sklearn.model_selection import cross_val_score
d = pd.read_csv('student-por.csv', sep= ';')

d['pass'] = d.apply(lambda row: 1 if (row['G1']+ row['G2']+ row ['G3']) >= 35 else 0 , axis=1)
d = d.drop(['G1', 'G2','G3'], axis=1 )

#Converting string data to binary using one-hot encoding
d=pd.get_dummies(d, columns =['sex','activities','school', 'address', 'famsize','Pstatus','Mjob','Fjob','reason','guardian','schoolsup','famsup','paid','nursery','higher','internet','romantic'])
#shuffle rows
d = d.sample(frac=1)
#split traning and test
d_train = d[:500]
d_test = d[500:]

d_train_att = d_train.drop(['pass'], axis=1)
d_train_pass= d_train['pass']

d_test_att = d_test.drop(['pass'], axis=1)
d_test_pass= d_test['pass']

d_att = d.drop(['pass'], axis=1)
d_pass = d['pass']

t = tree.DecisionTreeClassifier(criterion ='entropy', max_depth = 3)
t= t.fit (d_train_att, d_train_pass)

#to export the tree
dot_data = tree.export_graphviz(t,out_file = 'students-tree-3.dot', label ='all', impurity=False, proportion= True, feature_names=list(d_train_att), class_names=['fail', 'pass'], filled = True, rounded=True)
score =t.score (d_test_att, d_test_pass)
print (score)
scores = cross_val_score(t, d_att,d_pass, cv=5)

for maxx in range(1,20):
    t = tree.DecisionTreeClassifier(criterion ='entropy', max_depth =maxx)
    scores = cross_val_score (t, d_att, d_pass, cv=5)

    print ('Max depth %d Accuracy %0.2f (+/- %.2f )' % (maxx , scores.mean(), scores.std()*2 ))

If still not clear please tell me to clarify it further.


texts: How to build Non Linear Vector auto regression using Neural Nets

texts:
I have worked on VAR ( vector Auto regression), which forecasts multiple output values(continuous) when there is linear relationship between all variables. I want to use Neural nets to do so,sothat I can include non linear relation among variables also.

Is there any specific type of NN available?

If not how can I implement it using NN?


texts: Using a 3-D convolutional layer to simulate a 2-D convolutional layer

texts:
I've asked this question on the AI StackExchange
 , but I received no insight so I'd like to ask it here.

Is using a filter of size (1, x, y) on a 3-D convolutional layer the functionally equivalent as using a filter of size (x,y) on a 2-D convolutional layer?

I'm trying to predict some properties of videos with Keras using the following rough architecture:

Feed each frame through the same 2-D convolutional layer.
Take the outputs of this 2-D convolutional layer and feed them through a 3-D convolutional layer.

There are more hidden layers, but these are the main ones that matter and are messing with my dimensionality. The input of Conv2D should be (batch_size, height, width, channels). Each movie has dimensionality (number_of_frames, height, width, channels). I first had the idea to neglect batching of movies entirely, and treat the batch size and the number of frames equivalently. Then, Conv2D would output a 4-D tensor, and I would increase its dimensionality to make the ouput a 5-D tensor that I could input into Conv3D. To do this, Conv3D could only accept inputs of batch size 1.

I decided against this, because I wanted to batch movies. My current thought is to do this:

conv1 = Conv3D(filters=1, kernel_size =(1,12,12), strides=(1,1,1), data_format='channels_last')
conv2 = Conv3D(filters=1,kernel_size=(10,10,10), strides=(1,1,1), data_format='channels_last')

conv1 would represent the 2-D convolutional layer while conv2 would represent the 3-D convolutional layer. Would this idea work? I figure there is the advantage that I can batch now and when I train the 2-D filter, the same 2-D filter is running over every single movie frame. I'm just worried that  the filter in conv1 will fail to go over certain frames, or it will somehow overlap frames when I want the filter to go over every frame individuall


texts: xgboost or lightgbm to handle Binomial problems

texts: I have a dataset containing a column of trials, a column of successes and other features; and, obviously, I can generate a probability column. I would like to use gradient boosting methods (like xgboost or lightgbm) to model the success probability. Which parameter shall I set to handle this in lightgbm or xgboost?


texts: What is a basic object detection/localization ML algorithm that can be used for my relatively simple image set?

texts: I have a relatively simple object localization task. I have an image set that are either uniform or contain an 'object' consisting of a black circular shape at some position in the image. The labels include whether or not it contains the object, and the position of the object (center coordinates + length/width). YOLO feels like overkill and I'm having a hard time figuring out how to get it to work, but I'm having a hard time finding a simpler model that would work.


texts: Calculating the entropy of a neural network

texts:
I am looking to calculate the information contained in a neural network. I am also looking to calculate the maximum information contained by any neural network in a certain amount of bits. These two measures should be comparable (as in I can compare whether my current neural network has reached the max or is less than the max and by how much).
Information is relative, so I define it relative to the real a priori distribution of the data that the neural network is trying to estimate.
I have come across Von Neumann entropy which can be applied to a matrix, but because it is not additive I can't apply it to a series of weight matrices (assuming the weight matrices encode all the information of a neural network).
I found three other papers Entropy-Constrained raining of Deep Neural Networks , Entropy and mutual information in models of deep neural networks and Deep Learning and the Information Bottleneck Principle. The second contains a link to  this github repo, but this method requires the activation functions and weight matrices to be known which is not the case for finding the max entropy of any neural network in n bits.
How can I calculate the amount of information contain in/entropy of a neural network? And how can I calculate the same measure for any neural network in n bits?


texts: Why is this trained model misclassifying new cases?

texts:
Below is text / code from a recent SO post of mine that I hope you could comment on:

Using Keras/Theano, I successfully trained a model using the Keras sample code for CIFAR10. It ran for about 2 hours on a MacPro with a GTX98-Ti GPU, with Theano (latest version) backend.

All original parameters are unchanged and model reached ~81% accuracy. At the end of script the weights (weights.h5, 5MB on disk) and architecture (arch.json) were exported.

In another script I restored the model and weights, and tried to classify single images using:

from skimage.data import imread
import os
from keras.models import model_from_json
from keras.optimizers import SGD

model = model_from_json(open('./arch.json').read())
model.load_weights('./weights.h5')
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

for root, dirs, files in os.walk('./input/singles/'):
    for idx, file in enumerate(files):
        if not file.startswith('.'):
            file_path = root + os.sep + file
            img = imread(file_path)
            img = fn_resize(img, 32) # this is a resize function in another file
                                     # I have plotted images at this point and they are
                                     # correctly sized smaller version of the original
                                     # normalized between 0-1 as in training

            img = img.reshape((1, 3, 32, 32))

            y_pred = int(model.predict_classes(img, 1))
            pred_labels = ['airplane', 'automobile', ' bird', 'cat', 'deer',
                           'dog', 'frog', 'horse', 'ship', 'truck']
            print(pred_labels[y_pred], "prediction \\\", file, "truth")

Essentially it restores the model and architecture, then loops through 10 images from the Net in a folder, resizing them to 32x32 pixels, reshaping the array, and forwarding the data to model.predict_classes().

I was surprised to see the below output, in which none of the images was classified correctly. Has anyone done such an experiment and knows if there are issues with the model build or if my procedure is incorrect?

Using Theano backend.
Using gpu device 0: GeForce GTX 980 Ti (CNMeM is disabled, cuDNN 5005)
1/1 [==============================] - 0s
truck predicted \\\ airplane.jpeg truth
1/1 [==============================] - 0s
airplane predicted \\\ bird.jpeg truth
1/1 [==============================] - 0s
airplane predicted \\\ car.jpg truth
1/1 [==============================] - 0s
airplane predicted \\\ cat.jpeg truth
1/1 [==============================] - 0s
frog predicted \\\ deer.jpeg truth
1/1 [==============================] - 0s
ship predicted \\\ frog.jpeg truth
1/1 [==============================] - 0s
automobile predicted \\\ horse.jpeg truth
1/1 [==============================] - 0s
automobile predicted \\\ dog.tif truth
1/1 [==============================] - 0s
frog predicted \\\ ship.jpeg truth
1/1 [==============================] - 0s
airplane predicted \\\ truck.jpeg truth

Process finished with exit code 0


texts: Signature verification - machine learning

texts:
I have to do a project on signature verification.
My goal is having a program with two images as inupt (one is a genuine signature of a selected subject and the other one is the one that I want to test).
I have a dataset to train the algorithm with 3000 signatures: half of them are genuine and the other ones are forgery, and they are taken from 150 subjects.

My doubt regards the confrontation between the two images I use as input (the genuine one and the one I want to test).
How can I deal with it?


texts:
I was going through an article on kaggle blogs. Repeatedly, the author mentions 'LB score' and 'LB fit') as a metric for effectiveness of machine learning (along with cross validation (CV) score).

With a research for the meaning of 'LB' I spent quite a bit of time, I realised that generally people directly refer it as LB without much background.

So my question is - What is a 'LB'?


texts: Is there a cost associated with converting Koalas dataframe to Spark dataframe?

texts:
I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated with switching back and forth between Koalas and PySpark dataframes?

#convert to pyspark dataframe
df.to_spark()

#convert to kolas frame
koalas_df = ks.DataFrame(df)

Edit: With cost I mean, does it ks.Dataframe(ks) create additional overhead? For example, toPandas() results in the collection of all records in the DataFrame to the driver program. Therefore we can only do toPandas() on a small subset of data.

Since I am switching between Koalas and Spark I am wondering if there is any such overhead or if Koalas "interprets" Spark dataframes without collecting records on the driver. At the moment I am working with a small subset of the data, but I am interested in any drawbacks when using larger amounts.