texts:
I have an input array  X, which is of the shape (38000,32,1); the output array Y is of (38000,1), the element of Y can be 0 or an numerical value, such as 0.040139 or 1.075341, or some other numerical values, etc. A typical vector in X looks like as follows

I would like to build an model to predict Y based on X. This is the model, two convolutional layer, followed by max-pooling, then one convolutional layer, one pooling, then a fully connected layer. The detailed architecture is as follows.

outX = conv_filter(20, 50, strides=1, activation='relu')(inputs)
outX = conv_filter(5, 60, activation='relu')(outX)
 outX = MaxPooling1D(2)(outX)
outX = conv_filter(2, 70, activation='relu')(outX)
outX = MaxPooling1D(2)(outX)

outX = Dense(300, activation='relu')(outX)
outX = Flatten()(outX)
predictions = Dense(1,activation='linear')(outX)
model = Model(inputs=[inputs],outputs=predictions)

The loss function is 'mean_squared_error'.The result does not look very good. One of the reason I can think is about the numerical values for the input vector, which tends to be very small. Can it cause any numerical issue? Are there any suggestions to pre-process them? Besides, are there any improvement can be made on the network architecture? Any suggestions would be highly appreciated.


texts:
suppose I have a large multi-variate TimeSeries dataset with very long sequences (~50k)
I want to reduce each sequence size to constant size for LSTM training.
I thought about splitting each sequence to small groups (~100 batches) and average all of the features of each group.

Is Averaging a good algorithm for LSTM training or should I use RandomSampling for the
data?
How it will affect the sequences distribution?


texts: Self-attention model trained with active learning stops learning after a few iterations

texts:
I'm doing some active learning with uncertainty sampling on a self-attention model implemented in PyTorch. The algorithm works as follows (steps 3-7 are repeated for 14 iterations):
1. Take 10% of the data as training set, L
2. Train the model on L
3. Either rank the remaining samples U by a certain informativeness measure and pick a batch B of the top n samples, or randomly pick a batch B
4. Add B to L
5. Remove B from U
6. Re-train the model on L union B, after resetting the weights of the model (this is for cumulative training)
7. Evaluate the model on the test set (accuracy)

Now, the model performs well in the first 2-3 iterations and shows to learn on the validation set. However, in later iterations it completely stops learning and shows the same training and validation accuracy and loss. The test accuracy then always drops to the one of a random classifier (around 33%), as you can see here:

The test accuracy should increase instead, as the model is trained on a larger dataset. I really cannot understand why this happens. The code used in each iteration is exactly the same, and I always reset the weights before re-training. This is the code I use to reset the weights:
def reset_weights(m):
    for layer in m.children():
        if hasattr(layer, 'reset_parameters'):
            layer.reset_parameters()

And then:
model.apply(reset_weights)

Can it have something to do with how PyTorch handles the cache? The same procedure works perfectly fine on models implemented with Keras.


texts: Is there a fundamental difference from creating a model for each value in a category?

texts:
I am creating a few models based on service requests. The services being requested are not distributed equally, some services being used sparingly, whereas others are quite common.
I had these services as categorical variables and built pipelines to incorporate them through one-hot encoding. I got to thinking that it may make more sense to train a model per service(at least for the common ones). Or does it make more sense to lump in the less common ones in a special category?
I am struggling with the regression model, coming in at 0.41 for my R2 value.


texts: Really Bad Predictions while using LSTM (tensorflow)

texts:
I am trying to fit a LSTM network on path-dependent input-output data (basically strains-stresses from elastoplastic behaviour). I took 14 different paths of input - output data and trained an LSTM Network , such that it gives sequence of outputs based on sequence of inputs.
I scaled the input and output data using MinMax Scaler like:
scaler_strains = MinMaxScaler(feature_range=(0,1))
scaler_stresses =MinMaxScaler(feature_range=(0,1))

strain_scaler= scaler_strains.fit(train_strains)
stress_scaler= scaler_stresses.fit(train_stresses)

train_strains_norm = strain_scaler.transform(train_strains)
train_stresses_norm =stress_scaler.transform(train_stresses)
validation_strains_norm = strain_scaler.transform(validation_strains)
validation_stresses_norm=stress_scaler.transform(validation_stresses)

I also reshaped as (number_of_batches,number_of_time_steps,number_of_features)
model = Sequential()
model.add(LSTM(32,input_shape=(training_strains.shape[1:]),return_sequences=True))
model.add(Dropout(0.2))
#Since number of output features is 6
model.add(Dense(6))
opt = tf.keras.optimizers.RMSprop()
model.compile(loss='mse',optimizer=opt,metrics=['mse'])

After training, even the prediction of validation set is very very wrong.
I tried:
1.Changing normalization limits since the input and output features contain negative values,but no use.
2.Didn't use any normalization, the losses were high and stagnant
3. Changed layers, (increased and decreased)  , but still no change.
I ran out of ideas...
Would be much obliged for any suggestions.


texts: What is a "knowledge-free" neural network model?

texts:
In the paper Representation Learning: A Review and New
Perspectives by Bengio et al., Bengio writes

The beginnings of deep learning in 2006 have focused on
the MNIST digit image classification problem (Hinton et al., 2006; Bengio et al., 2007), breaking the supremacy of SVMs (1.4% error) on this dataset3. The latest records are still held by deep networks: Ciresan et al. (2012) currently claims the title of state-of-the-art for the unconstrained version of the task (e.g., using a convolutional architecture), with 0.27% error,
and Rifai et al. (2011c) is state-of-the-art for the knowledge-free version of MNIST, with 0.81% error.

In this context, what does "knowledge-free" mean?


texts: Any special considerations on learning based on a cyclic timeseries using LSTM?

texts:
I have a time-serie over one year. For each 15 minutes in daylight hours, there is a data point. So basically, a sequence for each day. I would like to benefit this information if possible, I mean resemblance in day sequences. Logically, and without a special knowledge, using RNN blindly on the whole year sequence, might or might not learn from days hypes, although, In the best scenario, the network will learn it all.

In other words, How could I tell the LSTM network, for the next 62 data points, it is a sequence, that goes relatively up and down in the same way.

If it changes anything, I'm using Tensorflow and Keras, and still a n00b.


texts: Reversing the input and output of an ML algorithm to Optimize

texts:
My dataset consists of multiple input variables (X) and multiple output variables (Y).

For example:

| X1| X2 | X3 |    | Y1| Y2 | Y3 |
----------------------------------
| 1 |  1 |  0 |    | 2 |  2 |  0 |
| 2 |  2 |  1 |    | 3 |  3 |  1 |
| 3 |  3 |  3 |    | 4 |  5 |  6 |

But, I don't actually want to predict the output variables Y. I want to actually predict the input variables X given Y. I understand that switching the variables X and Y might not be 'optimal' since there could be multiple input values that yield the same output values, but I don't see how else I solve this type of problem. I was planning on using a Random Forest or simple neural net.

From the help I've been trying to get, it seems that I should keep input and output variables as is, but rather use optimization techniques to find the best input(the variable I want to predict) for a given output. I am unsure if my initial intuition on reversing the inputs and outputs would 'get the job done'.

For a more detailed breakdown on the actual problem I am trying to solve, please see this link.


texts: Time series regression forecast next month from now (random forest,Lasso,Ridge)

texts:
I have a dataset about hedge funds. Its include data from 2010 january to 2019 december. This data are monthly financial ratios of hedge funds such as sharpe,alpha,beta,sortino and monthly returns of hedge funds. I normalized the relative returns of each fund. And I want to estimate these monthly returns. Using these ratios. Now Im using machine learning regression models, I created this function "$$Y_{t+1} = X_{0,t}+...+ X_{12,t}$$" for the regression model. "$Y_{t+1}$" shows the normalized relative return of the hedge fund in the following month(t+1)(next/future), "$X_{0,t}$" variables show the financial ratios of that fund in the t month(now). I thought about this function to predict future month's returns before data came so I want to predict the next month with the current data.Because the data comes as of the end of the month. I have to guess the return for the next month. As an example, I would like to forecast the return on hegde funds at the end of January 2011 with data from January 2010 to December 2010. Without ever seeing the data for January 2011. Is this regression function written correctly.

Does using $Y_{t+1}$ break the model and is there a model or mathematical function that you can suggest? for "predict future month's returns before data came so I want to predict the next month with the current data."

Additonal Infos:

Some additional information there are 2889 months of data, I keep all 27 different funds in a database, and I train a single regression model, not for each fund separately.I used the regression function I showed above($Y_{t+1} = X_{0,t}+...+ X_{12,t}$).

But it didn't work well on any algorithm, so the R-square score was negative. why do you think it didn't work and how can I fix it ?
I used both sliding window and expanding window as validation. I use dummy variables for fund names/tickers

Sample Data:


texts: Are there any annotators or Named Entity Recognition for license plate numbers?

texts: Most vehicle license/number plate extractors I've found involve reading a plate from an image (OCR) but I'm interested in something that could tag instances of license plates in a body of text. Are there any such annotators out there?


texts: Explained definition of the norm in Ordinary Least Squares

texts: I have recently started learning Scikit-learn and I am not able to understand the below equation. Could anybody please explain?


texts: What are the execution service and the event service in Netflix big data

texts:
From the following article  by the netflix engineering team:

The core architecture of the big data platform at Netflix involves
three key services. These are the execution service (Genie), the
metadata service, and the event service

What do they mean by the execution service and the event service?


texts: Forecasting as a supervised regression task, with uneven length timeseries - how to split?

texts:
Consider a number of timeseries. Here we have 3, just to make it dead simple. Note that they're all different lengths.

The very first thing I do is splitting by the original sample dimension:

Then, to deal with the uneven lengths, I chop it into smaller windows:

Where each window is a slice of w/h, and some spacing s to control for the amount of overlap (bonus question: is overlap a no-no?).
Then I simply take all my windows, shuffle them and voila, a dataset for any machine learning algorithm:

Now, my question is simply: is this correct? scikit-learn prefers to do splits along the time-axis https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split, which to me is
a) a bit strange, considering that each split is a different shape (unless the idea is to train a new model for each w/h shape combination? If so, cross-validation is a slightly misleading name) and
b) seems to assume that all future samples are simply continuations of pre-existing timeseries.

Am I correct in my assumptions, both about my own and scikit-learn's way of working?


texts: Exception: Data must be 1-dimensional appears when trying to make a submission

texts:
I am doing image classification, and until now I have built my network and evaluated it. The only thing that remains to do is to do the submission, so I have a blind test set which contains images with no labels, and using the model I have created, I have to make predictions on this blind test set.

I have 4 classes: HAZE, RAINY, SUNNY, SNOWY. My code is the following:

blind_testSet = '/content/drive/My Drive/Colab Notebooks/submission'

test_datagen_blind = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255
   )

test_generator_blind = test_datagen.flow_from_directory(
    directory=blind_testSet,
    target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

preds =transfer_model.predict_generator(test_generator_blind,verbose=1,steps=val_steps)

import pandas as pd
import numpy as np

predicted_class_indices=np.argmax(preds,axis=1)
labels = (train_generator.class_indices)
labels = dict((v,k) for k,v in labels.items())
print(labels)
predictions = [labels[k] for k in predicted_class_indices]
filenames=test_generator_blind.filenames
results=pd.DataFrame({"image ID":filenames,
                  "Predictions":preds})
results.to_csv("submission.csv",index=False)

the problem is that I get the error:

    Exception                                 Traceback (most recent call last)
<ipython-input-22-37b4cd72c225> in <module>()
      9 filenames=test_generator_blind.filenames
     10 results=pd.DataFrame({"image ID":filenames,
---> 11                       "Predictions":preds})
     12 results.to_csv("submission.csv",index=False)

4 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/internals/construction.py in
sanitize_array(data, index, dtype, copy, raise_cast_failure)
    727     elif subarr.ndim > 1:
    728         if isinstance(data, np.ndarray):
--> 729             raise Exception("Data must be 1-dimensional")
    730         else:
    731             subarr = com.asarray_tuplesafe(data, dtype=dtype)

Exception: Data must be 1-dimensional

Can someone help me? Thanks.

[EDIT]I have also tried to to :

    preds =transfer_model.predict_generator(test_generator_blind,verbose=1,steps=val_steps).reshape(-1,1)

but it does not work and I get the same error.


texts: Why does OPTICS use the core-distance as a minimum for the reachability distance?

texts:
The OPTICS clustering algorithm defines

$$\text{core-dist}_{\varepsilon,MinPts}(p)=\begin{cases}\text{UNDEFINED} & \text{if } |N_\varepsilon(p)| < MinPts\\ MinPts\text{-th smallest distance to } N_\varepsilon(p) & \text{otherwise}\end{cases}$$

and

$$\text{reachability-dist}_{\varepsilon,MinPts}(o,p) = \begin{cases}\text{UNDEFINED} & \text{if } |N_\varepsilon(p)| < MinPts\\ \max(\text{core-dist}_{\varepsilon,MinPts}(p), \text{dist}(p,o)) & \text{otherwise}\end{cases}$$

why isn't it simply

$$\text{reachability-dist}_{\varepsilon,MinPts}(o,p) = \begin{cases}\text{UNDEFINED} & \text{if } |N_\varepsilon(p)| < MinPts\\ \text{dist}(p,o) & \text{otherwise}\end{cases}$$


texts: Machine learning algorithm which gives multiple outputs from single input

texts:
I need some help, i am working on a problem where i have the OCR of an image of an invoice and i want to extract certain data from it like invoice number, amount, date etc which is all present within the OCR. I tried with the classification model where i was individually passing each sentence from the OCR to the model and to predict it the invoice number or date or anything else, but this approach takes a lot of time and i don't think this is the right approach.

So, i was thinking whether there is an algorithm where i can have an input string and have outputs mapped from that string like, invoice number, date and amount are present within the string.

E.g:
Inp string: the invoice #1234 is due on 12 oct 2018 with amount of 287
Output: Invoice Number: #1234, Date: 12 oct 2018, Amount 287

So, my question is, is there an algorithm which i can train on several invoices and then make predictions?


texts: How to build regression model with restricted range?

texts:
I am working on a (NN) regression model for a scientific sensor.  With regard to the model, I am primarily interested in the positive outputs of the sensor.  For the negative outputs, I only care that the output is negative, not its magnitude.  The data associated with the negative outputs are noisier too. Thus far, I have approached the problem as a standard regression problem, and I have a few models that work pretty well.  I am interested in building a better model though.  Intuitively, I feel like this should be possible.  A standard model will fit the noisier, negative data at the expense of the cleaner, positive data.  My current best model is a NN regressor built with Keras.
Are there special approaches or methods for handling this type of problem?


texts: Is it possible to plot decision boundaries for only a subset of features?

texts:
I have a sklearn Random Forest classifier with 59 features as input. I'd like to plot the decision boundaries of only two features at indices i1 i2.

If I use the average/median values for the remaining features, the classifier ends up in a path that ignores the features i1/i2.

I'd like to plot the decision boundary of i1/i2 while ignoring the remaining features.
Is this possible? or useful at all?


texts: How to solve ValueError: Negative dimension size caused by subtracting 3 from 1 for '{{node model/Conv1/Conv2D}} = Conv2D... in mobilenet_v2

texts:
I'm trying to apply a retrained model of mobilenet_v2 presented in https://github.com/balajisrinivas/Face-Mask-Detection
The main part of my adapted code is the following:
faces = []
locs = []
preds = []

for coord in coords:

    x = person[0]
    y = person[1]
    w = coord [2]
    h = person[3]

    # extract the face ROI, convert it from BGR to RGB channel
    # ordering, resize it to 224x224, and preprocess it
    face = frame[y:y+h, x:x+h]
    face = cv2.cvtColor(face, cv2.COLOR_BGR2RGB)

    if (face.shape) > (224, 224):
        face = cv.resize(face, (224, 224))

    face = img_to_array(face)
    face = preprocess_input(face) #from tensorflow.keras.applications.mobilenet_v2

    # add the face and bounding boxes to their respective lists
    faces.append(face)
    locs.append((x, y, w, h))

    # only make a predictions if at least one face was detected
    if len(faces) > 0:
        # for faster inference we'll make batch predictions on *all*
        # faces at the same time rather than one-by-one predictions
        faces = np.array(faces, dtype="float32")
        preds = maskNet.predict(faces, batch_size=32) #maskNet = load_model("mask_detector.model")

The error is raised when I try to apply maskNet.predict, which is the is the mobilenet_v2 model saved in https://github.com/balajisrinivas/Face-Mask-Detection/blob/master/mask_detector.model.
The error message is the following: ValueError: Negative dimension size caused by subtracting 3 from 1 for '{{node model/Conv1/Conv2D}} = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], explicit_paddings=[], padding="VALID", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true](model/Conv1_pad/Pad, model/Conv1/Conv2D/ReadVariableOp)' with input shapes: [?,1,1,3], [3,3,3,32].


texts: Is there an algorithm that imputes missing values based on n nearest columns? (KNN hybrid)

texts:
I have a dataset of 70 columns that have missing values. Each column has a few columns (3-5) that it is significantly more correlated than the others but each column's correlated columns are very different from other columns. I would like to perform a tweaked version of KNN imputation and before I start writing this from scratch, I'd like to know if there's something similar out there, So, I don't go about reinventing the wheel.
On initialization, a correlation matrix is obtained between all the variables. For every missing cell that needs to be imputed, the algorithm retrieves from the correlation matrix the top n highest correlated columns and uses only those N columns to select the k nearest neighbors and use the mean value of those neighbors.
Thanks and apologies if I am not very clear with my question. I'll try to clarify if you have any doubts.


texts: Is there a way to drop duplicated rows based on an unhashable column?

texts:
i have a pandas dataframe df with one column z filled with set values

i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values ( which are sets ).

import pandas as pd

lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]
lbls = [ 'x' , 'y' , 'z' ]
df = pd.DataFrame.from_records( lnks , columns = lbls )

Trying to drop duplicated rows based on column z values :

df.drop_duplicates( subset = 'z' , keep='first')

And i get the error message :

TypeError: unhashable type: 'set'

Is there a way to drop duplicated rows based on a unhashable typed column ?


texts:
Suppose I would like to train and test the MNIST dataset in Keras.

The required data can be loaded as follows:

from keras.datasets import mnist

digits_data = mnist.load_data()

Is there any way in keras to split this data into three sets namely: training_data, test_data, and cross_validation_data?


texts:
I am just playing with bert (Bidirectional Encoder Representation from Transformer)
Research Paper

Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non Linear (NL) layers on top of BERT model. How can I do this?

I am not able to figure out where should I change in code of BERT. I am using the pytorch implementation of bert
from huggingface.

This is what I want to do:

Please show steps to implement this using sudo code which will help me in implemention of cnn on top of BERT.


texts: Time series prediction with Lstm on patients data

texts:
My data includes different time series length (depends on number of exams each patient did) as well the interval between exams is different.
How can I run lstm on this kind of data? (Where the sequence length is different)
How should I build the batch?
Should each batch hold a patient time series data?
Where do I put the target value?


texts: line search in gradient descent applied to a convex function

texts:
I have been working on implementing a line search method for gradient descent where I made an assumption that at any given point on my surface of the loss function I can reach the minima by the single correct value of the learning rate $\eta$ which I should choose. I have been trying to find this learning rate using binary search but after the entire implementation, I came to realize that my assumption I made is wrong which means I cant directly reach my minima from any given point on the surface of the loss function for any given learning rate in a single step. Can I get a more intuitive explanation of why my initial assumption is wrong?

Edit:  my loss function is convex and has a large number of parameters I am trying to learn ( multidimensional)


texts: Prepare JSON data from sentiment analysis to perform Logistic Regression

texts:
I'm new to this field, so very sorry for this basic question.
I'm working on a text analysis project using Google's NLP API along with some other APIS. After performing the sentiment analysis I have its results in JSON format and now I need to apply the Logistic regression, I have no idea how can I prepare my JSON data to perform Logistic regress.

Here's a sample of my data:

{
   'documentSentiment':{
      'polarity':-0.3,
      'magnitude':0,
      'score':0
   },
   'language':'en',
   'sentences':[
      {
         'text':{
            'content':'{\\rtf1\\ansi\\ansicpg1252\\cocoartf1639\\cocoasubrtf100\n{\\fonttbl\\f0\\fswiss\\fcharset0 Helvetica;}\n{\\colortbl;\\red255\\green255\\blue255;}\n{\\*\\expandedcolortbl;;}\n\\paperw11900\\paperh16840\\margl1440\\margr1440\\vieww10800\\viewh8400\\viewkind0\n\\pard\\tx566\\tx1133\\tx1700\\tx2267\\tx2834\\tx3401\\tx3968\\tx4535\\tx5102\\tx5669\\tx6236\\tx6803\\pardirnatural\\partightenfactor0',
            'beginOffset':-1
         },
         'sentiment':{
            'polarity':-1,
            'magnitude':0,
            'score':0
         }
      },
      {
         'text':{
            'content':'\\f0\\fs24 \\cf0 This is the first text file.}',
            'beginOffset':-1
         },
         'sentiment':{
            'polarity':1,
            'magnitude':0,
            'score':0
         }
      }
   ]
}

How can I prepare this data to perform Logistic Regression?
I will use Pandas, Numpy and Sckikit-learn.

Help me, please!

Thanks in advance!


texts:
I am working on the development of an ontology for my dissertation project. I have read plenty of resources and tutorials on how to develop ontologies, why are they useful and how to use them.

I've been trying to find examples of ontologies that include user interface such as textboxes, buttons etc. to study them for the development of my ontology. I checked databases such as ontoligua for similar projects but I didn't find anything to relate to.

I would be grateful anyone can help me with that.


texts: Load training data sequentially to avoid memory error

texts:
How can I load data of shape:
X_train : (4864,6989)
X_test : (2085, 6989)
y_train : (4864, 270)
y_test : (2005, 270)

into the memory? I have 16 GB of RAM.
What I have tried:

Loading data in chunks
extracted_df = pd.read_csv("ExtractedData.csv", chunksize=6953)

Using virtual memory (paging) by going in the system settings

Reducing the chunk size.

I am able to fit the data on a relatively simple model. However for the model I have defined, I can't seem to get out of the memory error.
def build_model(embed_size, max_length, vocab_size):
    def build_model():
        model = Sequential()
        model.add(Embedding(vocab_size, embed_size, input_length=max_length))
        model.add(Conv1D(embed_size, 7, activation='tanh', padding='same'))
        model.add(MaxPooling1D(2))
        model.add(Conv1D(embed_size, 7, activation='tanh', padding='same'))
        model.add(MaxPooling1D(2))
        model.add(GlobalMaxPool1D())
        model.add(Dropout(0.2))
        model.add(Dense(150, activation="tanh"))
        model.add(Dropout(0.2))
        model.add(Dense(150, activation="tanh"))
        model.add(Dropout(0.2))
        model.add(Dense(output_nodes, activation="softmax"))
        model.compile(loss='categorical_crossentropy',
                      optimizer='adam',
                      metrics=['accuracy'])
        plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)
        return model

    return build_model

feature_nodes = X_train.shape[1]
estimator = KerasClassifier(build_fn=build_model(feature_nodes, feature_nodes, 1000), epochs=3, batch_size=16,
                                verbose=1, callbacks=[EarlyStopping(patience=3, monitor='accuracy')],
                                validation_data=(X_test, y_test))

I don't understand dask enough to try it but as I understand, it is for really huge data. My data has 98k rows and 2 columns. Any suggestions? Thanks.


texts: Layer-wise Relevance Propagation with the swish activations

texts:
I'm studying approaches to explain DNN. I've found a lot of papers on LRP (and Deep Taylor decomposition), but they all explain NNs with the ReLU activations. I'm wondering why nobody applied LRP to NNs with the swish activations? Is it possible? Should I compute the derivatives of the swish function to compute the coefficients in front of the relevance comping from the upper layers?
Thank you in advance.


texts: Binary classification: best ways to pre-procees the data

texts:
About the dataset
I have a training dataset of

129 columns(last column being the classes, i.e., y values)
6068 rows

I have to train some algo to do binary classification. The data set has

701 examples of type A
5367 examples of type B

The test set consists of 1398 examples. Here is the accuracy I got for various algorithms.

voting ensemble -> 0.73963
stochastic gradient boosting -> 0.77682
Adaboost -> 0.75107
bagging classifier(Decision tree) -> 0.76538
Random Forests -> 0.75250
Extra trees -> 0.75393

All the above results are from kaggle so they probably are just from half the test set.
The above methods were implemented using scikit-learn library in python

Could someone please suggest ways to improve accuracy, may be by things like dimensionality reduction or better algorithms. Also, please provide sample code in possible.


texts: How do I represent SURF Features into Bag of Words to determine Nearest Neighbors?

texts:
I'm trying to use Speeded Up Robust Features (SURF) to get the $k$ most similar images from a set of images in my directory. I'm planning to use $k$-Nearest Neighbours ($k$-NN) for this. As far as I know, the size of SURF descriptors are $n \times 64$ or $n \times 128$ depending on how many descriptors I want. There are suggestions and I've read about the Bag of Visual Words method, where these patches are converted to bags of words, similar to the common Natural Language Processing technique. I've also read that Bag of Visual Words are generated by clustering, whereas the feature patches are clustered together.

What I don't understand is, how can I use these Bag of Visual Words to train my $k$-NN such that I can get similar images? I really can't grasp how those clusters can generate BoWs. Say I have 1000 images, if I convert them, what will they look like? Will they be 1000 BoWs that still represent the same images?


texts: Siamese networks vs Semantic similarity (may be gensim)

texts:
I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural network(s). The idea was applied mostly to images in the tutorials provided on internet.

If I compare it with Gensim semantic similarity, there also we have vectors of two objects (words or sentences) and then do a cosine similarity to calculate the difference. (remember example of King-man+woman=Queen).

Am I missing some aspects of Siamese networks or these are actually same?


texts: How to train the machine so that it can give 'out of bound/classes' as an output for neural network

texts:
I know I was not able to word the title of the question properly. So I am trying to explain the problem here: Suppose, I built and trained a CNN to identify numbers from 0 to 9. However, when I deployed the CNN, someone gave "#" as an input (anything but 0-9). What can I do to my neural network during training, so that it's output can say that this is none of the training characters?

I'd like to present a second example: instead of classifying, suppose we want to do denoising using an autoencoder. Again, the CNN autoencoder is trained for 0 to 9. Now, how can we prepare it so that if someone gives "$"symbol (it can be anything that is not 0-9) as an input, it'll be able to identify that this symbol is none of what it was trained on? And the CNN autoencoder will be able to give an output according to that?


texts:
This piece of code should be self-explanatory and it's also a choke-point in my process. Any way to do this better?

geoset['MonthInt'] = pd.to_datetime(geoset['Date received']).dt.month # Add a month
geoset['YearInt'] = pd.to_datetime(geoset['Date received']).dt.year # Add a year
geoset['DayInt'] = pd.to_datetime(geoset['Date received']).dt.day # Add a day of month
geoset['DayOfWeekInt'] = pd.to_datetime(geoset['Date received']).dt.dayofweek # Add a day of week


texts: How to run a saved tensorflow model in the browser?

texts:
After doing my hello world models, I would like to let them available at Github pages, which means that I need to serve the model only with static files. Is it possible?
All the tutorials I found requires nodejs or some backend


texts: How the vector of weights is assigned to a Neural Network

texts:
Say for example, you have an artificial neural network like the one given below-

You have set of input nodes, hidden nodes and output nodes although I have one output node in my case.

We assign weights to the connections between those nodes. So is there any pattern of assigning weights to those connections i.e. See the first connection between first input(top) node and first hidden(top) node. So is it that the connection is assigned first value from the vector of weights,

the second the connection between first(top) input and second(second from top) hidden node is assigned second value from the vector of weights .

In short, what am I asking is that - Is there any correspondence between the position of the weight in the vector of weights and the connections in the structure. Is there any matrix that is assigned?


texts: Help with approaching this problem correctly

texts:
newbie here trying to figure this out:

I have a dataset which looks like:

facility_id   area_zipcode  staff_count   population
FacilityA       98007           21          24889
FacilityB       98290           52          32714
FacilityC       98065           43          12699
FacilityD       98801           9           40977
FacilityE       98104           64          13095

Here is my problem statement:

There are five healthcare facilities in five different areas, each with their respective staff count. Population in a certain area can only go to the facility which is nearest to them. So for people living in area A, facility A is the closest, but sometimes the staff count may not be enough to handle everybody.

We're assuming at least half of the population will require healthcare, and that one staff member can only tend to about 2000 patients per year. How would we optimize the staff's distribution in all these facilities such that all five facilities can tend to the majority of their populations, thereby minimizing everyone's travel times. Basically allocate staff count to where they are most needed, increasing or decreasing capacity in facilities.

My speculation so far:

This is certainly not a classification problem, I'm thinking it's a regression problem since we're interested in some real value outputs i.e optimal staff count for each facility. Because we have multiple outputs, I'm leaning towards multi-output regression. I'm also supposed to find travel times using bing maps API, which I have successfully figured out how to do, but I'm confused with what to do with all these travel times. (like a to b, a to c, a to d etc) or how to add them to my dataset. There seem to be two priorities in the problem statement:

1) Optimize staff counts

2) Minimize travel times

And currently I'm a bit confused as to how I should tackle this, I was hoping someone could point me in the right direction, with how to model this problem and how to achieve both those goals. I'm working in python, and I can make any assumptions to simplify the problem. Any help is appreciated!

Also if you know of research papers, or others who have worked on similar problems, please guide me on how to look for them, what type of problem is this etc?


texts: I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know that I can do text similarity approaches like Jacard or Levenhenstein but that would be too expensive for the job that I want to do because I will need to compare email from user i against all other users emails and do this same thing for all user ( i try already this and is too expensive ). I thought maybe i was able to embed the emails somehow ( this is my question - which way would be the best way to do that ? ). My final goal would say that vector from the embed of mereo923@gmail.com would be very similar to mereo329@gmail.com. There is any approach which could help me with the creation of this embedding?


texts: Create labels or annotation from two color images

texts: I have two categories of images to use in my deep learning model. The firs category is aerial images that contain roads (such as road-1.jpg). The second category is two color images that contains road and non-road areas (such as annotation-1.jpg ). I want use second category for labels. So how can I create labels or annotation from second category images.I know there are some image cutting tools. But I'm looking for a tool that does this automatically by using the color information in the images. For example, I want to automatically create label map for the red areas in annotation-1.jpg.   In addition, I am going to use the Python programming language and the Tensorflow library.


texts: One Challenging Neural Network Question and MATLAB Simulation

texts:
I see this example ON NN notes:

how this solution is achieved? is there any clear trick or usual step for solving this question quickly?I think the solution in the question be wrong. no idea anyway... please be aware that this is PhD entrance exam question and we should think the information is not enough in the question.
some idea tell us that 5, 5 is answer because for NN there is no difference on condition (1) and (2) in the question.
some idea tell us there is 3, 3 solution, but with complex neurons (this is not should used here in general, because this is general question even with lack of information)
some idea tell us there is at least 5, 5 solution because we cannot model with 3 nodes here... (not for first reason)
here one implementation of these structure in MATLAB for each case with 3 or 5 nodes and I think with $5$ we get the better result.

$A) 4, 5$
$B)  5, 3$
$C)  5, 4$
$D)  5, 5$


texts: np.shape[::-1] reversal of dims and np.moveaxis splitting array

texts:
I cannot find anything in the documentation but it was used in some starter code for a class I am taking at school.
Upon testing per below, it seems to reverse the order of the dimensions of an numpy array.

pic = np.ones((3,4,5))
print(pic.shape,"\n", pic)
#new_shape is the reverse of pic.shape
new_shape = pic.shape[::-1]
print(new_shape)

This outputs:

(3, 4, 5)
 [[[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]

 [[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]

 [[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]]
(5, 4, 3)

In the starter code it was used in this way:

res_img = cv2.resize(img, sub_img.shape[::-1])

There are some other tricks that are planted in the starter code (to show us that they are there I'm guessing).  Here is another example:

channels = blue, green, red = np.moveaxis(color_img, 2, 0) #move channels to pos 0

The np.moveaxis() moved channels to the first dim position but it's assigned to blue, green and red at the same time via Python multiple assignment?  When I looked up multiple assignment, I only saw stuff like var1, var2 = 1, 2.  Also np.moveaxis() returns an array with the new shape so I have to stretch my imagination to assume the first axis can be assigned to variables.  Is this correct?

My instructor just replied that np.split() does the splitting but the documentation does not say anything about this being implicitly called when np.moveaxis() is called.

Does anyone know what is really happening?


texts: Find best machine learning for predicting category of products

texts:
I have a dataframe that contains product and in this dataframe I have some features like: brand, cat1, cat2, cat3, city, desc, image_count, mileage, price, title, year.
The goal is predicting category of products. I have 1 billion training data and important features for prediction are title and description that are text type.
I like to know what algorithm is best for my prediction? I'm a beginner in machine learning and confused among different algorithms. Thanks


texts:
Im presented with a unique text classification problem.

Im given a list of descriptions each containing 3-8 words. I know that there are some descriptions that are nearly the same, but the majority of them are significantly different from each other. My objective is to group the descriptions that are roughly the same and to consider the rest of the descriptions as unique. This data set can be considered too long to manually label for supervised learning.

My thought process so far:

TF-IDF will come in handy as the frequency of words that match is extremely low (therefore any match is highly valuable).
Unsupervised clustering techniques like K Means might be useful, but there is an inherit, "K" selection issue with K Means in this space. For example, if we have 1,000 descriptions and only 10 descriptions (5 clusters of 2 descriptions) are supposed to be clustered with one another (leaving 9,990 unique descriptions), we would need 9,995 clusters to represent the structure accurately (9,990 clusters with 1 item and 5 clusters with 2 items each). This issue would lead to an extremely difficult model building phase as the number of clusters would be so high (perhaps this isn't an issue and this theory should be tested).

I am very new to the space of unsupervised learning being used with NLP. I have a few more thoughts that I could elaborate on, but I really need some advice on how others deal with clustering text in this way. I dont need to use K Means or any other specific idea if better alternatives fit the problem space better.

Edit:

I am starting to think that DBSCAN will be a much better clustering model. This documentation explains that each point can be clustered and outliers are not taken into consideration more so than real groups (under specific configuration). In more technical terms, I believe setting a low eps score with a minpts values of 2 will be a good theory to test. This essentially assumes that most data belongs to its own cluster (most of the data is unique), but some data (under a weak minimum points configuration) should indeed be grouped into a cluster. DBSCAN avoids the pitfalls of K Means by dealing with outliers in an algorithmic way. K Means would be heavily dependent on a genius guess of a "K" value.


texts: Why is the variance of my model predictions much smaller than the training data?

texts:
I trained a GRU model on some data and then created a bunch of predictions on a test set.
The predictions are really bad, as indicated by a near zero R2 score.
I notice that the variance of the model predictions are much smaller than the actual training data. i.e it seems like the model is overfit to the mean:

But why is this? I made sure to stop training/use hyperparameters where there was no model overfitting, so why are model predictions centred around the mean and less dispersed than the actual variance of the data set?
My model, if it is relevant:
model = Sequential()
model.add(GRU(100, activation='relu', input_shape=(3, 280), recurrent_dropout = 0.2, dropout = 0.2))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse', metrics=['mae'])
model.summary()

# fit model
history = model.fit(X_train, y_train, epochs=40, verbose=1, validation_split=0.33)


texts: Algorithms, techniques, papers for regression with vector output

texts:
I have a regression problem that has relatively low dimensional input (say 8 initial relevant features, without the engineered ones), but very high dimensional output vector (not a single value for the output, a vector of reals). Basically for each sample during training, a low dimension of features and 2500 reals as the output go into the model, then testing samples take the same types of input features in to predict the 2500 reals (a curve).

              inputs                               outputs
 _________________________________________ _______________________________________
| a | b | c | ... | (engineered features) | 2500 reals as output (y's of a curve) |

Since we are not machine learning experts, we had some trouble in the beginning even finding an algorithm that supported output vectors as opposed to a single point.

I'm not looking for generic data science 101 suggestions like: cleaning the data, picking the relevant features, understanding the data, etc. We've done that well enough I believe, and we've even had some limited success already learning these curves with about a predication rate of ~70% (which sounds rough, but isn't too bad given the nature of this data). However, we are not experts as I said, and I'm still convinced that we are not even considering all of our options in terms of algorithms, frameworks, feature engineering techniques, etc. for this specific type of problem; I'd like to squeeze everything we can to get the best possible predictions. Given this data, there definitely isn't a 100% relationship between the input and output. But I do think we can get it predicting at a rate of higher than 70%. Maybe. Sometimes the predictions are near perfect, other times they have the "general shape" but are sometimes warped in places we can't explain.

We are using Python/SciPy/scikit-learn. We are doing 20-fold cross validation and our dataset has ~22000 samples.

How would you approach this type of problem - which algorithm, or which papers, have you come across / used / are relevant for this type of problem? What should we be keeping in mind for this specific type of problem?

We are very happy with this:

Not bad:

Not good:

Interesting:


texts: How to generate training instances from multivariate normal distribution?

texts: I would like to generate a training and test set for the binary classification problem. I want the instances to be generated from a multivariate normal distribution and have 5 features. Can someone explain how to do this in python?


texts:
I have more then 100 models (.h5) to apply on a device that couldn’t have any python package larger then 150 mega byte. I have tried multiple 2.x versions, yet all was 400+ mega byte. Tensorflow Lite wont work here.
Which version could help here or any other idea?


texts: Improve classification accuracy on Iris data set - tf.contrib.learn Quickstart tutorial

texts:
I run through the Quickstart tutorial of the tf.contrib.learn, which uses the Iris data set. After training the DNN model I get an accuracy of ~0.967.

In an attempt to improve the accuracy, I tried to change the number and size of the hidden layers, I doubled the steps, and I changed optimizer and learning rates. However, the accuracy does not increase (or decrease).

Could that might be a dataset limitation?

Any ideas on how to improve the performance? I want to be able to dive into a NN and try to optimize it, before I go blindly into another tutorial.


texts: Does the score output of a classification model has a global meaning?

texts:
The scores-output layer contains the class scores that the model generated for the current sample and it is passed thru the softmax layer to get the final output in the form of a probabilities vector.

My question is whether these scores have any global meaning, when compared with scores of other samples from the dataset?

  The way I see it, a standard classification model is trained on the
  output of the softmax layer, so the model only "cares" about the
  relative scores within each sample. So it has no mechanism for
  decreasing the global scores of a sample - a scores vector of
  [101,102,103] gets the same treatment as [1,2,3]. Am I correct in my
  understanding?


texts: Differerent ways to detect the appropriate number of topics

texts:
I implemet the LDA topic modeling in R.

One essential parameter is the selection of the number of topics

Which of the following ways could it be the most suitable:

1. mallet
2. stm
3. or this way https://cran.r-project.org/web/packages/ldatuning/vignettes/topics.html


texts: What is the correct way to compute Mean F1 score?

texts:
I have a set of 10 experiments that compute precision, recall and f1-score for each experiment.
Now, average precision & average recall is easy to compute. I have some confusion regarding average f1-score.

There are 2 ways on how i can compute mean f1-score:

Take f1 scores for each of the 10 experiments and compute their average.
Take average precision & average recall and then compute f1-score using the formula f1 = 2*p*r/(p+r)

I could not find any strong reference to support any of the arguments. The closest document i could find is this: https://www.kaggle.com/wiki/MeanFScore

Can anyone explain with some reference (if possible) which of the methods is correct and why?

EDIT: One of the members suggested this source. Though, i still suspect the reliability of the source. I have seen people not using the method explained above in their research publications. (Even i would be using it in one of my publications) I would expect some more opinions from the community to verify this idea.


texts: Vgg16 model validation accuracy is stuck

texts:
I am working on a CNN model for MRI brain images classification (Alzheimer disease), I use transfer learning method for image classification - vgg16 model trained on ImageNet (1000 classes). I’ve modified last layer so the size of each output sample is set to 2.
I am using SGD optimizer, weighted CrossEntropyLoss as loss function because dataset is not balanced and learning rate scheduler - StepLR.
I encountered a problem with validation accuracy that stays (almost) the same or is decreasing (approximately 70% whereas state of the art in this topic is around 95-99%), I thought that the problem is overfitting but then I’ve tried to use L2 regularization, data augmentation and dropout but nothing helped. MRI images are normalized.
Here is a code of my training function

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()
            else:
                model.eval()

            running_loss = 0.0
            running_corrects = 0

            for inputs, masks, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs.float())
                    _, preds = torch.max(outputs, 1)

                    loss = criterion(outputs, labels)

                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    model.load_state_dict(best_model_wts)
    return model

I am training the model for 100 epochs.
Learning rate is 0.001
I would appreciate any help. Thanks in advance


texts: Is there a Spark library for computing canonical correlation?

texts:
I can't seem to find a library for computing canonical correlation in Spark. Canonical correlation analysis (CCA) is common in other data science-focused systems.

This exists in R's base and in Python in Scikit. Julia's Multivariate.jl has it but is quite broken.


texts: Difference between Sum of Squares and Maximum Likelihood Linear Regression

texts:
I'm new in Machine Learning and one of the first arguments I'm studying is linear regression. I understood that , in few words , the idea to use the Linear Regression is to learn an hypothesis that can map a new input x in a good approximation of y.

In order to do this , if my hypothesis is :

h(x) = wx + w0

I have to update my parameters minimizing an error function like Least Squares and optimize the w vector with the help of an optimization algorithm like Gradient Descent.

I understood how this works , but sometimes I see this "Maximum Likelihood Estimation" and I did not understand if it is another way to estimate the w parameters or something else.


texts: machine learning algorithms for 2d data?

texts:
I'm looking for a supervised learning algorithm that can take 2d data for input and output. As an example of something similar to my data, consider a black image with some sparse white dots. Blur that image using a full range of grayscale. Then create a machine that can take the blurred image as input and produce the original sharp image as output.

I could make some sample 1D data by taking a region/radius around the original sharp point, but I don't know the exact radius. It would be significant data duplication and a lot of guessing.

Any good algorithm suggestions for this problem? Thanks for your time.


texts: Alternatives with better GPU than Google Colab Pro

texts:
I am currently running/training MAchine learning models that are very GPU expensive, Google Colab Pro is not giving me enough GPU/RAM
Is there any alternatives with better GPU  and more RAM than Google Colab Pro??


texts: Clustering nodes based on structure of local neighbourhood in a graph

texts: I would like to cluster nodes in a graph based on the structure of their local neighbourhood, i.e. nodes with similar graph structure around each other should be in the same cluster. My issue is basically to define a good feature set with a metric that would result in reasonable clusters. Does anyone know of any such features? :)


texts: Training a LSTM on a time serie containing multiple inputs for each timestep

texts:
I am trying to train a LSTM in order to use it for forecasting. The problem is basically a multivariate multi-step time series problem.
It is simply an experiment to see how statistical models (ARIMA, Holts-Winters, ...) and neural networks compare for a given problem.
As my dataset is perfectly fit for a statistical model, I am having trouble when trying to format it to train the LSTM as I have multiple entries for one timestep (corresponding to different entities) and I don't really know how to deal with it since the sequence is no longer tied by the time of observation. Let's say my dataset looks like the following example :
time | ent | obs

  1 ---   1 ------      5

  2 ---   1 ------     6

  2 ---   5 ------     1

  3 ---   2 ------     7

  3 ---   5 ------     4

As you can see, not every entity have an entry for any given time, and one timestep can have multiple entries.
I thought of training the LSTM for each entity but I would have too few data for most of them. Some threads gave me the idea to separate each entity into batches but the number of observations is not constant so it wouldn't work for me.
How do you think I am supposed to tackle this problem?


texts: Can GPS coordinates (latitude and longitude) be used as features in a linear model?

texts:
I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and end points; and (2) estimating the amount of crime for a specific point.

I'd like to use a linear regression model. However, can I use these GPS coordinates directly in a linear model?

Latitude and longitude do not have an ordinal property, such as with a person's age. For example, the two points (40.805996, -96.681473) and (41.226682, -95.986587) do not seem to have any meaningful ordering. They are just points in space. I was thinking of replacing them with categorical US zip codes and then doing one-hot encoding, but that would result in a lot of variables.


texts: Does the add_regressor method on Facebook Prophet also work with categorical variables?

texts:
I went through the documentation of Facebook Prophet and was able to build a similar model for my time series dataset. The additional regressors I used were numeric. I achieved a reasonable MAPE score. However, I also want to analyse this time series on a zip code level, which adds a categorical variable to the mix. Would one-hot encoding do the trick? I'm doubtful about it's efficiency as I would need to individually add each category (zip code) as a separate additional regressor.

My dataset is in the following format:
Zip code | Date | Numeric Attr 1 | Sales (predictor)

I am also open to switching to a different model, should there be a good one which does both time series and regression. Any pointers would be appreciated.


texts: Deep learning - aesthetics data modelling

texts:
I want to train neural network on aesthetics. I am getting confused on how to go about for training data.

Assume, I have large data set of landscapes, portraits, wildlife etc which are aesthetic according to humans. But, I want to train them for the quality, the kind of colours involved, contrast levels, background blur etc.

How do I train images for this criteria? Is there a way for doing this by unsupervised learning?


texts: Outlier Detection using K-Means using one column

texts:
I have done and read a csv file and then plotted the values of a single column using K-means

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
style.use("ggplot")
from sklearn.cluster import KMeans

data=pd.read_csv(r'Plot_file.csv', encoding='unicode_escape', sep=';')
data.head()

feature_names = ['Plot_Column]
X = np.asarray(data[feature_names])

from sklearn.cluster import KMeans

labels = KMeans(5, random_state=10).fit_predict(X)
plt.scatter(X[:, 0], X[:, 0], c=labels,
    s=50, cmap='rainbow');

The output looks like this, it is linear because when clustering one column it can only look at the relative distance between the values in that column and will always be linear on any chart as it is only clustering one dimension

How would I go with detecting anomalies in this case?

The column that I am clustering the values from has around 12 thousand rows and varying numbers.


texts: Wanting to overfit model to say if data has been seen before

texts:
I would like to train a scalable model that has as an input row of a database and has an output of either 1 or 0 depending upon whether it has seen this entry of the database before or not. The purpose of this is to then be able to check against a similar database and determine which entries are the same in the first database, without ever having to share the database. I can't hash the entries and hash-match or any other techniques like that.

That is, let's say two organizations are both using a person database with just floats and with columns = ["name" : float, "age": float, "net_worth": float]. Now, imagine that I am the first group, and I train a model that will overfit on my data, 'memorizing' it in some way. Then, what I want to be able to do is send that model to the other organization, who could apply it to each element of their table. This will then tell the other organization which entries of their dataset are also shared by me.

I understand that this is a unique (if not downright weird) way to try and solve this problem, but this is the approach that I'm trying to take. Does anyone have any solutions they can think of?


texts:
I am looking to change careers and would appreciate some advice.

I have an undergraduate degree in English Literature and a JD. Needless to say, these were not the best decisions and I would like to change my career. I have always enjoyed math and science, and after months of research and self study, I have decided that I would like to pursue statistics.

My question is basically this : would it be better to get another Bach. Degree in statistics or should I take calculus 1-3, linear algebra, probably and statistics and some computer science courses at a community college then try to get into a grad program instead?

I am afraid that without the bachelors degree I will not have the required knowledge for a masters even after taking the courses at my local c.c. However, another bachelors degree may be a worthless waste of time.

Please share your thoughts. Also, sorry if this is not the right place for this question.


texts: how to handle outliers for clustering algorithms?

texts: I am wondering what's the best way to handle outliers when using non-supervised clustering algorithms?


texts: How to setup and run Conda on Google Colab

texts:
I am interested in using Google Colab for data modeling. How do I install conda, create an environment and run python in a notebook? I did some searching and found some helpful hints, but had several issues with this. I can only get a partially functional environment so far.
I get stuck in running another cell in the same environment. Seems that switching cells resets the environment back to default.


texts: Feature Importance - How to choose the number of best features?

texts:
What is the standard or what method do you use to select part of features.

For ex: Using random forest, I got the following feature importances:

a : 25.4884726
b : 17.2736393
c : 12.3493490
d : 8.9383737
e : 8.1083837
f : 6.8272717
g : 4.1999999
...
...
...

For final prediction, you need to select only few features from the above. How do you decide the following:

  No. of features that you choose?
  Does the score matter when you choose?


texts: Should we normalize test data by choosing maximum and minimum value of training data?

texts: I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that while performing min-max normalization we pick minimum and maximum value in each sample's features. For the test data should I choose minimum and maximum of each sample Or should I normalize it with  the minimum and maximum value of training data?


texts: Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

texts:
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.
Do I have this right?
It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.


texts: Training the Discriminative Model in Generative Adversarial Neural Network

texts:
What I know so far in DCGAN is that a discriminator is trained using the labeled data (so maybe that occurs before training the generative model). Also, I know that there is race between the generator and the discriminator, so maybe training occur online. So I have some concerns here:

How many outputs the discriminator should have (Is it one output that describes the probability, ex: P(x))?
How do we chose its output when feeding fake data vs. real data?
Is the discriminator trained before using it with the DCGAN or the training is done online (It is mentioned in the Original Paper: Generative Adversarial Nets https://arxiv.org/pdf/1406.2661.pdf, that the whole network is trained using the back propagation), hence I think its online?

Any help is much appreciated!!


texts: When to use different Word2Vec training approaches?

texts:
So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, Word2Vec in Tensorflow or Word2Vec trained with Gensim ?

In what cases would implementing it through the more manual first approach be useful vs. the second one? If there is already an easier way to train a word2vec model using gensim, why is that not used always?

Furthermore, what is the benefit in using a pre-trained model like the Google News dataset? What happens when there are words that are not included in the news dataset?

Sorry if this question is basic, I just want to get a clearer grasp of the overall picture.


texts: Architectures that take inputs of mixed sampling rates

texts:
Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time series for classification.
My question is, in this scenario, are there any tricks I can do to the data or the model itself to make it "aware" of the sampling frequency? I've tried naively just training them together, and I'm getting an ok result.  Are there any existing works? I tried googling but found nothing worth noting as of now.
I'm hesitant about just resampling the data is I don't want to accidentally remove/introduce important information by making the method of interpolation a hyperparameter.


texts: Why there is no exact picture of softmax activation function?

texts:
I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a sigmoid function?
I just found the following funny pictures from dataaspirant which is not available anymore:

Can anyone illustrate a Softmax function graph? and explain why one is faster than another one due to the above picture?


texts:
I am trying to perform Hindi To English translation using s2s model, following - https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/

I have used https://github.com/karimkhanp/Seq2Seq/tree/master/data dataset which contains 37726 sentences for training.

Training took around 44 hours on my 8GB machine. I considered 3 layers, 10 epoch, 20000 vocap size

Other parameters

ap.add_argument('-max_len', type=int, default=200)
ap.add_argument('-vocab_size', type=int, default=20000)
ap.add_argument('-batch_size', type=int, default=100)
ap.add_argument('-layer_num', type=int, default=3)
ap.add_argument('-hidden_dim', type=int, default=1000)
ap.add_argument('-nb_epoch', type=int, default=10)
ap.add_argument('-mode', default='train')

But now when I test using hindi sentence, it give UNK for all words. Though I take same sentence for testing as in training even it says UNK in results.

Test sentences:

डेली हिन्दी न्यूज - बुंदेलखंड का प्रथम अन्तरजालीय स्थल
मैं भारत से प्यार करता हूँ
ज़बूर जो कि दाउद को प्रदान की गयी
प्रशासनिक विभाजन
वे इस देश के प्रथम UNK -LRB- अफ्रीकी UNK -RRB-
नेपाली विदेश
वेल्श खिलाड़ी इंग्लैंड के लिए खेलने के लिए पात्र हैं
उनके बड़े भाई अजीत तेंडुलकर ने उन्हें खेलने के लिये प्रोत्साहित किया था ।
फिर एक मिनट के बाद किताब छत की ओर उछालकर उन्होंने कहा - चलो ।

Result :

of of of
of of of
the of of of

lrb lrb rrb rrb

the the of of of
the the of of of of
the the the of of of of

I could not understand the issue behind this result. I have used some sentences which are same as training sentences. Atleast answer should be correct for them.

Did I do anything wrong? I really appreciate any help

Answer -

if MODE == 'train':
    k_start = 1
    t1 = time()
    # If any trained weight was found, then load them into the model
    if len(saved_weights) != 0:
        print('[INFO] Saved weights found, loading...')
        epoch = saved_weights[saved_weights.rfind('_')+1:saved_weights.rfind('.')]
        model.load_weights(saved_weights)
        k_start = int(epoch) + 1

    i_end = 0
    for k in range(k_start, NB_EPOCH+1):
        # Shuffling the training data every epoch to avoid local minima
        indices = np.arange(len(X))
        np.random.shuffle(indices)
        X = X[indices]
        y = y[indices]

        # Training 1000 sequences at a time
        for i in range(0, len(X), 1000):
            if i + 1000 >= len(X):
                i_end = len(X)
            else:
                i_end = i + 1000
            y_sequences = process_data(y[i:i_end], y_max_len, y_word_to_ix)

            print('[INFO] Training model: epoch {}th {}/{} samples'.format(k, i, len(X)))
            model.fit(X[i:i_end], y_sequences, batch_size=BATCH_SIZE, nb_epoch=1, verbose=2)
        model.save_weights('checkpoint_epoch_{}.hdf5'.format(k))
    print("Time taken to train the data in hour=>", (time()-t1)/3600)

# Performing test if we chose test mode
else:
    # Only performing test if there is any saved weights
    if len(saved_weights) == 0:
        print("The network hasn't been trained! Program will exit...")
        sys.exit()
    else:
        # import pdb
        # pdb.set_trace()
        X_test = load_test_data('test', X_word_to_ix, MAX_LEN)
        X_test = pad_sequences(X_test, maxlen=X_max_len, dtype='int32')
        model.load_weights(saved_weights)
        # print(X_test)
        # print(model.predict(X_test))
        predictions = np.argmax(model.predict(X_test), axis=2)
        # print(predictions)
        # print(y_ix_to_word)
        sequences = []
        for prediction in predictions:
            sequence = ' '.join([y_ix_to_word[index] for index in prediction if index > 0])
            print(sequence)
            sequences.append(sequence)


texts: Is a neural network suitable for this application

texts:
Apologies if this question is not a suitable format. I am a novice in data science.

I have a database of species observation data consisting of ~16 million records.
Each record consists of:

latitude
longitude
date
time
species observed (that's species singular, not plural)

This data has been manually vetted by experts, so there is an additional field for each species in a record that classifies the observation as either valid or invalid (or more accurately speaking, likely correct/likely incorrect)

I am exploring the idea of training a neural network on this data to automatically classify new records as being valid or invalid ("invalid" data will be flagged for manual expert review.)

The vast majority of records are classified as 'valid', so my worry is that there isn't much information to train the model on what constitutes 'invalid'.

However, a good predictor of whether a record is valid is, informally speaking, "are there other records of this species close by (spatially and/or temporally)"

I'm not sure where to start with formulating a neural network for this problem. E.g.

Inputs: latitude, longitude, date, time, species

Output: validity

OR

Inputs: latitude, longitude, date, time

Outputs: one output for each known species indicating validity

I like the idea of this second model as I can input a time and location and get out a list of likely species.

So my concrete questions are:

Does this sounds like an application suitable for a neural network?
If so, where might I start with formulating a model for my problem? Or can someone point me in a good direction to learn more about this topic.


texts: How to handle the target variable being in the features

texts:
I am trying to predict the winner of a tennis match from the players participating and their respective ranks.

Labels of my data:

Winner
Loser
Winner rank
Loser rank

The problem is that the columns holding the player names in my data are labeled 'Winner' and 'Loser'. What is the best course of action to render this dataset usable for machine learning?

Could I just assign the 'Winner' column/vector as my target variable, and construct two new columns ('Player1', 'Player2') and populate them with random choices from 'Winner' and 'Loser'?


texts: Reentry in Computational Neural Networks

texts:
I'm interested in finding an analogue of neurobiological reentrant systems in computational neural networks. It has been found that certain pathways of the brain synapse on earlier regions in the pathway, and there are implications for functionality, learning, cortical organization, and more (Hoshi et al. 2005, Edelman and Gally 2013). Of course, it almost seems natural and intuitive that the brain would connect pathways back on themselves to perform corrections and error minimization.

Based on this intuition, I would have guessed that I could easily find examples of reentry in computational networks. There must be another name for such behavior that I'm missing. I'd like to hear more about work in this area, or if it doesn't exist, why?

I could imagine parallels being drawn between reentry and RNNs or GANs, although any variety of those aren't necessarily what I'm thinking of in terms of reentry.

Edit:

To clarify, here's a simple schematic of what I'm thinking of:

As you can see, there are multiple points of reentry: CA3 is auto-associative, the dentate gyrus and other hippocampal structures synapse back onto the entorhinal cortex, etc. Is there any attempt to replicate this behavior in, say, a feedforward neural network architecture? With some sort of stopping mechanism to prevent an infinite loop, I suppose.


texts:
Is there a way to know how much a sentence is related to a word/topic?
For instance the following dataframe and the topics/attributes Romantique, Feminine, ...:
    comments
0   Très contente de mon achat. Je cherchais ce parfum depuis un temps en magasin et je suis heureuse qu’il soit disponible en ligne il sent tellement bon !! En plus en promo, génial ! \r\nLivraison très rapide !
1   J’adore les parfums de cette marque car je trouve qu’ils sont captivant et surtout ils tiennent toute la journée ! Ils ont des odeurs originales et que l’on ne retrouve pas partout ! Je conseil fortement
2   Le parfum ideal pour porter pendant toutes les saisons du matin à nuit !!!
3   Très bon parfum floral, envoûtant au note de Jasmin qui reste toute la journée\r\nCorresponds aux personnes qui aiment les parfums florales assez imposante
...

As a start I thought about doing a jaccard_similarity distance ...
>>>from collections import Counter
>>>Counter(df['comments'].apply(lambda x: x.split(' ')).apply(lambda x: jaccard_similarity(x,['féminin'])))
Counter({0.0: 1344, 0.025: 21, 0.05: 21, 0.0625: 21})

But is there a better way to see how much a sentence relate to a targeted word?
Update
My main goal is to compare the proportion of people who used some topics in their comments of a product with the presence or absence of these topics in the description of the product. I used a model which use the synonyms:
d = {}
for product in collection.find():
  d_product = {}
  name = product['q0']['Results'][0]['Name']
  description = product['q0']['Results'][0]['Description']
  comments = short_comments_df(product['q2'])['comments']
  #for every attributes
  for attribut in attributs:
    consumers_approved = 0
    # Is the attribut, or its synonyms, in the comments?
    try:
      # if the attribute or it synonyms are in the description then the product has the attribut
      product_approved = presence(synonymes[unidecode.unidecode(attribut)], description)
      # we test every comment to see if they talked about the attribute
      for comment in comments:
        # We only take the nouns and the verbs
        lemmatized_comment = lemmatize_pos_filtering(comment)
        # if the attributes are in the comments then we increment the consumer approved counter
        consumers_approved += presence(synonymes[unidecode.unidecode(attribut)],lemmatized_comment)
      # we take the proportion of people who used the attribute, but shouldn't we normalize it?
      proportion_approved = consumers_approved/len(record['q2']['Results'])
    except IndexError:
      print("IndexError: ",attribut)
    # we use the difference between if we found it in the description and the % of people who found it as well
    d_product[attribut] = product_approved - proportion_approved
  d[name] = d_product

df = pd.DataFrame(d)

It produces the following graph:

It's weird because it shows that for most products the difference between the presence/absence of any topic compared to its presence/absence in the description is the same for most topics, but different from 0! Everything that is above zero means that at least the description has it and none of the comments, but everything below zero means that the comments have it but not the description. What strikes me is these straight lines below zero. It means that the presence/absence of a given attribute in absciss is the same for every comment...


texts: How to plot categorical data and encode it through colour and shape size?

texts:
Given the following data set:

I am wondering what kind of plotting technique can be used to produce such a visualization:

The Skill attribute is mapped to the y-axis, while the Participant is on the x-axis. The other two attributes are encoded as colour and size features of each point in the visualization.

Although I am trying to get this done with Pandas and pdvega, I'd be happy to learn another tool that could render that. But most importantly, I would like to know the proper technical terms used to describe this type of visualization.

I have reviewed various Pandas tutorials, but the problem is that the examples provided are targeting numeric data, rather than categories. I am thinking that perhaps I could partially fake it by turning each category into a number, then somehow overriding the labels on the axis - but it sounds like a convoluted solution to a mundane problem, so there ought to be a nicer way to do it.

Here is the raw data that can be used to produce a Pandas dataframe with the data above:

import pandas as pd
raw = {"Age":{"0":"27..35","1":"18..26","2":"18..26","3":"18..26","4":"18..26"},
       "Skill":{"0":"intermediate","1":"expert","2":"novice","3":"intermediate","4":"expert"},
       "Gender":{"0":"M","1":"M","2":"F","3":"F","4":"M"}}
df = pd.DataFrame(raw)


texts: Why can it be that my neural network is predicting the contrary?

texts:
I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.

Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.

I'm training it with the following:

X = [[-3], [-2], [3], [2]]

Y = [[0], [0], [1.], [1.]]

but when I use that same X to check, it predicts aprox. [[1.], [1.], [0], [0]]

the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.

If I change Y to [1, 1, [0], [0]] the prediction is 0.5 for all instances, and both bias and weight is close to 0. With this Y configuration the cost stays at 0.69 through all iterations.

If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.

This is my gradrient function

    def gradients(self, X, Y):
    dws = []
    dbs = []
    h, activations, zs = self.forward(X)
    da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput
    for d in np.arange(1, len(self.weights))[::-1]:
        a = activations[d] #output for layer d
        z = zs[d] #linear for layer d
        w = self.weights[d] #weights for layer w

        dz = da * a * (1-a)
        m = w.shape[1]
        dw = 1./m * dz.dot(activations[d].T)
        db = 1./m * np.sum(dz, axis=1, keepdims=True)
        da = w.T.dot(dz)
        dws.insert(0, dw)
        dbs.insert(0, db)
    dws.insert(0, None)
    dbs.insert(0, None)
    return dws, dbs

I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.

For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.

Here is the whole code:
GitHub


texts: Root cause analysis using Machine Learning or NN

texts:
I am trying to build a predictive model that will analyze pattern from a snapshot file (log) and give the root cause analysis of the file i.e the class to which the pattern in the logs belongs to.

I have a lift company data. Whenever an error occurs in the lift for eg. the shaft door was opened, a snapshot or log file is generated containing the information about the status of the various switches in the lift. These form a pattern inside the snapshot file, for example each use case(Shaft Door opened) will have a unique pattern. I want to apply Machine learning to try and learn the pattern automatically and predict the output class.

The problem I am facing is in the previous machine learning tasks I faced each observation was a unique example for eg in Loan prediction dataset each row was one observation. But in this case the entire snapshot is one example containing approx 23k rows. How can I apply machine learning in this task. How to consider one snapshot consisting of 23k rows to be one observation with Class label and feed it to the algorithm?.

Also how can I use NN for such tasks as I have come to know that LSTM is good for sequence pattern analysis. My dataset consists of 20 snapshots each of 2 use cases consisting about 61 columns or features. The result I want is whenever i give a snapshot file to the model it should be able to predict the class to which the error in the snapshot file belongs to. (not forgetting the snapshot file itself contains 23k rows)

Thanks in advance. Any help is appreciated.


texts: Do we have any method which handles the imbalanced classification with sample weighting instead of class weighting?

texts: I am looking for methods that use sample weighting instead of class weighting for the imbalanced classification. I think sample weighting is more precise than weighting all the samples from one class with the same weight. Am I thinking right? Is there any paper that discusses this issue?


texts: How to best accommodate snapshots of data over time into a single dataset for training

texts:
Say we have customers who acquire or not a product, and we have snapshots of the customer's profile monthly, with the information if at that given month they acquired or not (binary label).
I have two questions:
(1) What should be the best way to handle this dataset; concatenating the information as columns (for say, the last 3 snapshots so that the model can find a trend), with only the result of the last month? Or appending the snapshots as rows, and do some aggregated calculations (for example, averages for the same customer, how many times it acquired the product in the past x months, etc)
(2) If I go with the appending rows route, I guess all the aggregations have to consider only the previous rows based on the time on the snapshot (otherwise, I guess there would be a leakage of information), but how would I manage a cross validation scenario, should I take a specific caution here?


texts: Spatial Transformer Networks: how is theta differentiable?

texts: In the paper Spatial Transformer Networks, the localization network's output, theta, is differentiable, given the current input feature map. How is this theta differentiable?


texts: When should we consider a dataset as imbalanced?

texts:
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.

My question is, are there any rules of thumb that tell us when we should subsample the large category in order to force some kind of balancing in the dataset.

Examples:

If the number of positive examples is 1,000 and the number of negative examples is 10,000, should I go for training my classifier on the full dataset or I should subsample the negative examples?
The same question for 1,000 positive example and 100,000 negative.
The same question for 10,000 positive and 1,000 negative.
etc...


texts: How to keep a subsetted value for calculating mean

texts:
I am currently learning R and I have to solve an Issue were I have to extract values from a data set which are from a specific month and from this values I should calculate the mean Temp.

I did it like that:

data[data$X..Month.. == 6,]

mean(data$X..Temp.., na.rm=TRUE)

It gave me the mean value but without taking my first statement into consideration.
What do I need to do that both statements are taken into consideration?


texts: I need to know what the abbreviations mean, e.g 'VBD', 'NN', etc. I am learning text processing and now I want to get verbs in the sentences, so that I need to know which abbreviations represent any form of verbs.


texts: Imbalanced dataset - Positive majority class

texts:
My dataset consists of 150 patients where 50 are controls/healthy (negative) and 100 are sick (positive). If I want my model to have high sensitivity at hight specificity, in other words to have low false positive rates, should I correct my model by applying weights to it? Because usually the positive class is the minority class and I see why you need to correct for it but should I in my case?

Thanks


texts: The exact meaning of cost function ? (Machine Learning)

texts:
I'm reading the "Python Machine Learning" book by Sebastian Raschka, and we use different cost functions.
For Adaline model (with a linear activation function) we use the MSE error : (Phi is the activation function and y the target)
$$
Err(w) = \frac{1}{2}\sum(\phi(z) - y)^2
$$
But for the Logistic Regression model we use another cost function :
$$
Err(w) = \sum[-y\log(\phi(z)) - (1-y)\log(1-\phi(z))]
$$

I just want to know what does the cost funciton really mean, it's a probability ? Or just something arbitrary and convex ?
Is there a way to find "the best" cost function for a specific activation function ?


texts:
Im currently doing a subject for data science, and have the following point that im trying to understand.

We are looking to calculate distance in data sets where values may not be present. Now i know that R does this by default, but we are learning the "how" behind the what.

Literature we are give states.
"The idea is to normalise the inner sum by the number of valid (non-missing) terms, so distances computed from different amounts of terms are commensurable. Otherwise, distances computed with fewer missing values tend to be artificially larger."

Given the following data set.

We have the following sample formulas for Euclidean and Manhattan

Euclidean distance: d(x1, x2) = √(4/3) *( (2 – 7)2 + (1 – (-4))2 + (0 – 8)2 ) = √(4/3)*114 = 12.328

Manhattan: d(x3, x4) = (4/2) * ( |3 – 10| + |2 – 5| ) = (4/2)*10 = 20

Assuming the normalization section fro euclidean is number of non missing terms of each row divided?

How do you derive the normalization section on the Manhattan formula?


texts: Importing TensorFlow trained model in OpenCV

texts:
I want to import a TensorFlow pretrained model in OpenCV dnn C++ module.
How can this be done? Are there any examples available?


texts: Is NLP suitable for my legal contract parsing problem?

texts:
My company has a product that involves the extraction of a variety of fields from legal contract PDFs. The current approach is very time consuming and messy, and I am exploring if NLP is a suitable alternative.
The PDFs that need to be parsed usually follow one of a number of "templates". Within a template, almost all of the documents are the same, except for 20 or so specific fields we are trying to extract. That being said, there are sometimes slight inconsistencies such as missing/added spaces, punctation, etc. Currently there are about 15 templates, but sometimes another gets added. Most of the data that needs to be extracted is in tabular form, but a few values are littered in paragraph text.
With the current, non-ML approach, we use a proprietary PDF parser that extracts out all of the tables into a json format. This parser must be custom configured for every new PDF template that appears, and this is time consuming. After converting to json, a custom regex expression must be made for each value in each template, leading to several hundred regexes total. Usually this approach is pretty accurate, but sometimes the slight inconsistencies can break it, meaning a the regex needs to be revised.
I'm wondering if NLP Named Entity Recognition would be a cleaner solution to this problem. My idea is to label all of the values in a few hundred sample documents and then train a custom NER model in a library like Spacy or Flair. Ideally, we could feed in the raw, extracted text from the PDFs instead of having to configure the custom parser to extract json.
The advantage I see in using the NLP approach is that we wouldn't have to configure the custom parser and write a bunch of regexes every time a new template appears. At worst, we would have to label a few of the new documents every time a template is added, which would presumably be faster and easier than the current approach. I think we could also generate tons of synthetic training data easily by swapping labeled values between different documents.
I am concerned that, using an ML approach, we wouldn't be able to achieve near perfect accuracy, which is a requirement. I'm also not sure how well NLP can perform on raw text from tables as opposed to paragraphs. The nice thing is that the documents tend to be very similar within templates.
I've never done NLP before, so I'm wondering if anyone here thinks this approach would be worthwhile exploring. If this is feasible, does anyone have suggestions on how to get the best results?


texts: How to find the initial hyperplane in a Support Vector Machine (SVM)?

texts:
As far as I understand Support Vector machines, we are trying to find the optimal hyperplane, out of all hyperplanes that are equidistant from the support vectors. There are an infinite number of these hyperplanes, so we can start with an initial hyperplane, H1, which is not optimal. We can then vary the length of the normal vector w, so that we slightly change H1 into H2, which has a larger margin. We continue this process until we find H_max, the hyperplane with maximum margin.
My question is how do I find the initial formula of the first hyperplane H1? Given a linearly separable data set in the plane, how can I calculate a hyperplane that separates the data into two classes and is equidistant from the support vectors? Additionally, I am perhaps forgetting the significance of the variable b in this process.


texts: Does MS PowerBI directly connect with Azure's ML workbench?

texts:
Someone wants me to try to see if I can establish a workflow for them using PowerBI and Azure's ML, then show them how to use it. The idea is to create kind of a seemless experience for them, so they done have to worry about coding or dealing with ETL, etc.

Right now I can access Azure but I have to go through some red tape to access the cloud version of PowerBI.

Does anyone happen to know if these 2 product services can be directly linked, such that the person I'm trying to help wouldn't have to worry about importing or exporting files (assuming I pipe the data into Azure's ML for them)?


texts:
I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage.
I am using BeautifulSoup module. Following is the code I am using:
r = requests.get(url) # where url is the above url
bs = BeautifulSoup(r.text)
for movie in bs.findAll('td','title'):
    title = movie.find('a').contents[0]
    genres = movie.find('span','genre').findAll('a')
    genres = [g.contents[0] for g in genres]
    runtime = movie.find('span','runtime').contents[0]
    year = movie.find('span','year_type').contents[0]
    print title, genres,runtime, rating, year

I am getting the following outputs:
The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. (1994)

Using this code, I could scrape title, genre, runtime,and year but I couldn't scrape the imdb movie id,nor the rating. After inspecting the elements (in chrome browser), I am not being able to find a pattern which will let me use similar code as above.
Can anybody help me write the piece of code that will let me scrape the movie id and ratings ?


texts: Is there any proven disadvantage of transfer learning for CNNs?

texts:
Suppose I know that I want to use a ResNet-101 architecture for my specific problem. There are ReseNet-101 models trained on ImageNet.

Is there any disadvantage of using those pre-trained models and just resetting the last (few) layers to match the specific number of classes or should that be the default option?

Please don't simply post your gut feeling... I have a gut feeling as well, but I want to know.


texts: What is the meaning of each element in input_shape of Conv1D in Keras?

texts:
I have a time-series data for 3 classes (each class is 35 second) as I extract each 1 second for 95 feature extracting so my final data has shape (105,95) (rows for time and column for feature).
I am currently building a 1D-CNN model for classification 3 class. But I got stuck as first layer of Conv1D. I have learnt that the input_shape of Convd1D is (batch_size, new_step, input_dim) but honestly I dont know what exactly what each element mean and how can I modify (reshape) my input data into Conv1D layer shape?
And one more question, I know that CNN required fixed input size. But I split my data into train and validation data as k parameter = 5, mean that my data will become: Train = (84,95) and Validation = (21,95). So with the same model how can we train and validate data with difference size?
I just start to learn about CNN but still not know how Conv1D operate?
I am appreciate your help.
Thank you very much.
Hear is my code (Which still get error because of first Conv1D input_shape):
X = dataset[:, 0:95].astype(float)
Y = dataset[:, 95].astype(int)

# Encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoder_Y = encoder.transform(Y)

y = np_utils.to_categorical(encoder_Y)

acc_per_fold = []
loss_per_fold = []

fold_no = 1

kfold = KFold(n_splits = 5, shuffle = True)
for train, test in kfold.split(X,y):

    model = Sequential()
    model.add(Conv1D(100, 3, activation = 'relu', input_shape = (I really dont know what to put in here??))
    model.add(Conv1D(100, 3, activation = 'relu'))
    model.add(MaxPooling1D(3))
    model.add(Conv1D(150, 3, activation = 'relu'))
    model.add(Conv1D(150, 3, activation = 'relu'))
    model.add(GlobalAveragePooling1D())
    model.add(Dense(3, activation = 'softmax'))

    model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

    #model.summary()

    print('------------------------------------------------------------------------------')
    print(f'Training for {fold_no}...')

    history = model.fit(X[train], y[train], batch_size = 7, epochs = 150, verbose = 1)

    scores = model.evaluate(X[test], y[test], verbose = 0)
    print(f'Score for fold {fold_no}: {model.metrics_names[0]} of {scores[0]}; {model.metrics_names[1]} of {scores[1]*100}%')
    acc_per_fold.append(scores[1] * 100)
    loss_per_fold.append(scores[0])

    # Increase fold number
    fold_no = fold_no + 1


texts: Is it beneficial to use the last $N$ data points to train an RL agent?

texts: Given that an environment in reinforcement learning is a Markov Decision Process (MDP), are there ever any cases where it is beneficial (or indeed where it makes sense) to use the last $N>1$ data points to train the agent rather than just the current data point?


texts: When to use a Sigmoid function and when to use ReLu in a Convolutional Neural Network

texts: Can someone please refer a good article explaining why we use a Sigmoid activation in the final layer of a neural network and a ReLu activation in the middle layers and input layer while building a Convolutional Neural Network? I am not getting how that results in the right output.


texts: Algorithm to capture maximum number of points spread across the four quadrants

texts:
I was preparing for the data science interview questions and encountered the following problem:
You are given some points in four quadrants and the points are fixed. You need to point the camera at an angle theta from +x axis.

Where will you point the camera so as to capture the maximum number of the points?

How will you improve your search algorithm?

There was no answer given and it was an open question. How to approach this problem.
My attempts

think about clustering algorithms


texts: In feature selection, I came across a situation where NaN were filled by median of the column values

texts: Why the median value is used for NaN? Why not something else like mean? What is the logic behind using the median value?


texts: How to find and use the top features for XGBoost?

texts:
Suppose I have data with X_train, X_test, y_train, y_test given. As it is a classification problem I want to use XGBoost.

The issue is that there are more than 300 features.

I have found online that there are ways to find features which are important. But as I have lot of features it's causing an issue.

My current code is below. How can I modify it to say select top n ( n = 20) features and use them for training the model. I tried sorting the features based on importance but it doesn't work.

   import xgboost as xgb
    gbm = xgb.XGBClassifier(max_depth=3, n_estimators=300, learning_rate=0.05).fit(X_train,y_train)
    predictions = gbm.predict(X_test)