texts: Use feature-importance to decide what features to increase to increase target

texts:
(Please suggest another title for this question if you like - I find it rather difficult to phrase)
Say I have an ice-cream stand and I record 3 features of my customers f1,f2,f3 to predict the amount of purchases. Say we train a logistic regression, logreg, and calculate the feature importance using e.g SHAP and denote i(f_n) the importance of feature n. It turns out that i(f2)>i(f1)>i(f3), see the table below
Feature | Importance rank | correlation with target |
--------+-----------------+--------------------------
f1             2                    +
f2             1                    -
f3             3                    +

Based on that one might say that "To increase the purchase we just need to maximize f2 and reduce f1".
That way we can draw some very wrong decisions, since f2 might increase the abillity to predict the target (amount of purchases) but we aren't sure that just by increasing f2 we increase the amount of purchases.
For instance; the number of ice creams sold is correlated with the number of people who drowns; we cannot save people from drowning just by not allowing people to sell icecream (which would be bad for my ice-cream stand as well) - there is a latent/hidden variable here called "temperature" which we have not taken into account.
Further more the table above might change if we add other features (or chose another model e.g RegressionTree).
So, my question is; can we ever use things such as feature importance to "reverse engineer" i.e say: "if we want more of our target, we need to increase feature f1" and not just say "f1 is positively correlated with y and has a high predictive power"?


texts:
I am new to Data Science and CNN.

My understanding of CNN is that:

An image's pixel data is convoluted over with filters which extract features like edges and their position.
This creates filter maps.
Then we apply max pooling which will down sample the data.
Then we feed this data to a neural network which learns to classify.

Now I know I can do things with convolution like blurring or get edges from image.

But lets say I am trying to make a "Hotdog or not" app, how do I make my filters? What will the matrix be?


texts:
I'm trying to automatize cereal counting from an image. The problem is that I have only a few pictures and I don't know how to train a CNN to do the job.
Should I create mini images with single cereals and use those to train the network?
I would like to extract the coordinates, not only count them.


texts: Keras val_acc unchanging when training (same label assigned to all images)

texts:
I am using Keras to make a "set" identifier for the card game Set.
Here is my script (some code may be unnecessary but was used during experimentation):
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Activation
from keras.layers import Flatten
from keras.layers import Dense
from keras.layers import Dropout
from keras.callbacks import EarlyStopping
from keras.optimizers import SGD
from keras.utils import to_categorical
from keras.layers.advanced_activations import LeakyReLU
from keras.regularizers import l2

import os
import glob
import shutil

import numpy as np
from keras.preprocessing.image import load_img, img_to_array
from keras.applications.imagenet_utils import preprocess_input

earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5,
                          verbose=1, mode='auto')

callbacks_list = [earlystop]

# opt = SGD(lr=0.01)

setIdentifier = Sequential()

setIdentifier.add(Conv2D(32, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))

setIdentifier.add(MaxPooling2D(pool_size = (2, 2)))

setIdentifier.add(Conv2D(64, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))

# setIdentifier.add(Conv2D(64, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))

setIdentifier.add(MaxPooling2D(pool_size = (2, 2)))

setIdentifier.add(Flatten())

setIdentifier.add(Dense(units=256, activation = 'relu'))

setIdentifier.add(Dropout(rate=0.5))

setIdentifier.add(Dense(units = 2, activation = "softmax"))

setIdentifier.compile(optimizer = "adam", loss='categorical_crossentropy', metrics = ['accuracy'])

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(rescale=1./255,
                shear_range = 0.2,
                zoom_range = 0.2,
                horizontal_flip = True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory('Data_Training',
                                                target_size = (64, 64),
                                                batch_size = 32,
                                                class_mode = 'categorical')

test_set = test_datagen.flow_from_directory('Data_Testing',
                                            target_size = (64, 64),
                                            batch_size = 32,
                                            class_mode = 'categorical')

setIdentifier.fit_generator(training_set,
                                steps_per_epoch = 486,
                                epochs = 50,
                                validation_data = test_set,
                                validation_steps = 60,
                                callbacks = callbacks_list)

setIdentifier.save("setIdentifier.h5")

The file Data_Training has a folder notSet and a folder Set, and the file Data_Testing likewise. There are two times as many notSets than Sets in each category.
No matter what I do, in both training and testing phases, the output is always notSet. I am not sure what is the cause for this.
I have tried a binary_classifier with the sigmoid activation function and final density layer with 1 node. I have tried changing activation functions and using leakyRelus.
I have double checked my training steps and test steps.
I have $5184\text{ sets }+ 2 \cdot 5184\text{ notSet = }15552$ train images, which when divided by a batch size of $32$ gets me $486$ steps.
I have $648\text{ Sets }+ 2 \cdot 648\text{ notSet = }60.75$ >> rounded to $60$ validation steps.
I am confused by what is going on with the neural network - why isn't anything approving and why is it labelling everything with one class?
For more info, here is the Github repo which includes the training images.
Any help is appreciated!


texts: How to put multiple features into RNN input vector

texts:
I am trying to code a recurrent neural network (LSTM) to create music in python and was considering using multiple features instead of just the note pitch as an input into the network. Initially I had just the note pitch so it was fed into the network by one-hot encoding it. The other two features I want to add are the note duration and the offset between the notes. How should the input vector be organised so that all the data is fed through the network?

I have tried combining all of the data into a long vector with all 3 features one-hot encoded and then concatenated but this caused the output to become 'NaN'. Any help would be appreciated.

Link a gist of my code:

https://gist.github.com/treutm/1b1f49e3d0a9de53cd67c136292f329f


texts: How clustering is used in data management?

texts:
Can anyone please tell me how is clustering is used in data management?

I was recently asked this question and was left totally stumped.


texts: How to improve SGD classifier performance?

texts:
I am having some issues with SGD to run with a sample of approx 5000 obs for a classification problem with imbalance.
I am doing
c_sgd = Pipeline(steps=[('preprocessor', preprocessing_countV),
                      ('classifier', SGDClassifier(loss= 'log', alpha = 0.1, tol = 0.0001, max_iter = 10000, shuffle = True, random_state = 0))])
print("SGD:", cross_val_score(c_sgd, X_train, y_train.values.ravel(), cv = 5))

But I am getting very poor performance: SGD: [0.96613546 0.52624585 0.00531561]
I already removed highly correlated features, cleaned data, used resampled method for balancing classes...
The other classifiers (Logistic regression, Naive Bayes) perform quite well (I think):
For example:
c_lr = Pipeline(steps=[('preprocessor', preprocessing_countV),
                      ('classifier', LogisticRegression(C=10, tol=0.01, solver='lbfgs', max_iter=10000))])

LR: [0.94422311 0.98006645 0.98671096]

It seems to be a problem of classifier.
Do you have any suggestions on how to improve SGD classifier performance?


texts: Multi-object detection within single image

texts: Given an image with multiple objects within it, I would like to train a CNN to output vector of labels corresponding to the presence/absence of objects within the image. I would like to know whether there are publications/experimental work on such an approach to gauge it's feasibility on whether it is possible to implement it for my own use case. So far, I haven't been able to find literature on multi-object detection within a single class.


texts: Machine learning for state-based transforms?

texts:
If I provide:

A list of possible transforms, and,
A list of input states, and,
A corresponding list of output states for each input state, and,
A fitness function to score each output state

Which subset of machine learning can direct me towards an optimization algorithm that can map each input state to a dictionary of input states, and, failing to find a match, apply the necessary transforms to get me to the closest-related output state?

An example involving polygon legalization:

Any given "window" can contain N different polygons, where each polygon has lower-left and upper-right co-ordinates, as well as a polygon "type".
The input state of the polygons may or may not be "illegal".
A list of transforms includes: move, copy, rotate, resize
If the input state maps directly to any output state, the input state is decided to be legal. Nothing more to be done; move on the next window.
If the input state matches any previously seen input state, transform to the matching (known-legal) output state. Nothing more to be done; move on the next window.
Attempt transforms in different sequences until a state is reached that satisfies a fitness function. Store this input:output state combination. Move on to the next window.

Would this imply some combination of neural networking (for classification) and genetic/evolutionary algorithms? Or, does the presence of a fitness function negate the need to store combinations of input:output states?


texts:
I am struggling to understand why i am getting such a high loss/val_loss rate on my training. I am training a regression network. I've normalized the input data to range between -1 to 1, and left the output data unaltered, its range is approx. between  -100 to 100.

I chose to normalize the input such that i could use tanh, as activation function as it outputs within this range.

The neural network consist of 3 layers.

  print "Model definition!"
    model = Sequential()

    #act = PReLU(init='normal', weights=None)
    model.add(Dense(output_dim=400,input_dim=400, init="normal",activation=K.tanh))

    #act1 = PReLU(init='normal', weights=None)
    model.add(Dense(output_dim=400,input_dim=400, init="normal",activation=K.tanh))

    #act2 = PReLU(init='normal', weights=None)
    #model.add(Dense(output_dim=400, input_dim=400, init="normal",activation=K.tanh))

    act4=ELU(100)
    model.add(Dense(output_dim=13, input_dim=400, init="normal",activation=act4))

The mapping between the input and output consist of mapping audio samples into MFCC features.  The samples are the ones i've normalized to the described range.

Why am i getting these results?..

am I making anything that is unclear?

Normalizing the output_range +-1:

Why is the val_loss loss lower than the loss?


texts: What is difference between leave one subject out and leave one out cross validation

texts:
What is the difference between leave one subject out cv and leave one out cross validation (loocv)? are they same or different?. I have images of 24 subject and according to literature, leave one subject out is best cross validation for pain expression detection because  of its subjective nature.
is there any function for leave one subject out cv in sklearn like loocv? or I have to manually hold out one subjects images  (one subject out of 23) and train for rest 23 subjects? is it right approach ?


texts: Policy gradient on data only, without emulators

texts:
It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have a lot of data, that represents a sequence of state, action, reward. We can train our agent on this data.

We also need continuous actions, as the set of actions is big. Policy gradient is therefore the way to go but it generally uses an actor-critic that requires an emulator. We can not emulate, what would be the other options?


texts: What is the main architectural difference between DCGAN and WGAN?  For which problems each models can be more useful than the other one?


texts: How can I use all possible spelling correction of documents before clustering those documents?

texts:
I have the data set with many documents of 50 to 100 words each.

I need to clean those data by correcting misspelled words in those documents.

I have an algorithm which predicts possible correct words for misspelled word.

The problem is I need to choose or verify the predictions made by that algorithm in order to clean the spelling errors in the documents.

Can I use all the possible correct words predicted for correct spelling in word vector to perform clustering on those data?


texts: Inputs required for Random Forest Regressor and ways to improve performance

texts:
I am using Random Forest Regressor to predict inventory needs. The data I am using to train the model lists the total quantity picked for each product per date, but does not include rows where total quantity picked for a product on the specified date is 0. The model considers the features that allow it to take date into consideration. For details on the data being used see the example data below:

                 UPC  day_ID  month    day_of_year  day_of_week  quantity_picked
0      0000000002554  7500.0      5            141            1              4.0
1      0000000002554  7503.0      5            144            4              2.0
2      0000000002554  7512.0      6            153            6              2.0
3      0000000002554  7527.0      6            168            9              2.0
4      0000000003082  7494.0      5            135            2              2.0
5      0000000003082  7495.0      5            136            3              2.0
6      0000000003082  7496.0      5            137            4              8.0
7      0000000003082  7497.0      5            138            5              4.0
8      0000000003082  7498.0      5            139            6              4.0
9      0000000003082  7499.0      5            140            0              9.0
10     0000000003082  7500.0      5            141            1              3.0
11     0000000003082  7501.0      5            142            2              5.0
12     0000000003082  7502.0      5            143            3              3.0
13     0000000003082  7503.0      5            144            4              8.0
14     0000000003082  7505.0      5            146            6              2.0
15     0000000003082  7506.0      5            147            3              7.0

Will the model be less accurate at predicting inventory needs because it is missing dates for items where quantity picked is 0?
I have tried running the same model with the rows where quantity picked = 0 but the total number of rows changes from approximately 50k to 5 million and my computer literally can't handle it, it just freezes.
Without the rows where quantity picked = 0, the model reports mean squared log error level of .39448 and runs successfully within 4 minutes 37 seconds.

Any guidance on if that data is necessary or not would be very much appreciated and/or advice on how to improve performance/accuracy of such a model.


texts:
I have built a correlation matrix to check multicollinearity in a regression model. But how to interpret this?

Can we say that there is a certain correlation value from which the independent variables related to this correlation should be removed from the model? Is this the value over 0.5?
Are there any better solutions than removing the conflicting variables (when correlation value is too big)? I tried normalization but it didn't help.


texts: Logistics Demand Forecasting with 20k Different Time Series

texts:
I'm trying to tackle a very challenging problem and I would appreciate your help.
My organization has a lot of different items which can be demanded by our clients. Those items can also be returned back to our warehouses. Our goal is to predict how many instances of each item our clients (which are actual warehouses) will demand next year.
As for now, my organization is predicting the demand using a weighted average, based on the quarterly demands of the last 7 years. This gives us a prediction for the demand for the next quarter. Then, they multiply this prediction by 4 to get the next year's prediction. This is my baseline to beat.
Our data is irregularly sampled, meaning that demand can occur any day. This also means that we have to resample the data ourselves (we tried monthly and quarterly).
Each demand has a few features:

Item Id
Date of Demand
Receiving Warehouse - our organization has a lot of warehouses spread out across the country, this is the warehouse that received the item.
Sender Warehouse - the warehouse that sent out the item.
Type of Demand
Quantity - can be either positive (if the client consumed the item from the warehouse) or negative (if the client returned the item back to the warehouse)

Again, for clarification: the goal is to predict the Total Quantity of the next year for each item.
Obviously, this is a time-series forecasting situation. The problem is that we have over 20K different time series (because each item’s history is a distinct time series).
What we have done so far:

Simple statistic models (exponential smoothing, Holt-Trend Methods, etc.)
Classical machine learning models (linear regression, decision trees, etc.)
Ensemble methods (random forest, boosting, etc.)
Simple deep learning models – neural networks with a few linear layers and an RNN layer.
Complex state of the art neural networks for time series forecasting (we tried Google's Temporal Fusion Transformer)
Calculating the regular weighted average (as the company does now) but only after removing anomaly demands (based on a DBSCAN algorithm).

The main problem is that none of the above methods can beat the regular weighted average for all the time series we have – meaning that for many of them the new proposed model preformed even worse than the weighted average. Also, we couldn’t find any pattern that allowed us to match each time series to a "winning" method.
What we are looking for is a method which will perform better for all the different time series.
Any help will be highly appreciated.
Thanks in advance.


texts:
Like above, I'd like to know, what exactly a skewed dataset is.

The explanation from stats.se.com sounds to me more like what I call an imbalanced dataset?

What is the distinction?


texts: Light GBM Regressor, L1 & L2 Regularization and Feature Importances

texts:
I want to know how L1 & L2 regularization works in Light GBM and how to interpret the feature importances.

Scenario is: I used LGBM Regressor with RandomizedSearchCV (cv=3, iterations=50) on a dataset of 400000 observations & 160 variables. In order to avoid overfitting/reguralize I provided below ranges for alpha/L1 & lambda/L2 parameters and the best values as per Randomized search are 1 & 0.5 respectively.

'reg_lambda': [0.5, 1, 3, 5, 10]
 'reg_alpha': [0.5, 1, 3, 5, 10]

Now my question is about: Feature importance values with optimized values of reg_lambda=1 & reg_alpha=0.5 are very different from that without providing any input for reg_lambda & alpha. The regularized model considers only top 5-6 features important and makes importance values of other features as good as zero (Refer images). Is that a normal behaviour of L1/L2 regularization in LGBM?

Further explaining the LGBM output with L1/L2: The top 5 important features are same in both the cases (with/without regularization), however importance values after top 2 features has been shrunk significantly by the L1/L2 regularized model and after top 5 features the regularized model makes importance values as good as zero  (Refer images of feature importance values in both cases).

Another related question I have is: How to interpret the importance values and when I run the LGBM model with Randomized search cv best parameters do I need to remove the features with low importance values & then run the model? OR should I run with all the features & the LGBM algorithm (with L1 & L2 regularization) will take care of low importance features and won't give them any weight or may be give minute weight when it makes predictions.

Any help will be highly appreciated.

Regards
Vikrant


texts: Multivariable real time system for fraud detection

texts:
This is the scenario:

Client -> Server

The client sends multiple voice calls to Server.
Call info:

calling number
called number
call duration
source IP
destination IP
SIP header information (User-Agent, Version, Proxy information)

Other patterns:

Numeric Patterns - Destination.
Time Patterns - Calls occurring too regularly, activity at unusual times or dates.
Geographic Patterns - Proximity relationships between apparently unrelated entities.

In case of Toll fraud and client being insecure, the attacker can send calls via original agent hence the Service Provider won't reject calls immediately at least based from IP information. (Toll fraud).
I'm exploring which approach is the best to implement an ML model, instead of static rules to be able to detect Toll fraud in a live system:

Classification (Fraud or not)
Reinforcement Learning (If fraud then reward)

I'm new to RL, does this scenario makes sense for live systems?
Can I use offline data to train an RL model? Offline data includes Fraud cases detected historically.

Reference.


texts: Aggregating target-encoded array-like categorical features?

texts:
I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables for machine learning classification algorithms.

One-hot encoding leads to very high dimensionality. The approach I've landed on is target-encoding/mean-encoding. I understand how to use this when the categorical feature is a single choice (eg current zip code). But, when the feature can take on multiple values from a large list (eg favorite hobbies, illness symptoms, university coursework), I am not sure how to combine the values.

My intuition says that the wrong approach would be to take each unique combination as its own factor and encode that, as it would lead to overfitting. Other things that come to mind would be simple aggregations like sum/avg/product/variance.

How should target encoded values be combined?


texts: How to solve this classification problem: multi-class or multi-label?

texts:
In a supervised cancer classification task which is given the data containing metrics we want to classify whether the patient has cancer or is at high risk (label 1) or low risk (0). However, there is another attribute which is screening status which I want to include in the prediction. But I don't know how -- should it be a label or an input feature? The information on patients screened have the screening year as the value. Those who were not screened have the value for the year empty. Let the label for is screened be 3 and not screened be 4. So, the idea is:
-- a patient is at high risk and was screened : [1,3]
-- patient is at high risk but not screened: [1,4]
-- patient is at low risk and was screened: [0,3]
-- patient is at low risk but not screened: [0,4]

Basically, I want to assess the importance of early screening in predicting benign/malignant. The patients who have been screened early may not have cancer. On the other hand, it may happen that those who were screened developed cancer in later stages. This implies that the patients who did not pass the screening tests were diagnosed with cancer. So, what are the factors that influence the patient to pass or reject the screening test and whether the screening test itself can help in predicting/classification of malignant/benign. This is the research problem.
Confusion 1: How can I include this screening status? Will the information on screening be included as a label ?
Confusion 2: Will it be multi-label or multi-class classification?
Is my approach correct? Please let me know if I am in the correct direction an please correct me otherwise. Links to papers with similar problem will also be helpful.


texts: What does it mean that an hypotesis is consistent?

texts:
I am studying concept learning, and I am focusing on the concept of consistency for an hypotesis.
Consider an Hypotesis $h$, I have understood that it is consistent with a training set $D$ iff $h(x)=c(x)$ where $c(x)$ is the concept and this has to be verified for every sample $x$ in $D$.
For example consider the following training set:

and the following hypotesis:
$h_1=<?,?,?,Strong,?,?>$
I have that this is not consistent with D because for the example $3$ in $D$ we have $h(x)!=c(x)$.
I don't understand why this hypotesis is not consistent.
Infact, consider the following hypotesis:
$h=<Sunny,Warm,?,Strong,?,?>$
this is consistent with $D$ because for each example in $D$ we have $h(x)=c(x)$.
But why the first hypotesis $h_1$ is not consistent while the second,$h$, is consistent?
Can somebody please explain this to me?


texts: Can you interpret probabilistically the output of a Support Vector Machine?

texts:
I am trying to build a binary classification system using different classification algorithms like random forests, support vector machines, AdaBoost. I want to use the output of these classifiers to visualize a score. For example, when using random forests, I would like to use the probability of a sample belonging to class A to build a score from 0 to 100. Given that random forests output a probability (from 0 to 1) using it as the score is intuitive (I would just multiply it by 100). However, given that SVMs output a classification but not a probabilistic output (i.e. distance to the hyperplane, but not probability), would it be legitimate to use the distance to the hyperplane as some sort of "pseudo probability"?  I would, for example, do max-min scale on the distance to the hyperplane for all samples so all distances are scaled from 0 to 1.

I want to be sure that I can use the distance to the hyperplane as a pseudo-probability and that this pseudo-probability is comparable to the probability to belong to a given class outputted by the random forest. For example, that a sample with a probability of .80 of belonging to class A is the same than another sample of a (min-max transformed) probability of belonging to class A according to the SVM.


texts: Convert a binary neural network classifier to one verses all classifier

texts:
I have a neural network model (implemented from scratch) which gives me some continuous outputs and I have used a sigmoid layer, in the end, to convert it into a binary classifier. But my original problem is a multiclass problem.
I tried to use the softmax layer but somehow that cannot be applied to my model architecture as the output is not in that format (I get some continuous value as output).

Thus I realized that an alternative way would be to use one versus all classifier but I am not able to figure out how to put that on top of my binary classifier.
Please forgive me if my concepts are a bit weak, but I would really be thankful if someone could help me with this.

Like, how can I convert the binary classifier to a multiclass classifier while keeping the original architecture and what kind of loss function should be used? Would that be different from the binary classifier loss function?

Thanks in advance.


texts: XOR problem with neural network, cost function

texts:
I am having a problem understanding the cost function in a neural network. I have read many books and blog posts, but all of them describe that point in neural networks is to minimize the cost function (like sum squared error):

I tried to look at code for solving a problem with a multi layer neural network and back propagation. My question is: where in the code can I find the cost function? How can I plot the error surface?

import numpy as np
X_XOR = np.array([[0,0,1], [0,1,1], [1,0,1],[1,1,1]])
y_truth = np.array([[0],[1],[1],[0]])

def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def sigmoid_der(output):
    return output * (1 - output)

np.random.seed(1)
syn_0 = 2*np.random.random((3,4)) - 1
syn_1 = 2*np.random.random((4,1)) - 1

for i in range(60000):
    layer_1 = sigmoid(X_XOR.dot(syn_0))
    layer_2 = sigmoid(layer_1.dot(syn_1))
    error = 0.5 * ((layer_2 - y_truth) ** 2)
    layer_2_delta = error * sigmoid_der(layer_2)
    layer_1_error = layer_2_delta.dot(syn_1.T)
    layer_1_delta = layer_1_error * sigmoid_der(layer_1)
    syn_1 -= layer_1.T.dot(layer_2_delta)
    syn_0 -= X_XOR.T.dot(layer_1_delta)
    if i % 10000 == 1:
        print(layer_2)

print(layer_2)


texts:
I have event dataset in a factless table. It has list of events

timestamp-> event name -> node ( In network)

There is always a sequence of event happening. So how do I start predicting future event based on past one and discover list of nodes that will be affected from past experience.

I am programmer without knowledge of machine learning. I have installed spark,R and have dataset in oracle database.Is there any tutorial/algorithm that I can use to get started. I taught myself scala/R but have no idea on getting started. My dataset are huge i.e. more than 9billion rows for 3 months.

Node            Eventtime       alarmname
192.168.1.112   6/14/2016 19:41 Main power supply has a fault alarm
192.168.1.113   6/14/2016 19:23 Association path broken
192.168.1.113   6/14/2016 19:23 NA
192.168.1.113   6/14/2016 19:23 Association broken
192.168.1.112   6/14/2016 19:23 Mains Failure
192.168.1.112   6/14/2016 19:23 Mains Failure

Additional Information:

I have 98 nodes. I would like to predict:

i. No of node that has alarm or goes down when a single node goes down
e.g. if node A has alarm if in 1 month list of node that has alarm at the same period

ii. The sequence of event occurrence i.e. if one node has mains failure then the next event would be node down.


texts:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('iris.csv')
X = dataset.iloc[:,:4]
y = dataset.iloc[:,4]

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse=False)
y=encoder.fit_transform(y.values.reshape(-1,1))

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.naive_bayes import GaussianNB
classifier = GaussianNB()
classifier.fit(X_train, y_train)

I am getting these small errors which hinder my projects. Can you please provide a solution to this error?
Also how can I avoid these mistakes?


texts: Extraction information from PDF files using ML (Invoice number, line items in table)

texts:
First of all, I am fairly new to the ML world. I have researched quite a lot on the different use cases that ML have, both in regards to working with text and images.

I am trying to build a "pipeline", that can extract a few data points from various supplier invoices:

Invoice number
Line items (that resides in tables)

For each line item, I would need: quantity and line amount

My first thought was to just use a classic OCR parsing tool such as DocParser (which is basically a template-based OCR parsing tool, where you can create parsing rules for each different type of invoice layout). However, I took a look at my suppliers, and I have a lot of different layouts (with new ones being added regularly).

I was thinking if ML can be used to accomplish this task?

My idea for a pipeline:

All supplier invoices are converted from PDF to an image file (.jpg) and then resized so all have the same width and height.
Train a custom model to extract invoice number using named entity recognization (NER)
Train a custom computer vision model to identify tables that contain line items (product information)

For each table found, extract that as an image and train another model to identify the entities that I need for each line: quantity and line amount

I am not sure if this is a good approach to this problem? Does it make sense to ultimately end off with three models to extract the information that I need? Is there an easier way to detect the quantity and line amount, than by first locating the table on the PDF file?

Does anyone have any experience with a similar process?


texts: Machine Learning algorithm for predicting number of cases in pandemic

texts:
I’m giving my first steps with AI and Machine Learning so I have the following issue. I’m trying to predict an outcome from COVID-19 number of day vs confirmed cases using scikit-learn library. I mean, my input is the number of days since the pandemic started in my country and my output is the number of confirmed cases in that corresponding date. However both using GradientBoosting and RandomForest I get the same output values for the test values…I post below the code in Python as it is very short…

import numpy as np

from sklearn import ensemble

import pandas
datos = pandas.read_csv('covid.csv',";")

entrada = np.array(datos['ORDEN']).reshape(-1,1)

salida = datos["CASOS"]

regr = ensemble.GradientBoostingRegressor(random_state=0,n_estimators=500).fit(entrada,salida)

test = np.array([i for i in range(63,70)]).reshape(-1,1)

print(regr.predict(test))

regr = ensemble.RandomForestRegressor(random_state=0,n_estimators=500).fit(entrada,salida)

print(regr.predict(test))

My output is this:

[1782.99976513 1782.99976513 1782.99976513 1782.99976513 1782.99976513
 1782.99976513 1782.99976513]
[1773.99 1773.99 1773.99 1773.99 1773.99 1773.99 1773.99]

What am I doing wrong?? Thanks in advance.


texts: Preparing custom dataset for object detection using ML

texts:
Seeking clarity on single class object detection model using ML. I have prepared a custom database for this purpose up to 400 images which is split in 80%-20% as training and testing data-set. These are top view only images. The data collection followed the basic guidelines provided at here.

The objective now is to detect the zebra crossing in below contexts.

Model is failing in terms of accuracy. Although it identifies the class but fails to localize correctly in some contexts(greater extent).The desirable result is in blue whereas the model throws up red.

What changes are required to training dataset to rectify this?

EDIT1:-
Tensorflow object detection API is used for this task. Detection accuracy attained is above 90%. Looking forward for suggestions to fix Localisation issue.

EDIT2:-
The illustrations here are only for outlining the issue.All my images are real world pics.


texts: How do I test a difference between two proportions representing fatality rate for Covid 19 in Philippines and World (except Philippines)?

texts:
I'm trying to analyse if the fatality rate from my country (A third world country) vary significantly from the world's fatality rate.

So I'd basically have two samples, labeled (Philippines) and (World excluding the Philippines) then i can compute the fatality rate for the 2 groups.

Does Mcnemar's test apply here for me to check if fatality rate in the Philippines is higher, or do you have any suggestions? Thanks


texts:
I'm using a Lasagne network, but this is a pretty generalized question:

say I ingest N number of tweets using the same LSTM, grab the final output of each to make a NxOutputSize tensor (Merged_output) to be condensed into a small 10-vector output through a shallow neural net. What would be the best way to train this LSTM?

Thanks


texts: Text extraction from documents using NLP or Deep Learning

texts:
I am looking for references(Papers/github projects) on how to use deep learning in a text extraction task.

Recently I was given a task to extract important information from documents of similar type, say for example legal merger documents. I have thousands of legal merger documents as inputs. A paralegal would go through the entire document and highlight important points from the document. This is the extracted text.

What I want to do: Given a document(say legal merger document) I want to use DL or NLP  to extract the information from the legal document that would be similar to that of the information extracted by paralegal.

I am currently using bag of words model to extract text from the document, calculating sentiment and displaying the sentences with positive or negative sentiments. This yielded very bad results.

My knowledge in DL/NLP is very limited and I am particularly looking for some interesting papers and github projects related to text extraction using these frameworks. Can anyone please provide me with some references and suggestions on how to tackle this issue?


texts:
im fairly new to neural networks.
I want to ask what exactly does the activation output, is it the probability the combined summation of inputs and weights lead to a match for the next neuron?
Thanks


texts: Difference between Word Embedding and Text Embedding

texts: I am working on a dataset of amazon alexa reviews and wish to cluster them in positive and negative clusters. I am using Word2Vec for vectorization so wanted to know the difference between Text Embedding and Word Embedding. Also, which one of them will be useful for my clustering of reviews (Please consider that I want to predict the cluster of any reviews that I enter.) Thanks in advance!


texts: Memory efficient encoding logic for group categories

texts:
I have a huge dataset with categorical data. It is comprised of alerts having multiple properties. Each alert belongs to a group, and some even belong to multiple groups. It looks somewhat like this:

     GroupID           System        State       TimeStamp        etc...
0    [1, 2, 3, 4]         A           REC           ...
1    [1, 2, 3, 4]         A           SNT           ...
2    [2, 4]               B           REC
3    [2, 4]               B           PND
4    [2, 4]               B           COM
5    [2, 4]               B           SNT
6    [2]                  C           RCV
7    [2]                  C           ACC
...

There are more than 100000 different group IDs in over 3 mil alerts.

Creating a column with a single Group ID value (not a list) means some alerts will appear more than once, which is not good given the already huge dataset.
Creating a separate column for each group (binary encoding) would expand my data too much horizontally.

What is a memory efficient way of encoding Groups?


texts: Python and GridSearchCV how to eliminate input contains NaN error when using cross validation and decision tree classifier?

texts:
I am trying to do cross validation on Decision tree classifier for kaggle's titanic dataset. The first step after cleaning data is to split into train and test sets:

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(train, Y, test_size=0.2, random_state=0)

Then transform numbers into scaled values:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train = sc.fit_transform(x_train)
x_test = sc.transform(x_test)

In addition:

from sklearn.tree import DecisionTreeClassifier
classifier = DecisionTreeClassifier()

For the grid search, I used GridSearchCV:

#Make a grid search
from sklearn.model_selection import GridSearchCV
tree_param = [{'criterion': ['entropy', 'gini'], 'max_depth': [2,3, 4]}]

And at the end to fit the GridSearchCV classifier into data:

clf = GridSearchCV(classifier, tree_param, cv=4)
clf.fit(X=x_train, y=y_train)

The error I am getting is as follows:

  ValueError: Input contains NaN, infinity or a value too large for
  dtype('float32').

I checked my x_train and y_train sets, and they both contains numeric values within a specific range:


texts: Identify significant features in clustering results

texts:
I'm a student in Data Analysis, working on a data clustering exercise.
Two clusters have been identified based on a dataset with 40 features. To interpret and label these clusters, I'm wondering if there is a way to determine which features are the most determinant in the clustering output. For instance, I would imagine that I could take out one feature from the clustering and see how much it affects the output. However there are probably smarter ways.
I would greatly appreciate if someone could point me in the right direction.
Thanks!


texts: Distribution of error values in linear regression vs logistic regression

texts: Why do error values in linear regression have to be normally distributed and why not in logistic regression?


texts: Is it normal that a classifier always wrongly predicts the same samples?

texts:
I'm trying to improve the accuracy of a classifier, a random forest one.
I built different models with the same hyperparameters but with different random seeds, trained them with the same training data, used the same test daata to make the predictions and compared the results. I discovered that 50% of the errors were always made on the same samples. Therefore, do these samples which are always wrongly predicted deserve a particular attention or is it kind of logic ?

I hope the question is clear enough.


texts: remove seasonality from weekly time series data

texts:
I need to decompose a series to remove seasonality. The series has 2 columns date and volume.

This is what my time series object looks like:

salestsDec <- ts(salests, frequency=52, start=c(2010, 1), end=c(2014,12))

I ran the decompose() function on a 'ts' object.

salests = sales[, c(1,6)]
View(salests)
salestsDec <- ts(salests, frequency=52, start=c(2010, 1), end=c(2014,12))
salestsDec <- decompose(salestsDec, type=c("additive"))
plot(salestsDec)

Upon, running the decompose() function, I get a list of 6 components, observed, trend, seasonal, random for both date and volume. I should only be seeing, observed, trend, seasonal and random component for Volume in my plot.

I've attached an image of what the plot looks like.

Moreover, when I try to remove seasonal component from the series, I am getting an error. It appears that it's the same underlying issue.

Error:

Error in salests - salestsDec$seasonal : non-numeric argument to binary operator In addition: Warning message: Incompatible methods ("Ops.data.frame", "Ops.ts") for "-"


texts: Appropriate visualization for this kind of data

texts:
Late edit: currently working through the FlameGraphs bibliography here. It looks like what I'm doing is similar to an icicle chart, but with passage of time on the x-axis. The author mentions inspirations for this FlameGraph that have this specific trait (time on x-axis), but I'll need to follow through to see if anyone implemented exactly the same thing.
I'm looking to visualize the flow of nested sub-tasks in my web service. The resulting visualization is probably going to look like a hybrid of a treemap, a dendrogram and a Gantt chart, and I've a feeling that there is a standard chart for it that I'm struggling to find.
Data
Each time a request is served, I log an object that contains a multi-level N-tree of everything the service did to serve the request, and how long it took.
For example, I log a start and end (relative to the request timeline) of the database call, as well as how long it took to connect, execute a query, and process the result into a model. The latter three operations are treated as sub-tasks for the database call.
This is what a simplified execution log would look like the listing below. Note that I have hundreds of operations, and some of them are 5-10 levels deep.
{
    "name": "total", // scope name
    "s": 100,        // start
    "e": 700,        // end
    "c": [           // scopes nested under this one
        {
            "name": "database",
            "s": 200,
            "e": 500,
            "c": [
                {
                    "name": "connect",
                    "s": 200,
                    "e": 300
                },
                {
                    "name": "query",
                    "s": 300,
                    "e": 500,
                    "c":[
                    {
                        "name": "subquery",
                        "s": 400,
                        "e": 470,
                    }]
                }]
                }
            ]
        },
        {
            "name": "ui",
            "s": 510,
            "e": 700,
            "c":[{
                "name": "render",
                "s": 520,
                "e": 690
            }]
        }
    ]
}

Current approach
I've written a small Haskell program to parse the tree and output the following chart onto an HTML Canvas:

At the moment, I can nicely represent the nested nature of the sub-tasks, and see where the tasks are relative to each other on the timeline.
I don't really want to build a chart renderer from scratch though.
Difference from chart types I've considered
Nested bar chart: bar charts usually have the bars starting at the 0, and all stacking usually happens based on bar size. I need more of a timeline view.
Gantt chart: very similar idea-wise, but usually tracks 'jobs' or 'projects' as rows, and normally doesn't deal with nested jobs.
Treemap: in treemap, all values get assigned the area based on their scalar value. Here, my area is made up of rows (number of sub-tasks) and width (task duration in milliseconds)
Dendrogram: close in terms of data structure, but I can't think of a way to render it in a way that helps to show the start/end/duration of nested tasks well.
Flame Chart/Flame Graph (late addition, see edit at the top): at last I've found someone with similar ideas! Looks like people working with performance profiler software have been working to visualize nested call stacks - in a very similar fashion, but sizing things by call count vs. duration.
The question
Am I reinventing the wheel?
I feel that there should exist a standard visualization type (probably well supported by common tools as well) that can render the chart above just as well.


texts: How to Combine Oracle and Snowflake/Redshift data on the fly

texts:
Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift.
Most obvious: I cannot take content of Oracle table to Snowflake/cloud. Restricted. Cannot use any service which can store Oracle data outside of it. Can use a transient service though.
Any thoughts on how to combine the data on the fly? Any sql which I write on Oracle will work on Snowflake as is. How I can combine the two SQL outputs on the fly. Want to write a single query to get data from two places.


texts: How to include lifecycle configuration in Sagemaker studio's user notebook

texts:
I want to use lifecycle configuration in Sagemaker studio so that on start of user's notebook it runs the given lifecycle configuration.
My lifecycle configuration will have shell script which will launch cronjob having python script to send attached notebook's running duration.
#!/bin/bash
set -e

# PARAMETERS
IDLE_TIME=120

echo "Fetching the autostop script"
aws s3 cp s3://testing-west2/duration-check.py .
aws s3 cp s3://testing-west2/on-start.sh .

echo "Starting the SageMaker autostop script in cron"
(crontab -l 2>/dev/null; echo "*/1 * * * * /bin/bash -c '/usr/bin/python3 $DIR/duration-check.py --time ${IDLE_TIME} | tee -a /home/ec2-user/SageMaker/auto-stop-idle.log'") | crontab -

echo "Changing cloudwatch configuration"
cat $DIR/on-start.sh | sudo bash -s auto-stop-idle /home/ec2-user/SageMaker/auto-stop-idle.log

Referred lifecycle configuration from here:
https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/tree/master/scripts/auto-stop-idle


texts: Do you have any real example of Data Science reports?

texts:
I recently found this use cases on Kaggle for Data Science and Data Analytics.

Data Science Use Cases

However, I am curious to find examples and case studies of real reports from other professionals in some of those use cases on the link. Including hypothesis, testing, reports, conclusions (and maybe also the datasets they have used)

Do you have any kind of those reports to share?

I am aware of the difficulty to share this kind of details and this is why it is not a Google-search-and-find answer.


texts: Linear combination of weak estimators over fuzzy classifiers?

texts:
Having:

a set of soft fuzzy classifiers (classification onto overlapping sets) $C_i(x) \to [0,1]$;
a corresponding set of weak estimators $R_i(z)$ of the form $R_i(z) = \mathit{EX}(y\mid z)$.

The estimators $R_i$ are just some kind of regression, Kalman or particle filters. The classifiers $C_i$ are fixed and static. How to make a strong estimator out of a weighted combination of the form:

$$L(x, z) = \sum_{i}C_i(x)R_i(z)Q_i$$

In other words how to choose the weights $Q_i$? Is there some kind of online approach to this problem?

Here is brief description of a practical application. When an event $E$ is registered, multiple measurements are made. Based on these measurements, the classifiers $C_i$ make a soft assignment of the event to multiple overlapping categories. What we get is fit ratios for the soft clusters.

Now there is some chance that event $E$ may trigger a subsequent event $D$, depending on another variable $z$ -- independent from the event $E$. We know that all the soft cluster "memberships" may influence the probability of event $D$ being triggered.

We want to estimate the probability that $E$ triggers $D$, given the $C_i$ fitness ratios and value of $z$.


texts: Powers of 2 for batch_size in model fit in deep learning

texts:
I am currently reading Deep Learning with Python by Francois Chollet, the author of Keras, and in one of his definitions for Mini-batch, he explains that the power of 2 for the batch_size is due to memory allocations in gpu/ Could anyone elaborate on this?

  Mini-batch or batch—A small set of samples (typically between 8 and
  128) that are processed simultaneously by the model. The number of
  samples is often a power of 2, to facilitate memory allocation on GPU.
  When training, a mini-batch is used to compute a single
  gradient-descent update applied to the weights of the model.


texts: What is the purpose of reward threshold in OpenAI Gym?

texts:
I've seen that OpenAI Gym environments can be registered with an optional reward threshold (reward_threshold) which represents:

  The reward threshold before the task is considered solved

How does this value affect the learning process? Or does one have to manually compare the reward obtained in each episode with the reward_threshold and stop the learning process if it surpasses it?


texts:
I have data on mchip, chip schip and mcaid that contains the following columns/features: year (between 2000 and 2016), state, age and month. I have a problem with several years. For example, I want to replace the numbers in 2003 (Jan, Feb, March) with numbers from 2002. I used this code:
states = df.state.unique()
for i in range(1, 4):
 for j in range(0, 20):
     for state in states:
         print('0')
         df.loc[(df['year']==2003) & (df['month']==i) & (df['age']==j)& (df['state']==state),'mchip'] = df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j) & (df['state']==state),'mchip']
         df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j)& (df['state']==state),'chip'] = df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j)& (df['state']==state),'chip']
         df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j)& (df['state']==state),'mcaid'] = df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j)& (df['state']==state),'mcaid']
         df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j)& (df['state']==state),'schip'] = df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j)& (df['state']==state),'schip']

The dataset is large and at the moment there is no way to validate whether the above churns out my desired output. I printed out the tails for 2002 and 2003 after running this code and they are different. Is there a way to do this? Thank you

I have data that looks like this. I want to replace values in 2003 with values in 2002. I had a mistake in the code, I already changed 2002 with 2003. After I ran this code values in 2003 are replaced with Nan. I printed out df.loc[(df['year']==2002) & (df['month']==i) & (df['age']==j) & (df['state']==state),'mchip'] and it returns an object instead of a value, this has to be the reason why values in 2003 are replaced with Nan. Is there a way to get a value from the pandas' data frame instead of an object? Thank you.


texts: Extracting encoded features after CatBoost

texts:
I have a dataset containing numerical as well as categorical variables.

After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables encoded in whatever method the classifier decided to encode them.

How can I extract the fully transformed (encoded) features? (similar to what a fit_transform method would return)


texts: Estimating Predictive Uncertainty for unlabeled data

texts:
I am trying to estimate the predictive uncertainty for a deep neural network. While I do have a labeled training set, I´m trying to measure uncertainty for some unlabeled production data.

This paper proposes the use of Deep Ensembles and Adversarial Training to compute a measurement of uncertainty. However, it uses Brier Score as a metric which requires me to know the real label of my production data.
Is there a similar way or metric which does not require labeled data?

Another approach was described by Yarin Gal utilizing Monte Carlo Dropout. However I can´t get any useful results using that technique on my model.

I want to use my model in an online learning task. As the given data may very over time, I need to detect examples where my model is highly uncertain, so I can manually classify those examples.


texts: Can someone please guide for implementing weight decay in transfer learning approach? I want to regularize the pre-trained model ResNet50, where I'm fine-tuning the model for an image classification problem with 22400 data points.


texts:
I am interested if someone can review this process and give me some tips. (I don't have any data science coworkers or friends to collaborate with...) When the script (.py file) below runs, it will output/print:

The total savings are 9793 kWh + - 192 kWh
The total savings are 9793 kWh + - 2.0 %

Ultimately I am attempting calculate "precision" with my results (+ - 192 kWh AND + - 2.0 %) ..

For starters, I cant find this info online so here's a snip from a book I am referencing on calculating data set standard error, then absolute precision, relative precision.

For retrieving my critical t value, I am following these steps from machinelearningmastery.com. This is where the scipy.stats import t package is required.

I cant remember where degrees of freedom comes into play. The machinelearningmastery example, the author uses df = 10 So until I can figure out this, I left that as a default value in my code below. If someone wanted to copy and paste the code below it should run...

I cant remember from college where degrees of freedom comes into play and how to utilize it. The length of the data when its converted into df2 Pandas dataframe is 31, which represents one months data 31 days. Any tips greatly appreciated..

import os
import numpy as np
import pandas as pd
import math
from scipy.stats import t

#actual electrical energy in kWh
actual = 40000

#calculated savings in kWh, output from Keras regression
data = [np.array([[1083.8748]], dtype='float32'), np.array([[998.98773]], dtype='float32'), np.array([[1137.0487]], dtype='float32'), np.array([[1077.2798]], dtype='float32'), np.array([[926.41284]], dtype='float32'),
np.array([[1030.7125]], dtype='float32'), np.array([[1028.0048]], dtype='float32'), np.array([[523.9799]], dtype='float32'), np.array([[1125.092]], dtype='float32'), np.array([[1119.7738]], dtype='float32'),
np.array([[918.6966]], dtype='float32'), np.array([[1112.5186]], dtype='float32'), np.array([[555.6942]], dtype='float32'), np.array([[1096.5643]], dtype='float32'), np.array([[826.35657]], dtype='float32'),
np.array([[1014.35406]], dtype='float32'), np.array([[1027.6962]], dtype='float32'), np.array([[924.20087]], dtype='float32'), np.array([[1035.217]], dtype='float32'), np.array([[1008.9658]], dtype='float32'),
np.array([[970.54047]], dtype='float32'), np.array([[847.0671]], dtype='float32'), np.array([[913.5519]], dtype='float32'), np.array([[1047.0747]], dtype='float32'), np.array([[920.0606]], dtype='float32'),
np.array([[994.2266]], dtype='float32'), np.array([[991.4501]], dtype='float32'), np.array([[972.43256]], dtype='float32'), np.array([[934.8802]], dtype='float32'), np.array([[912.04004]], dtype='float32'), np.array([[1131.297]], dtype='float32')]

#convert data to pandas series then DataFrame
df = pd.Series(data)
df2 = pd.DataFrame(df)

#define sum, standard deviation, mean of calculated savings data
total = df2.sum()
totalStd = np.std(df2.values)
totalMean = df2.mean()

#calculate saings
diff = int(actual - total)

#compute precision of results, starting with standard error
stdErr = totalStd / math.sqrt(len(total.index))

# define probability & degrees of freedom
p = 0.90
degf = 10

# retrieve t value <= probability
t = t.ppf(p, degf)

#calculate absolute precision
absPrec = t * stdErr

#calculate relative precision
relPrec = (100 * (absPrec / diff)).round(decimals=1)

#print results
absSavings = f'The total savings are {int(diff)} kWh + - {int(absPrec)} kWh'
print(absSavings)

relSavings = f'The total savings are {int(diff)} kWh + - {float(relPrec)} %'
print(relSavings)


texts: What pre-processing of the image is needed before feeding it into the convolutional neural network?

texts:
I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning process of a neural network occurs on 100 by 100 px (https://www.kaggle.com/smeschke/four-shapes/data). I'm having a little difficulty wrapping my head around the situation when the circle in the input image is much larger (or smaller) than 100x100 px. How then the convolution neural network determines that circle if it was learned on a dataset of a different picture's size.

For clarity, I want to submit a 454 by 430 px image to the network input:

Example of the dataset for the learning process (100 by 100 px):

Finally, I want to recognize all the circles on the input image:


texts: What is the name of the sub-field within spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) Models?

texts: What is the name of the specialism / sub-field in spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) models?, ... that is, excluding GIS and BIM systems. One such example being when applied to the analysis of 3D models of machine parts created by CAD systems in Engineering, where Data Analytics and Data Visualisation may be applied to data arising from that analysis.


texts: Keras - understanding ImageDataGenerator dimensions

texts:
I'm trying to implement custom object detection by taking a trained YOLOv2 model in Keras, removing the last layer and retraining it with new data. I'm confused about how to feed the data to Keras, though. I have annotated a bunch of pictures with bounding boxes using the YOLO annotation, and put them in two separate folders (images where the .jpgs reside and annots where the .txt annotations are).

I also removed the last layer from the model and added a custom one (I'm trying to predict bounding boxes for 2 classes).

I'm trying to pass my data with an ImageDataGenerator, as my dataset is quite small.

I have the following input objects:

np.shape(train_images) # this contains RGB data from 79 pictures
(79, 1, 608, 608, 3)
np.shape(train_y)
(79,)

I'm trying to pass these to the ImageDataGenerator, but I get an error:

train_datagen = ImageDataGenerator(
      rotation_range=20,
      width_shift_range=0.2,
      height_shift_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

train_generator = train_datagen.flow(
        train_images,
        train_y)

ValueError: `x` (images tensor) and `y` (labels) should have the same length. Found: x.shape = (1, 608, 608, 3), y.shape = (79,)

I don't understand what the problem is. Somehow the first dimension of my images data is completely gone and thus does not match... What's wrong with it?


texts:
I have a question regarding exercise 14.17 in An Introduction to Information Retrieval by Manning et al.

The problem is:

  "Assuming two classes, show that the percentage of non-separable
  assignments of the vertices of a hypercube decreases with
  dimensionality M for M > 1. For example, for M = 1 the proportion of
  non-separable assignments is 0, for M = 2, it is 2/16. Solve the
  exercise either analytically or by simulation."

The total number of assignments of vertices of an N-dimensional hypercube is: $2^{(2^N)}$

And as I found in here the number of separable assignments is O($2^{(N^2)}$).
So the percentage of non-separable assignments is increasing with N (which is the opposite of what is said in the exercise).

What am I missing here?


texts: What algorithm to use for a specific 'Named Entity Recognition'/'Information extraction' problem

texts:
I am trying to write a model that will extract certain details from financial documents.

It must be able to extract the; contract start date, contract duration, contract value and all named entities. Preferably it would also be able to categorise the named entities.

I have roughly 600 tagged documents that I can use for training and testing purposes.

What model should I use?

I have had a look at different named entity recognition models, such as the Skip-Gram model. However these models do not utilise the fact that each one of these documents will contain all of this information. To illustrate this, the Skip-Gram model is often implemented in such a way that only the words in the same sentence contribute to its semantic vector.


texts: Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?

texts:
I'm trying to find a definitive way to conclude the R^2 score from a prediction accuracy point of view rather than variance. How should I do it?

Conceptually, most blogs / articles explain R^2 as:

The variation in the data points that can be explained by the model.
A measure of how closely fitted the data points are to the regression line.

I find these answers to be unsatisfactory.

If the R^2 score is 0.7, is it right for me to say that 70% of the data has been predicted accurately by the model? Since most explanations are tied to the explanation of the variance I'm thinking that looking at it from an 'accuracy' stand point of view is wrong. It implies that if I predicted 10 random points on the x-axis I would accurately guess 7 of its actual values.

According to scikit-learn's documentation page at the 'score' section:

The coefficient R^2 is defined as (1 - u/v), where u is the residual sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum()

Apart from scikit-learn's way of calculation, another way of calculation is simply the squared of R (coefficient of correlation).

Which way is right? Are both the same? Or are they related in some way?

Assuming the right way of calculation is based off scikit-learn's documentation where the R^2 score is the variance from the predicted mean divided by the variance from the actual mean, is it even possible that the variance from the predicted mean is smaller than the variance from the actual mean?

How can the variance from the prediction model which is built upon another dataset (let's say dataset A) be smaller than the actual variance of dataset B?

Deeply appreciate any thoughts, comments or clarification.


texts: how does splitting occur at a node in a decision-tree with non-categorical data?

texts:
According to a website (:http://dataaspirant.com/2017/01/30/how-decision-tree-algorithm-works/) , these values are chosen randomly:

I don't think this is the case with any optimized way of creating a decision tree. In this image(different example) the value is 2.45 for the root node:

Was this value chosen randomly like explained in the website? If not and the value is not chosen randomly then how is it calculated?


texts: Decision tree classifier: possible overfitting

texts:
I have a dataset with following specifications:

Training dataset with 52968 samples with 8562 positives
Test Dataset with 13242 samples with 2135 positives
There are 137 features

I want to perform a binary classification. I create DecisionTreeClassificator in pipeline:

imp = Imputer(strategy="most_frequent", axis=0)
var_thr = VarianceThreshold(threshold=1.7)
pca = RandomizedPCA(n_components=16)
clf = DecisionTreeClassifier(max_features=0.86, max_depth=42)

return Pipeline(steps=[('imp', imp),
                       ('var_thr', var_thr),
                       ('pca', pca),
                       ('clf', clf)
])

I also tried to increase training data with positive results:

series = y_train[y_train==1]
dupli = x_train.loc[series.index.tolist(), :]
for _ in range(5):
    x_train = x_train.append(dupli)
    y_train = y_train.append(series)

return x_train, y_train

After fitting my model, the score result for my test data is 0.9954, and the cross validation is:

cross_val_score(clf, x_train, y_train, cv=5)
[ 0.90225866  0.90638078  0.90592215  0.90007453  0.90632345]

Classification report for training data is perfect:

             precision    recall  f1-score   support

          0       1.00      1.00      1.00     44406
          1       1.00      1.00      1.00     42810

avg / total       1.00      1.00      1.00     87216

The confusion matrix is:

[[44203   203]
 [  190 42620]]

but the test data is much worse:

             precision    recall  f1-score   support

          0       0.85      0.85      0.85     11107
          1       0.21      0.21      0.21      2135

avg / total       0.75      0.75      0.75     13242

Confusion matrix is:

[[9428 1679]
 [1687  448]]

I used GridSearchCV for the threshold, n_components, max_features and max_depth.
How can I improve my model and obtain better prediction?

EDIT ------->
I changed clf in pipeline. I used RandomForestClassifier.

clf = RandomForestClassifier(
    n_estimators=500, n_jobs=-1, max_features=0.5, max_depth=15,
    random_state=1
)

Now cross validation is

[ 0.81552396  0.81218827  0.82331021  0.81488276  0.81769191]

Classification report for training data with confusion matrix:

score train result: 0.8514148780040359
             precision    recall  f1-score   support

          0       0.86      0.85      0.85     44406
          1       0.85      0.85      0.85     42810

avg / total       0.85      0.85      0.85     87216

[[37757  6649]
 [ 6310 36500]]

Classification report for test data with confusion matrix:

score test result: 0.7341791270200876
             precision    recall  f1-score   support

          0       0.89      0.79      0.83     11107
          1       0.30      0.47      0.36      2135

avg / total       0.79      0.73      0.76     13242

[[8719 2388]
 [1132 1003]]

It looks better but I search model with + 0.90 recall for traning and test data sets.


texts: Is there a comprehensive learning plan for a self-teaching beginner in data science with no programming background?

texts:
Situation

I want to be a data scientist, but I have no programming or maths background.
I have more than the basic understanding of python which I taught myself over a couple of months. Right now I am stranded, because I do not know how to frame a study plan?

I am working in a marketing firm with lots of data, so I have an income for the time that I study and I have data to apply my knew knowledge on. I am not trying to rush up and want to build up a strong base and grow upon it over a longer period of time. I know it is not going to be easy, but I feel strongly motivated.

Questions

Could you point out comprehensive learning plans with references to books, courses, and learning materials for self-teaching beginners? Could you also point out any articles or presentations in which former beginners and transitioners share their experiences about successfully becoming data scientists?


texts: Is it possible to call from Keras unsupported backend function directly from tensorflow?

texts:
I am working to create a Keras custom layer. this layer needs to use a function that doesn't exist in Keras TF backend, although it exist in TF.

What would be the best way to solve it?

Is there a way to call TF function directly from Keras and not from backend?
Should I work to modify backend to include a new unsupported function?

Any example or pointer how to implement the best solution.

Thank you


texts:
I am using Python 3.6 and Tensorflow 2.0 for the following code for linear regression:
tx = tf.constant(x, dtype=tf.float32)
ty = tf.constant(y, dtype=tf.float32)

tw = tf.Variable(initial_value=np.random.randn(), dtype=tf.float32, trainable=True)
tb = tf.Variable(initial_value=0, dtype=tf.float32, trainable=True)
tyhat = tx * tw  + tb

cost = lambda: tf.reduce_mean(tf.square(tyhat-ty))
optimizer = tf.optimizers.SGD(0.01)
train = optimizer.minimize(cost, var_list = [tw,tb])

and got an error: No gradients provided for any variable: ['Variable:0', 'Variable:0']
Any help?


texts: How to select regression algorithm for noisy (scattered) data?

texts:
I am going to do regression analysis with multiple variables. In my data I have n = 23 features and m = 13000 training examples. Here is the plot of my training data (area of houses against price):

There are 13000 training examples on the plot. As you can see it is relatively noisy data. My question is which regression algorithm is more appropriate and reasonable to use in my case. I mean is it more logical to use simple linear regression or some nonlinear regression algorithm.

To be more clear I provide some examples.
Here is some unrelated example of linear regression fit:

And some unrelated example of nonlinear regression fit:

And now I provide some hypothetic regression lines for my data:

AFAIK primitive linear regression for my data will generate very high error cost because it is very noisy and scattered data. On the other hand, there is no apparent nonlinear pattern (for example sinusoidal). What regression algorithm will be more reasonable to use in my case (house prices data) in order to get more or less appropriate houses' price prediction and why this algorithm (linear or nonlinear) is more reasonable?


texts: How can I perform stratified sampling for multi-label multi-class classification?

texts:
I am asking this question for few reasons:

The dataset in hand is imbalanced
I used below code

x = dataset[['Message']]
y = dataset[['Label1', 'Label2']]
train_data, test_data = train_test_split(x, test_size = 0.1, stratify=y, random_state = 73)

but the error message that I am getting is The least populated class in y has only 1 member, which is too few. The minimum number of labels for any class cannot be less than 2. I removed those classes where the class count is < 2 in each individual label.

I am not sure why this error is popping.

So, I am thinking to go to implement the stratified sampling myself. Here,  I need help in deciphering the cause of the problem here and the implementation of the stratified sampling in multi-label classification so that it works well for the individual batches too while training.


texts:
What are the points called in a dataset where the points have the same features but different labels?

For example, I am trying to predict whether some object is a hot dog. But I only have access to features is_wrapped_in_bread and has_meat_in_the_middle.

| object        | is_wrapped_in_bread | has_meat_in_the_middle | is_hotdog |
| salad         | false               | false                  | false     |
| hotdog        | true                | true                   | true      |
| burger        | true                | true                   | false     |
| veggie burger | true                | false                  | false     |

In the example above, burger and hotdog are not distinguishable based on the available features. Is there a name for that?


texts:
I'm studying the knn classification algorithm.
Why can the euclidean distance be considered a nice measure of affinity between examples ?

In one dimension (1 attribute) this seems correct, but if I add dimensions, can the euclidean distance still be considerd a good measure of affinity? Why?


texts: How the embedding layer is trained in Keras Embedding layer

texts:
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.


texts: Generating labeled dataset for training a neural network

texts:
I am currently working on a project in which i'm supposed to classify whether an image contains a translucent watermark or not. This is hard to do with standard object classification or template matching because of the translucency and very low image quality.

I thought about generating a dataset with the watermark pasted somewhere on the image. Given enough generated data, could I train a neural network to classify the real images? If not, why not? Thank you for your help.


texts: Pandas, R, and interoperable data frame formats

texts:
I want to have a single format which is compatible both with R and Python's Pandas (and preferably, other things like Stata and SAS), for which I will not ever need to impute whether things are numbers or dates.

Does such a thing exist? Or has every package got their own system (Pickle, RDS, Stata .dta, .sasb7dat) for the purpose of lock-in?


texts:
An example time series would be the stock market, which is sometimes described as a random walk.  Over time, this is clearly not the case as it has essentially gone in one direction (up) with only occasional setbacks.

I'm coming up with several characteristics that could be used to measure orderliness of a specific segment of a time series:

Net movement from start to finish
Linearity of movement
Length of time

In short, the larger the absolute value of the movement relative to the standard deviation of the series over similar lengths of time and the more linear the move, the more significant that segment.  These characteristics could also form a kind of signature of each given segment.

Maybe the answer to this is simply that a time series whose segments don't fit a normal distribution is a non-random time series, but I'm wondering if there are methods that capture the significance of a time segment relative to #1-3 above, i.e., "how rare is this segment"?


texts: Would you recommend feature normalization when using boosting trees?

texts: For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using regularization. However, in my experience, boosting tree regression works less well when I use normalized features, for some strange reason. How is your experience using feature normalization with boosted trees does it in general improve our models?


texts:
I'm quite new to Theano and was reading the deep learning documentation here :

deep learning Theano

The function called accumulator with "inc" and state was explained really unclear, that I couldn't figure it out what happens that while the input is just one scalar, but you have both state and inc that have to be defined in the function definition. What happens every time you introduce a new number to the function?!!!


texts:
I want to use InceptionV3 on one class of image (ie. to detect one type of object/one label only). For example, to detect 'penguin' or 'no penguin'. All my images have been cropped down to mostly display the object, so there is barely any background present and the object is in clear view. I have used bottlenecking; I have frozen the top layer of Inception so that I can retrain Inception to classify my object. However, this means that I have no alternative class to train with since I only want to classify one object.
The issue is that val_accuracy is always equal to 1, implying a perfect model, which my better nature says is too good to be true. Therefore, I believe the problem is that I have only provided the model with 1 class. How can I overcome this?
Finally, I have tested with InceptionV3 and an out-of-the-box model just to check that my Inception set up was not the issue. Both yielded varying training accuracy, but again, the basic model produced val_accuracy equal to 1 every time. I have also experimented with train-test splitting such as 10% to train with and 90% for validation. To my knowledge, a benefit of transfer learning is that you can use small datasets to train with. Even so, my dataset is 3300.


texts: is padding input images better than resizing image?

texts:
I am working with images and was thinking of pre-training VGG19 and EffecientNETB0 model on my dataset. However, a query:-
Is padding the images then using EfficientNET better than having dynamic shapes with VGG19?
My data is of a very weird shape (it's a panorama) so that's why I was confused as to whether I should use VGG19 or EfficientNET because EN would require heavy padding and cropping while VGG won't.

In short, is it more beneficial to pad images and use them, or does it yield more performance if I have a model that can accept data of any dimension?

Is there some theoretical explanation for what I should use?


texts: Reinforcement learning - How to deal with varying number of actions which do number approximation

texts:
I am a new to Reinforcement learning, but I am trying to use RL in this task:

Given a function definition in written e.g. in C with 1 to 10s of input arguments (only numerical ones - integer, float, etc.) and the body of the function (represented as a Abstract Syntax Tree/ Abstract Decision Tree with data dependencies - how the internal variable values change) I would like to approximate the values of these input parameters so for e.g. a certain decision block is executed. For this I thought of a recurrent network with LSTM cells.

Now, to achieve this, I would traverse one path in the tree leading to the block and take note of any data changes and decision blocks in the path. These steps would influence my parameter input predictions - what values to insert into/change in the input parameters if I wish to have a certain decision block executed.

Action: Changing the value of one chosen input parameter of the function OR Changing the value of all input parameters individually (with mathematical different operation). After action execution, moving onto the next node in the tree.

Reward: How close I am to executing the given decision block (thus satisfying the condition) with given input parameter values.

Goal: Have a condition in the code satisfied and a decision block thus executed (e.g. an if condition is met).

State: Current position in the AST/ADT with data dependencies.

Assuming that I already have a way to evaluate, how far I am from executing the wanted decision block given current parameter input values, I came across two problems:

How would I deal with varying number of function input parameters in RL? If I want to change their values to be closer to the execution of the wanted decision block, the number of given actions changes with the number of parameters defined for the given function.
If I already did chose one parameter, what is the best way to do number approximation using RL? In the function body there could be numerous very complex mathematical operations happening, so should there be defined action as logarithm, exponentiation, division, multiplying, etc. or is there a better way with maybe just adding/subtracting from the current value?


texts: Can I use confident predictions to correct incorrect labels?

texts:
From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

When I train on the corrected labels for the corrected test set I get:

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.

EDIT EDIT:
After experimenting for a while I've reached the following conclusion:

It seems that predictions can be used for label corrections, but a different estimator should be used to try and determine which labels are incorrect. When I used the same estimator, despite most of the label corrections being valid (I determined 92-93% based on visual inspection of a 500 sample subset), it nevertheless caused the new prediction estimates to be biased. The new estimates were overly confident (drastically tended towards zero and one). This is either due to the correction, or it may be due to too little noise in the dataset (I considered the possibility that the noise is actually helping the estimator not to overfit. Neural networks have been found to be poorly calibrated, the author of this article suggests that overestimation may in fact be a form of overfitting).


texts: Sklearn Random Forest Prediction Correlation Issue

texts:
I'm having issues with fitting a Random Forest model to a completely new dataset. I'm trying to predict tenancy lengths for current tenants. I have a dataset with tenancy information since 2008, with both tenants that have finished their contract and others which are still in their properties.

Steps undertaken so far:

1) Remove the tenancies that are still on going. This is to actually get target values for "Tenancy Length".

2) Find a suitable model to train and test on the finished tenancies. I'm happy with a Random Forest, with MAE of ~35 on train and ~70 on test.

3) Create a new dataframe in python that only has the previously excluded information (tenancies that have not yet finished) in order to apply the .predict method on these. I've also combined the previous 'train' and 'test' into a single train, as there is no further need to split that data up.

df = finished tenancies;
currentDf = ongoing tenancies

#Select the features and target variable from finished tenancies data
X, y = df.iloc[:, 5:].values, df.iloc[:, 4].values
clf = RandomForestRegressor() #with relevant params
clf.fit(X, y)
yPred= clf.predict(X)

This results in a 'new' training set consisting of all the finished tenancies.

4) I move on and predict my current tenancies:

X2 = currentDF.iloc[:, 5:].values
y2Pred = clf.predict(X2)

While I cannot create a residual plot for this (as I have no previously known information regarding targets), I've noticed that 'Tenancy Length' strongly correlates with 'Tenancy Start', which was not the case in my trained model.

Does anyone have any idea what's going on? Why would something like this be happening? I can't for the life of me figure out if I'm doing something wrong, yet I doubt it's just coincidence for some reason.


texts:
I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it.
What is the right tech/algorithm/Python framework to approach this?


texts: Understanding regularisation and a preference for small weights

texts:
This has been a spot of confusion for me lately upon reading more into regularisation in neural networks.

I have always viewed weights as being a measure of the importance of a feature in a model. E.g. A model of ice cream sales starts off with random weights and eventually learns to increase the weights connected to the temperature feature.

With regularisation (L2, I have been looking at most recently), small weights are favoured more by the model. This makes good sense to me when we are trying to prevent overfitting in our models. The model generalises well between different pieces of input data because the smaller weights give their associated feature less of an impact in the model. This will draw out the true patterns of the model as opposed to a few cases that may skew the weights.

Now my confusion is what if a feature like temperature is a consistent predictor of ice cream sales in our model, what does its associated weights look like after regularisation? Is their impact on the cost function so great that the model doesn't prefer smaller weights instead? Also how do folks usually set a value for lambda?


texts: I have a 20.2 GB dataset which is entirely consisting of 1 file. Need help on how to open the file

texts:
Whenever I am dealing with datasets that have weird extensions that I have not encountered before, I typically open them with notepad to see how the data looks like before using pandas to analyze them. But this dataset (link), that has an extension of .tr (stands for trace files I suppose), is not supported by notepad due to its huge size (that's what the error message says).

Which software should I use to open the trace file? Also is there a safe way to segment the file into smaller parts, as I typically upload my data to kaggle and work on their kernel.

I am not sure if this is the right place to post, if not please point me towards the correct place before removing the post.


texts: Suggestions for predicting the roots of a polynomial, and auto encoders

texts:
I am trying to improve a model for predicting the roots of a polynomial. The data set consists of integers arrays that represents the coefficients of a polynomial and the roots.Inputs are zeropadded for which polynomials of lower than the highest degree. I could not come up with another idea yet.
An example data set is as follows
[[       0.          0.          0.          1.         81.78     1597.3 ]
[       0.          0.          1.         -0.64     -446.88    -3372.03]
[       0.          1.        -29.88      -28.05     2700.11     1057.81]
[       1.          4.24    -2807.77   -22044.78  1498373.97 10298202.12]]

[[[-49.53   0.  ]
[-32.25   0.  ]
[  0.     1.  ]
[  0.     1.  ]
[  0.     1.  ]]

[[-14.1    0.  ]
[ -9.77   0.  ]
[ 24.5    0.  ]
[  0.     1.  ]
[  0.     1.  ]]

[[ -8.56   0.  ]
[ -0.39   0.  ]
[ 11.62   0.  ]
[ 27.21   0.  ]
[  0.     1.  ]]

[[-41.62   0.  ]
[-29.62   0.  ]
[ -6.78   0.  ]
[ 25.57   0.  ]
[ 48.21   0.  ]]]

I have tried with both mean absolute error, mean squared error, mean absolute percentage error(which I believed, it should give the best results but didnt). No matter how many polynomials I put in the dataset, it seems like it does not learn from it at all. Even with very low mean absolute errors, around 4, the predictions are way off, almost as random even for second/third degree polynomials.
And the model
    h_size = 128
    model = Sequential()
    model.add(LSTM(h_size , input_shape=(None, 1)))
    model.add(RepeatVector(degree))
    model.add((LSTM(h_size , return_sequences=True)))
    model.add(TimeDistributed(Dense(2)))

    model.compile(loss='mean_absolute_error',
              optimizer='adam',
              metrics=['mae'])

And more thing, what kind of an auto encoder should I use?


texts: Is there a NN-Model which has multiple outputs?

texts:
This is what I am trying to accomplish:
Predict Y based on X,that is Y ~ X:
1.X = {n, x_1, x_2, x_3, ...}

a vector of factors
size unchanged: X.size() = DIM_X

2.Y = {}

also a vector
size changing according to n in X: Y.size() = 4 * n

So, can I do this with NNs ? is there a NN-Model I can refer to ? Or Is there other approaches can solve this ？
Edit:
Sample data:
1 0.1 109 123.7 0 0 512 512
2 0.8 201 88.7 384 216 384 246 0 0 1024 768

In the above, first 4 elements each row represents X, what's left is Y.
Appreciate your help.


texts: How do I know that model performance improvement is significant?

texts:
Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one convolutional layer) and run the model again, leading to an accuracy of .8+x.
My question is how I would determine which in-/decrease in performance allows me to say that the new network architecture is better than my old one? I assume that x=.0001 falls within a reasonable margin of error, whereas x=-.2 is a significant decrease in performance - however, the use of "significant" here would be purely colloquial without any scientific backing.
I understand that some kind of hypothesis testing would in theory be appropriate here, but as far as I know, these require multiple samples (i.e. running the network many times), which in case of large ML models which take sometimes days to train isn't really feasible.


texts: I am working on a pose dataset and want to convert this data into CSV format. How to convert this data into a single row of data

texts:
This is the data for one pose. I have a lot of data like this which i want to convert to CSV to apply machine learning algorithm.
I am assuming I have to convert this data into a single row of comma separated values which will define a single pose.
Any suggestions will be appreciated.


texts: Reducing (Variance) | the gap between my weights

texts:
I have ML ready samples. And each sample has a weight.
The weights distribute between [0-1]
My problem arise because there are a lot of samples which are 0.001, 0.00x
And a lot of samples which are 0.997, 0.99x
I am going to sample data based on these weights. And samples with 0.99x will overshadow the other samples in the data set while 0.00x samples will have 0 significance.
The solution I am looking for, is some kind of function over those weights that will balance them a little bit / reduce those huge gaps (Therefor reducing the variance) AND still preserve their order
So if 0.997 turned into 0.88, 0.996 will turn into something < 0.88

For example:
in [0.01, 0.1, 0.4, 0.6, 0.8, 0.997]
I would want something like:
[0.15, 0.2, 0.42, 0.6, 0.78, 0.9] (Just an example)


texts:
There is a text summarization project called SUMMARIST. Apparently it is able to perform abstractive text summarization. I want to give it a try but unfortunately the demo links on the website do not work. Does anybody have any information regarding this? How can I test this tool?

http://www.isi.edu/natural-language/projects/SUMMARIST.html

Regards,
PasMod


texts: Why isn't the neural network updated after every example in the dataset

texts: Why the neural network is updating only after the whole batch passes?


texts: Figuring out what's wrong with the box plot. Outliers?

texts:
What a 'box plot' of this kind has to say?
that basically I have a lot of outliers and I should focus on data in proximity of zero?


texts: Need help in choosing units. Data processing or big data management

texts:
I'm having dilemma of choosing between the following 2 units for my next semester. As I can only do 1 unit out of these 2, which one is more useful in terms of usefulness and industry application?

1. Data processing. It covers the following topics: Scala programming,
Apache spark and graph processing, data streaming algorithms and methods.

2. Big data management. It covers the following topics: NoSQL,
parallel data processing/distributed databases, MapReduce and Hadoop Framework,
Streaming data processing

Thank your for your attention.

I'm looking forward to replies from experienced individuals.


texts: machine learning mode are suitable for using word features as x, and y as click rate of text

texts: Hi I want to extract words of english texts using click rate with machine learning model. Now I know the click rate of text, and I know How to extract words(unigram) of each text, for example, there are about 10000 texts,and the click rate of each text is provided. How to extract words features for click rate. How to extract key words and compute the importance of each word for computing click rate.


texts: How to get nearest 5 points mean with Nearest Neighbours?

texts:
In a dataset of longitude, latitude and price (of houses) I'm using sklearn's KNearestRegressor to get the 5 nearest neighbors mean price for each point. The problem is I want to do this for the whole dataset and each point is taking its own price into consideration since KNearestRegressor is a supervised algorithm, and I'm predicting the training set. How can I get the 5 nearest neighbors mean price for each point, not taking into consideration that point's price?

I have tried sklearn's unsupervised NearestNeighbours but it only outputs the indexes of the nearest neighbors, not the mean price.


texts: Opensource Speech Recognition Library that is secure and trained on large data

texts:
For all those who are working on developing a chatbot/assistant and care about the privacy of users consuming the speech recognition library, can you suggest an open souce library which is trained on a large data. Big concern is the privacy that's why not going for Google Diagflow or IBM Watson or Amazon Lexa or Wit.

Would appreciate a lot if someone can suggest a good library.


texts: Web Scraping: Multiple small files or one large file?

texts:
I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each scrape, so that pre-processing (via Python) later would be as simple as possible. My options are the following:

Dump data from each scrape into a fresh file (timestamp as filename). Resulting in 12,960 files of approx. 150kb each OR
Maintain 1 single large file, whereby the scraper simply appends the new output to the end of the file. Resulting in 1 file of size approx 1.9Gb after 3 months

Does anyone have any recommendations or warnings from their experience about either approach and how this affected pre-processing? I am cautions that having a pre-processing script work on a larger file might take longer to process but then again, opening and closing thousands of files will also be time-consuming.


texts: Pruning and parameter reduction for decision trees

texts:
I am trying to perform a classification using a decision tree classifier. I was wondering whether using a Feature reduction method is relevant for decision trees since they automatically use pruning?

My idea would be to perform a loop from 5 to 15 parameter reduction and then compare the classification accuracy of each decision tree, and then conclude the optimal number of parameters for my classification.

Thank you.


texts: Should I expect major performance improvements by scaling my features?

texts:
I'm trying to decide whether I should scale my features & responses for training, and I'm in a situation where I can't just try both scaling and not scaling.

My features currently have an std around 0.05, and the behavior of the timeseries I'm studying is very much scale dependent (0.5 means a VERY different thing than a 0.05 in terms of what's happening in the market).

Can I expect major improvements in performance, optimizer-wise, if I scaled my features to have std of 1?

My models are different combinations of MLP and 1d conv, and my algorithm is gradient descent with the Adam optimizer.

Thank you! [and yes; I'm another one of those people who are trying to forecast the stock market]

Louis is right that I should try both. But I'm leaving the question up just in case someone comes along and gives a mathematical proof of exactly normalization is useful /useless.


texts: Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

texts:
I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes probabilities?

Let's say our model is two trees. A terminal node of tree 1 has example 14 in a node with 20% class 1, 60% class 2, and 20% class 3. A terminal node of tree 2 has example 14 in a node with 100% class 2. Then our prediction for training example 14 is [10%, 80%, 10%].

Why use Softmax instead of this averaging approach?

Note: I am looking to apply this knowledge to understanding xgboost better, as well as a simple 1 tree classification model.


texts: difference betwen predicting seen and unseen data

texts:
I tried to test my model with seen and unseen data (seen data are data that i used to learn the model). I figure out that as much as i increase the number of features seen data can be properly predicted, while when using a feature selection technique unseen data  can be properly predicted. Is there any explanation about this.

Thanks in advance