texts:
I have basic understanding of how to perform linear regression with sklearn and statsmodels. There are several questions that I would like to ask regarding Linear Regression (OLS estimator) :

Is standardization always necessary for OLS estimator? I am still learning a bit of theoretical aspect of regression analysis. Standardization is mentioned in my data science class but almost None throughout my study in regression analysis. If no, when do I apply standardization (also do I need to standardize the predicted variable)?
I tried comparing both, in terms of evaluation metrics(mse) the results are relatively similar, but in terms of coefficient it is very different (due to different scales of predictors). If this is the case how to compare the significance of a predictor?
I tried comparing the result with statsmodel OLS on unstandardized data. Coefficient result is different by huge margin and on top of that mean squared error is worse with statsmodel than with sklearn model. However, sklearn model is less informative in terms of statistical information, for example if I want to remove a variable based on p-value from regression result. Is this supposed to happen(different result)?


texts: How to measure/rate the effect of a exogenous covariate in a ARIMAX Model?

texts: I have an ARIMA model, I'm trying to figure out how much an external variable (exogenous covariate) could improve the forecast, so I need to "synthesize" a rate that tell me the usefulness (or impact) of a new external regressor. Does anyone know a good way to do it?


texts: What causes the Shape (convex or non-convex) of the cost function?

texts:
There are different types of cost functions like Cross-entropy, absolute error, mean squared error.

When can we expect the cost function to be non-convex? Does this depend upon the type of cost function we choose? Or does this depend upon the Model we choose like when we have higher order polynomial Model?


texts:
I am trying to develop a custom sentiment analysis model on Google AutoML. I have achieved the same, however, I have the following query:
Scenario1: I have excluded records from my training data where Classic Language API is already giving accurate sentiment labels and I have trained the AutoML model only on fallouts.
F1-Score in this scenario pretty less when tested on benchmarking data
Secnario2: I have trained the AutoML model on the whole of training data.
F1-Score in this scenario has shown significant improvement than Classic Language API.
Hence,wanted to check with you all on what could have wrong on AutoML when I trained the model only on the fallout from Classic Language API


texts: Is there a general guideline for experience replay size, and how to store?

texts:
I am trying to use deep Q-learning on color images of size (224 x 224 x 3). I have read that implementations of DQN use experience replay sizes around 50,000. If my calculation is correct, this is over 56 Gigabytes for my data (two images per tuple, one image for state and next_state, totaling 100,000 images). Is this correct?

If yes, how can I go about this without that much RAM?


texts: Can't get Keras functional model to work: "could not broadcast input array..."

texts:
For a really trivial MWE, here is some code that I expected to work and does not:

import tensorflow.keras as keras
import numpy as np
input_ = keras.layers.Input((256, 768))
layer = keras.layers.Lambda(lambda x: x[0, :], output_shape=(768,),
                            input_shape=(256, 768))
model = keras.models.Model(inputs = [input_], outputs=layer(input_))
model.predict(np.zeros((1,256,768)))

I expect this to return np.zeros((1, 768)), but instead it throws the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/holmes5/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1078, in predict
callbacks=callbacks)
  File "/Users/holmes5/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 370, in
model_iteration
aggregator.aggregate(batch_outs, batch_start, batch_end)
  File "/Users/holmes5/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_utils.py", line 169, in aggregate
self.results[i][batch_start:batch_end] = batch_out
ValueError: could not broadcast input array from shape (256,768) into shape (1,768)

What am I doing wrong?


texts: Load large .jsons file into Pandas dataframe

texts:
I'm trying to load a large jsons-file (2.5 GB) into a Pandas dataframe. Due to the large size of the file  pandas.read_json() will result in a memory error.

Therefore I'm trying to read it in like this:

S_DIR = r'path-to-directory'

with open(os.path.join(S_DIR, 'file.jsons')) as json_file:
    data = json_file.readlines()
    data = list(map(json.loads, data))

df = pd.DataFrame(data)

However, this just keeps running, slowing/crashing my pc.

What would be the most efficient way to do this?

The final aim is to have a subset (sample) of this large file.jsons dataset.

Thanks


texts: How far from a specific state in a Markov chain

texts:
Problem:
Lets say we have an irreducible Markov chain.
Given a failure state or non desirable state F and a current state S
Is it possible to find how far we are from the failure state F.

Example: if we try to model an engine failure using Markov process and using historic data we have a state transition matrix P, one of which is a state where the engine was failed .
Now , in a dynamic system, if the current state S and a state transition matrix are given, can we calculate how far we are from failure state F.

Mathematically:

x(n+1) = (x)P^n

Right hand side of the equation gives us a vector v which consists of probabilities of transitioning to different states at stage n.
With different value of n, the values in this vector v will change
one of the values in v will be the probability of going to state F.
I want to find for what n , transitioning probability to F is maximum.


texts: Model for balancing 2 sport teams with multiple players?

texts:
I need some guidance on what to start researching for this. I have 10 players and I need to auto-generate 2 teams. Every time a match is finished, I need to re-balance the teams. For that, I have two features:

Win/loss ratio.
Players that have better performance playing together.

Any ideas on which model can I use to get this automatic balance running?

Thanks!


texts:
As the title suggests, I have a question regarding the trees produced through the bagging procedure.

Namely, since the bootstrap samples created to fit trees on are independent and identically distributed (iid), are the resulting trees also iid?

In other words, is there any reason there may be correlation between the "bootstrap trees"?


texts: Loss is decreasing correctly but upon prediction, totally wrong results

texts:
I've made a pretty basic stock prediction RNN with it's only input being the past stock price from apple. On this case, I want to input the apple stock price and the samsung stock price (2 features). I have created a timestep of 120 days.

Data preprocessing:

import numpy as np, matplotlib.pyplot as plt, pandas as pd, datetime as dt
import pandas_datareader.data as web

#Desired dates
start = dt.datetime(2008, 1, 1)
end = dt.datetime(2020, 1, 26)

#Stock data retreive
entire_apple_dataset = web.DataReader("AAPL", "yahoo", start, end)
entire_samsung_dataset = web.DataReader("005930.KS", "yahoo", start, end)

#Joining the 2 datasets and filling in missing data
mixed_datasets = pd.concat([entire_apple_dataset, entire_samsung_dataset], axis=1)
mixed_datasets = mixed_datasets.fillna(method="ffill")

#Grabbing both high values for each stock price and inputting them into a list
entire_set = np.concatenate((mixed_datasets.iloc[:, 0:1].values, mixed_datasets.iloc[:, 6:7].values), axis=1)

#Normalizing each column separately
from sklearn.preprocessing import MinMaxScaler
sc_apple = MinMaxScaler(feature_range = (0, 1))
sc_samsung = MinMaxScaler(feature_range = (0, 1))
entire_set[:,0:1] = sc_apple.fit_transform(entire_set[:,0:1])
entire_set[:,1:2] = sc_samsung.fit_transform(entire_set[:,1:2])

training_set = entire_set[:-16, :]

# Creating a data structure with 120 timesteps and 1 output
X_train = []
y_train = []
for i in range(120, training_set.shape[0]):
    X_train.append(training_set[i-120:i, :])
    y_train.append(training_set[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)

The actual RNN

#--------------------------RNN-----------------------------------
# Importing the Keras libraries and packages
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout

# Initialising the RNN
regressor = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 100, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a second LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 100, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a third LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 100, return_sequences = True))
regressor.add(Dropout(0.2))

regressor.add(LSTM(units = 100, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a fourth LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 100))
regressor.add(Dropout(0.2))

# Adding the output layer
regressor.add(Dense(units = 1))

# Compiling the RNN
regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set
regressor.fit(X_train, y_train, epochs = 100, batch_size = 32)
#--------------------------RNN--------------------------------------------

Predicting data

inputs = entire_set[-120-16:,:]

X_test = []
for i in range(120, len(inputs)):
    X_test.append(training_set[i-120:i, :])
X_test = np.array(X_test)

predicted_stock_price = regressor.predict(X_test)
predicted_stock_price = sc_apple.inverse_transform(predicted_stock_price)
real_stock_price = entire_apple_dataset.iloc[:, 0:1].values
real_stock_price = real_stock_price[-16:,:]

As said on the beggining, it's predictions are awful. Not only the price is ages away, also the trend isn't accurate at all.
My only objective here is to predict the trend.


texts: svm.LinearSVC: larger max_iter number doesn't always increase the accuracy/precision/recall

texts:
Background:

Supervised machine learning
Data shape

10+ features, target = 1 or 0 only, 100,000+ samples (so should be no issue of over-sampling)

80% training, 20% testing

train_test_split(X_train, Y_train, test_size=0.2)
Use svm.LinearSVC(max_iter = N).fit( ) to train labelled data

Scaling not applied yet (all feature values are around 0-100 (float64))
Other parameters (e.g., c = ) use default value

Results:

print("Accuracy:", metrics.accuracy_score(y_test, y_pred))
print("Precision:", metrics.precision_score(y_test, y_pred))
print("Recall:", metrics.recall_score(y_test, y_pred))

Question:

I increased max_iter = from 1,000 to 10,000 and 100,000, but above 3 scores don't show a trend of increments. The score of 10,000 is worse than 1,000 and 100,000.

For example, max_iter = 100,000

Accuracy: 0.9728548424200598
Precision: 0.9669730040206778
Recall: 0.9653096330275229

max_iter = 10,000

Accuracy: 0.9197914270378038
Precision: 0.9886761615689937
Recall: 0.8093463302752294

max_iter = 1,000

Accuracy: 0.9838969404186796
Precision: 0.964741810105497
Recall: 0.9962729357798165

What could be the reason?
Do I need to test different max_iter values and select the best performance? For example, use GridSearchCV( )


texts: What does the co-ordinate output in the yolo algorithm represent?

texts:
My question is similar to this topic.
I was watching this lecture on bounding box prediction by Andrew Ng when I started thinking about output of yolo algorithm.
Let's consider this example, We use 19x19 grids and only one receptive field with 2 classes, so our output will be => 19x19x1x5.
The last dimension(array of size 5) represents the following:

1) The class (0 or 1)
2) X-coordinate
3) Y-coordinate
4) height of the bounding box
5) Width of the bounding box

I don't understand whether X,Y coordinates represent the bounding box with respect to the size of entire image or just and receptive field(filter). In the video the bounding box is represented as a part of receptive field but logically receptive field is much smaller than bounding box and also people might tinker with filter size, so positioning bounding boxes with respect to filter makes no sense.

So, basically what does the coordinates of bounding boxes of an image represent ?


texts: Obtaining a confidence interval for the prediction of a linear regression

texts:
The data I am working with is being used to predict the duration of a trip between two points. There are about 100 different trips in the data and ~90k observations.

I am using the standard pattern:

feature_cols = df_features.columns.drop( [ 'log_duration' ] )
X            = df_features[ feature_cols ]
y            = df_features.log_duration

X_train, X_test, y_train, y_test = train_test_split( X, y, random_state = 42 )
linreg = LinearRegression()
linreg.fit( X_train, y_train )
linreg.score( X_test, y_test )

To perform the regression and obtain my score (~.74).

However, let's say, that it predicts that it will take 40 minutes to make the trip between two points. Obviously, it will not take exactly 40 minutes. What I am looking for is a way to report that it will take 40 minutes +/- C number of minutes for the trip.

Using Pandas and SciKit, how can I obtain C?


texts: How to pass multiple bounding boxes coordinates to CNN model?My goal is to predict the coordinates of texts in an image.


texts: ImportError: DLL load failed while importing _pywrap_tfe when running exe file created by Pyinstaller

texts:
I have created the exe file of my code which uses tensorflow on gpu. It seems that building process using Pyinstaller goes fine. However, when I run the exe file in a system without GPU, I get the following error:
ImportError: DLL load failed while importing _pywrap_tfe
Does anyone know how I can solve this issue? I have included the cuda and cudnn libraries when creating the exe file.


texts: How to combine categorical and continuous input features for neural network training

texts:
Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. It seems that simply using concat(A, B) is not a good choice because A, B are totally different kinds of data. For example, unlike B, there is no numerical order in A. So my question is how to combine such two kinds of data or is there any conventional method to handle them.

In fact, I propose a naive structure as presented in the picture

As you see, the first few layers are used to change(or map) data A to some middle output in continuous space and it is then being concated with data B which forms a new input feature in continuous space for later layers. I wonder whether it is reasonable or it is just a "trial and error" game. Thank you.


texts: Why is it necessary to overfit a yolov2 on a subset of images before model training?

texts:
I am using yolov2 for passport details detection with 8 classes? My object detectiion bounding boxes has errors though recognition being fine.

I want to know if object_detection by overfitting with a sample subset of training data, say 3 images, helps determine prediction performance? If yes how?


texts: Statistical distances for time series of distributions

texts:
I am interested in clustering $N$ time series of $T$ 'values' each. These values are distributions (which can be represented by their cumulative distribution functions (cdf), or their probability density functions (pdf), or more convenient forms such as square-root pdfs yielding a simple spheric geometry).

For comparing given distributions, there is an extensive literature on statistical distances (KL, Hellinger, Wasserstein, and so on), but for comparing given time series of distributions, I am not sure whether there is any literature at all?

Such distances should somehow take into account dynamics information besides the distribution proximity at time t. Ideally, I wish I could have a kind of information factorization similar to this result.

I am wondering if such distances already exist and whether this kind of problem has already been formulated in the literature?

--
edit for further precisions and answer to comments:

Thanks for your answer, but dynamic time warping does not suit to my need. This dp technique only captures a rough similarity of shapes by allowing non-linear time distortion. But, it does not amount for the whole information in these time series, e.g. what about the distribution of distortions? Do the distributions of a given time series vary smoothly through time or violently? DTW is not always the solution, for instance, when working with random walks, it does not make sense to use a DTW since there are no time patterns! In this case, the only information is "correlation" and "distribution" (cf. Sklar's theorem in Copula Theory), and the paper cited above.

-- edit 2 Here are the papers that are somehow related to my question:

Predicting the Future Behavior of a Time-Varying Probability Distribution
Clustering on the unit hypersphere using von Mises-Fisher distributions
Unsupervised clustering of multidimensional distributions using earth mover distance
Hilbert space embeddings of conditional distributions with applications to dynamical systems


texts: Passing tuples (key, value) into parameterized SQL query in Python

texts:
NOTE: I am new to this community, so please let me know if I can help you help me!

I built a scraper in python that creates a tuple of (key, value).

I would like to pass both key and value as parameterized variables in an SQL query that updates a table in my sqlite database.

I have exhausted researching online and trial and error and hope you guys can help. Is this even possible?

Example of what I want to achieve:

    cur.execute(''' UPDATE Ads SET ?=?''', (tup.keys(),tup.values()))

Columns in my SQLite3 database (identical to tuple.keys() ):

Marke, Kilometerstand, Erstzulassung, Kraftstoffart, Leistung (PS), Getriebe

Example of my tuple:

{'Marke': 'Weitere Automarken', 'Kilometerstand': '88.888 km', 'Erstzulassung': 'Juli 1963', 'Kraftstoffart': 'Diesel', 'Leistung (PS)': '69 PS', 'Getriebe': 'Manuell'}


texts: Could you suggest any articles, or work that has been done in the Geometrical Deep Learning field, I was recently exploring Pytorch3D and I was curious about some work that has been done recently.


texts: How to get the tagset for hindi pos tagging?

texts:
I am trying to tag Hindi text using python(nltk library). I have been successful but I am unable to understand some of the tags.

I tried searching for tagset but the only information I could find is about some upenn_tagset. The tags that I get matches only some of the tags from upenn_tagset.

Does anyone know how to get the tagset for hindi?


texts: data science / machine learning resources?

texts:
In a few weeks I'm starting a new job that will be involved in machine learning and data science.

I have a masters degree in probability / mathematics but I have no knowledge of machine learning and data science.

Are there any online resources like courses or books that could help me prepare for my new job? Basically I'm trying to get a head start on my new job.

Right now the only things I really know related to this fields are regression methods like linear regression and logistic regression, and I have a decent grasp of mathematical concepts so I'd prefer something that is not totally dumbed-down


texts:
I plan to design a recommendation system, especially with Scikit-Surprise.
A bit of background:

I want to recommend products to shop for.
 Here, the user is the shop and the items are products (water, chocolate, cheese,...).

So to create the scores and the rating matrix, I use a quota calculation that I normalize to get a score between 0 and 1, where 1 means that the store is the biggest seller of this product(in percentage) (all proportion kept).
First question: is it a good scoring ? is it a good idea to have a rating between 0 and 1?
Second question,
I want my system to take into account the time of year.
For example, it recommends more chocolate at Christmas.
How can I do that? I have an idea where I do "post-filtering", (at the end, when I already compute SVD or a system like this). So in this idea, I try to modify the rating of a product according to the period of the year and according to the evolution of the sales of this product on the sales history. So if the product is very well sold,
for example, 35% more than the average at this time of year, I increase the score by 35%.
Do you think this is a good idea? Do you have any suggestions for doing this? Do you know some libraries that can do that easily?


texts: printing theano.tensor without having any idea about the shape and type of the tensor

texts:
I am debugging a code which I haven't written. I want to print out state_below_ variable which is calculated as following:

state_below_ = tensor.dot(state_below*emb_dropout[1], tparams[pp(prefix, 'W')]) +\
        tparams[pp(prefix, 'b')]

When I use state_below_.eval(), I get

  MissingInputError: ("An input of the graph, used to compute
  Reshape{1}(y_sampler, TensorConstant{(1,) of -1}), was not provided
  and not given a value.Use the Theano flag
  exception_verbosity='high',for more information on this error.",
  y_sampler)

error. How can I print this damn "state_below_" value?

Thanks,


texts: when DataFrame has mutliIndex,how to visualize it?

texts:
I'm doing the "video games sales" project downloaded from kaggle, the data is like this:

and I want to know on each Platform, which are the most three popular Genres? but I don't know how to visualize it, I use pandas to filter the data, here is the code:

so how could I use pandas(or seaborn,etc.) to finish this target? I'm appreciate for your answers...


texts:
I am trying to understand logistic regression for a multi-class example and I have the following code:

import numpy
import theano
import theano.tensor as T
rng = numpy.random

num_classes = 3
#N = number of examples
N = 100
#feats = number of input neurons
feats = 784
#training rate
tr_rate = 0.1

D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=num_classes))
training_steps = 1000

# Declare Theano symbolic variables
x = T.matrix("x")
y = T.ivector("y")

w = theano.shared(rng.randn(feats), name="w")
b = theano.shared(0.01, name="b")

# Construct Theano expression graph
sigma = T.nnet.softmax(T.dot(x,w) + b)
prediction = T.argmax(sigma, axis=1)          # The class with highest probability

# Cross-entropy loss function
xent = -T.mean(T.log(sigma)[T.arange(y.shape[0]), y])

cost = xent.mean() + 0.01 * (w ** 2).sum()      # Regularisation
gw, gb = T.grad(cost, [w, b])                   # Compute the gradient of the cost

# Compile
train = theano.function(
          inputs=[x,y],
          outputs=[xent],
          updates=((w, w - tr_rate * gw), (b, b - tr_rate * gb)),
          allow_input_downcast=True)
predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    train(D[0], D[1])

However at this point the code gives me the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-246-c4753ce8ccc7> in <module>()
      1 # Train
      2 for i in range(training_steps):
----> 3     train(D[0], D[1])

/Users/vzocca/gitProjects/Theano/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    869                     node=self.fn.nodes[self.fn.position_of_error],
    870                     thunk=thunk,
--> 871                     storage_map=getattr(self.fn, 'storage_map', None))
    872             else:
    873                 # old-style linkers raise their own exceptions

/Users/vzocca/gitProjects/Theano/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)
    312         # extra long error message in that case.
    313         pass
--> 314     reraise(exc_type, exc_value, exc_trace)
    315
    316

/Users/vzocca/gitProjects/Theano/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    857         t0_fn = time.time()
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:
    861             if hasattr(self.fn, 'position_of_error'):

ValueError: number of rows in x (1) does not match length of y (100)
Apply node that caused the error: CrossentropySoftmaxArgmax1HotWithBias(Elemwise{Add}[(0, 0)].0, Alloc.0, y)
Toposort index: 12
Inputs types: [TensorType(float64, row), TensorType(float64, vector), TensorType(int32, vector)]
Inputs shapes: [(1, 100), (100,), (100,)]
Inputs strides: [(800, 8), (8,), (4,)]
Inputs values: ['not shown', 'not shown', 'not shown']
Outputs clients: [[Sum{acc_dtype=float64}(CrossentropySoftmaxArgmax1HotWithBias.0)], [], []]

Backtrace when the node is created:
  File "<ipython-input-244-834f04b2fb36>", line 16, in <module>
    xent = -T.mean(T.log(sigma)[T.arange(y.shape[0]), y])

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Can anyone help understand what is wrong?
Thank you!


texts:
I have a dataframe which has two date columns - Start Date and End Date,
Let's suppose start date is 4/10/2019 and end date is 4/10/2020 (MM/DD/YYYY) now I want to split these dates in a list which will have all the intermediate months (from october 2019 to october 2020). Is there any function in python that can hep me with this?


texts: Xgboost performs significantly worse than Random Forest

texts:
I have a dataset of 3500 observations x 70 features which is my training set and I also have a dataset of 600 observations x 70 features which is the test set.

The target is to classify observations correctly either as 0 or 1. 2000 observations of the training set are 0 and the rest 1600 of them are 1.

I aim at the highest possible recall for precision>=90%.

I did grid search for ensemble algorithms only in relation to number of trees (from 50 to 650 trees). Analytically the best recall results for precision >= 90% for each of the algorithms are the following:

Random Forest (375 trees)

from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(random_state=0, n_estimators=375, class_weight='balanced')
classifier.fit(X_train, y_train)

Precision: 90%
Recall: 24%

Xgboost (550 trees)

from xgboost import XGBClassifier
classifier = XGBClassifier(n_estimators=n_trees, seed=0, scale_pos_weight=1.5)
classifier.fit(X_train, y_train, eval_metric='map')

Precision: 90%
Recall: 15%

Why Xgboost is performing so much worse than the Random Forest?


texts: Neural network accuracy for simple classification

texts:
I am trying to develop a NN for a very simple classification model with keras/tensorflow:

Ex:

input: "Do" => class output: "Dog"
input: "Ca" => class output: "Cat"
input: "Mo" => class output: "Mouse"

I train the model with many "Do", "Ca", etc. (as dictionary), indexing the input to categorical arrays ([0,1,0] for example is "Cat", [1,0,0] is "Dog", [0,0,1] is "Mouse" in case of three classes).

I know that for this type of problem every other traditional classification algorithm should be used, and not DL, but I'm doing it with a purpose (I need to merge it with an image classifier).

My question is: since the classification is easy ("Do" is always "Dog", "Ca" is always "Cat") a Decision tree would have an accuracy of 100% always. Why do Neural Networks need tons of data and epochs to get an acceptable accuracy? With 10 classes, it takes 10 epochs and 8 thousand entries in order to get an accuracy higher than 90%. I'm using sparse categorical crossentropy as loss, and SGD as the optimizer. (2 Dense layer, relu - softmax). Also, I am a bit lost on how to choose the number of neurons, I guess trial and error is the way.


texts: Creating graphs of CNN models: cannot use plot_model from keras.utils.vis_utils

texts:
I am using keras w/ PyCharm on windows 10. I am trying to print my tensorflow.keras.Model using

from keras.utils.vis_utils import plot_model
...
res_net_model = tf.keras.Model(inputs=inputs,outputs=outputs)
...
plot_model(res_net_model)      #, to_file='cnn_resnet_model.bmp') #, show_shapes=True, show_layer_names=True)

I am for some reason getting

  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\keras\utils\vis_utils.py", line 240, in plot_model
    expand_nested, dpi)
  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\keras\utils\vis_utils.py", line 79, in model_to_dot
    _check_pydot()
  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\keras\utils\vis_utils.py", line 28, in _check_pydot
    pydot.Dot.create(pydot.Dot())
  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\pydotplus\graphviz.py", line 2032, in create
    status, stderr_output))
pydotplus.graphviz.InvocationException: Program terminated with status: 1. stderr follows: Format: "ps" not recognized. Use one of:

Process finished with exit code 1

I have confirmed that graphviz & pydotplus has been installed by conda. I've also tried to installing graphviz as an executable that's within my path.

I've looked through similar postings such as this and this with no luck: nothing matches my error. It doesn't even tell me what formats I should use (although this may have the answer). It shouldn't matter because atm I'm not asking it to generate an image. I have no idea where it is getting 'ps' from.


texts: Dropping attributes leads to better classifier accuracy? (Titanic Set)

texts:
I am currently tackling the Titanic Dataset on Kaggle. The goal is to find a classifier that can predict whether a passenger will survive or die.
The dataset has features that I believe are strongly linked to the likelihood of survival, i.e. the passenger class. Clearly, the survival probability is higher for passengers travelling in class 1 than in class 3. Additionally, there is a "Fare" attribute, which describes the exact price a passenger has paid.
I have tried different classifiers (e.g. Stochastic Gradient Descent and Random Forest) and both have a better accuracy when I drop the passenger class attribute.
Another thing I have noticed is, that if I condense the attributes "Siblings" and "Parents/Children" into a single attribute "TravelledAlone" (which can be 1 or 0), the classifier becomes more accurate.
In total these steps increase the accuracy from about 70% to almost 90%.
Can this be explained? After all, my results have become better by throwing away information...


texts: Signal correlation - matching specific points

texts:
Question: What are some recommended techniques for matching specific patterns in data sets?
Background
I have several thousand sites for which I have collected time series data. In the example image below we have increasing time in y-direction with data streams from 2 different instruments shown from two sites. I have added hand-picked, color-coded correlation markers for four significant events.  Normally we would pick 10-15 of these markers at each measurement location.
Typically, these markers are correlated by hand, however, with 1000s of sampling sites this is not feasible in a reasonable amount of time.  I have performed hand correlations on ~150 of the data streams, but would prefer to use an automated or semi-automated process to do the correlation.
We have the following beliefs about the data:

By picking a sparse data set (e.g., the first 150 hand picked correlations) we can make a strong guess about the location of the marker in adjacent sites.
The time-separation between markers is not constant, but has only minor variation
Marker A will always come before Marker B, and Marker B before Marker C and so on.
Not all markers will be present at all measurement sites
Not all measurement sites were able to evaluate a full time series (in example below, imagine if one had been cut off at 8000ms instead of going past 10,000ms
The data can easily be normalized to have a similar range of values.

Prior Approach
I have used both DTW and Fast DTW (dynamic time warping) to perform the task, but it only works well when there is a full and complete data stream and when all markers are present.  The downside to DTW is that it is an O(N2) approach, and with data streams of 20,000+ samples and about 5000 measurement sites, it is simply too computationally slow.
Restatement of Question

What are some recommended techniques for matching specific "markers"
in time series data?
Are there techniques that just evaluate certain
portions of the data stream vs. something like dynamic time warping
that evaluates the whole data stream?
What data pre-processing should
I consider to make this computationally easier?

Example Data


texts: Can someone explain the lambda collocation metric?

texts:
I'm trying to understand the lambda collocation scoring metric from the quanteda package in R, but the explanation in the documentation is incredibly difficult to understand.

Can anyone explain it better?

Having an example worked out on a toy set of text would help too, like something with 5-10 words.  This line (and the one right before it) is especially confusing:

  The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.


texts:
I'm trying to understand exactly how feature_importances in scikit-learn's RandomForestClassifier works. I managed to find this helpful link explaining most of the process: https://towardsdatascience.com/the-mathematics-of-decision-trees-random-forest-and-feature-importance-in-scikit-learn-and-spark-f2861df67e3
I just have two questions about the equation for ni_j (equation for node importance, first equation in the section on feature importance):

What is meant by weighted number of samples reaching node j? Are they running all the training samples through the tree and counting how many reach node j? If so, why is the word 'weighted' necessary?
Can we be mathematically certain that ni_j >= 0?

Edit: Looking at the code for forest.py, I see this rather mystifying function:
def feature_importances_(self):
        """Return the feature importances (the higher, the more important the
           feature).
        Returns
        -------
        feature_importances_ : array, shape = [n_features]
        """
        check_is_fitted(self, 'estimators_')

        all_importances = Parallel(n_jobs=self.n_jobs,
                                   backend="threading")(
            delayed(getattr)(tree, 'feature_importances_')
            for tree in self.estimators_)

        return sum(all_importances) / len(self.estimators_)

I can see it's calculating a feature importance at each tree, and then taking the mean over all trees. But how it manages to calculate the feature importance at a tree without calling any functions to count samples or compute impurities, I don't understand.


texts:
I'm moving my project from Python's libraries to opencv, and I have one big problem.

In Python skylearn, I have kmeans object, which has two useful methods: fit_predict and fit, which both are nescessery for my project.

The problem is I can't find any equivalent in opencv.
There is only function cv::kmeans, which is not enough for me.

When I'm using Python skylearn the pseudo code of my program looks like:

kmeans.fit_predict(train_data) // it gaves me labels, and compute centers

kmeans.predict(test_sample) // it gaves me labels for test sample computed from centers which were computed above

Is there any equivalent or way to do the same in opencv?


texts:
I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments which may contain multiple sentences and the task is to predict whether the second segment is the direct successor of the first one. RoBERTa's authors proceed to examine 3 more types of predictions - the first one is basically the same as BERT, only using two sentences insted of two segments, and you still predict whether the second sentence is the direct successor of the first one. But I can't understand what the goal is in the other 2. I will cite their explanation below:
• FULL-SENTENCES: Each input is packed with full sentences sampled contiguously from one or more documents, such that the total length is at most 512 tokens. Inputs may cross document boundaries. When we reach the end of one document, we begin sampling sentences from the next document and add an extra separator token between documents. We remove the NSP loss.
• DOC-SENTENCES: Inputs are constructed similarly to FULL-SENTENCES, except that they
may not cross document boundaries. Inputs sampled near the end of a document may be
shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as FULL-SENTENCES. We remove the NSP loss.
So from what I understand in these two training strategies they already sample consecutive sentences, or at least consecutive sentences from neighbouring documents,  and I can't see what they are trying to predict - it can't be whether they're consecutive text blocks, because to me it seems that all of their training examples have already been sampled contiguously, thus making such a task redundant. It would be of enormous help if someone were to shed some light on the issue, thanks in advance!


texts: Machine learning algorithm to classify matrices

texts:
I want to know what could be first choice for a machine learning algorithm to classify matrices.

Each matrix belongs to either class A or class B.
The classification problem is: To classify each matrix into either class A or class B (say signal matrix or noise matrix).

Each matrix is arranged in a fashion where each row contains certain features of an object associated with the matrix.

I obviously have thousands of such class A and class B matrices to train on.

I will be really obliged if you could help. Thanks.


texts: Is there a real multi-dimension timeseies dataset in which data samples have feature based similarities?

texts:
I'm looking for real datasets which are multi-dimension time-series and there exist partial similarities between different categories based on some of the features/dimensions.

For example, consider two persons waving, but one is doing that while walking and the other while driving in a car. So their generally they are related to different motion categories,  are different but both are similar according to their arm movements.

Thanks!


texts: Is there an implementation of pocket algorithm in Keras?

texts:
As explained in Wikipedia, the pocket algorithm is a very simple variant/addition of/to ANN which keeps a copy of the best model seen so far and returns that one as the trained model (instead of the actual final state of the model). Implementing it is very simple and straightforward.

I was wondering if this algorithm is implemented in Keras.


texts: My goal is to extract the feature importances from already trained random forest classifier and transfer them to another classifier. How this can be done? and How can access to modify feature_importances of Random Forest Classifier model? using scikit learn.


texts: Error is occurring setting an array element with a sequence

texts:
I have the columns in my Data Frame as shown below:
 Venue             city                Venue Categories

  Madison         London      [1, 1, 1, 1, 0, 0, 0, ...,0,0]
  WaterFront      Austria     [0, 1, 1 0, 0, 0,  0, ....0,1]
  Aeronaut        Marvilles   [0, 0, 0, 0, 1, 1, 1, ....1,1]
  Aeronaut        Paris       [0, 1, 1, 0, 0, 0, 0, ....1,1]
  Gostrich        New York    [0, 0, 1, 0, 0, 0, 0, ....1,0]

I am passing this data to my machine learning model , but model.fit is not accepting the input , My code is shown below , that I am trying ,
labelencoder = LabelEncoder()
dff['Venue']=labelencoder.fit_transform(dff['Venue'])
dff['city']=labelencoder.fit_transform(dff['city'])

Let's say , if I want to increase the number of features . I want to add more columns , then I again write everything for each feature just like shown below , if i want to add type column and owner column
city = dff['city'].values
owner = dff['owner'].values
type = dff['type'].values
categories = dff['Venue Categories'].values
labels = np.array(dff['Venue'].values)
data = np.array([make_sample(city[i], owner[i], type[i] categories[i]) for i in range(len(city))])

This will looks weird , I want to make it global , means there should not need to touch the code if we may increase the number of columns .


texts: It is helpful to normalize target variables for a regression neural network?

texts: It is customary to normalize feature variables and this normally does increase the performance of a neural network in particular a CNN. I was wondering if normalizing the target could also help increase performance? I did not notice an increase in performance with the data set I am using at the moment but was curious if anyone has tried in the past. Of course the normalization happens only on the training data.


texts: Is there any problem with dropping only part of the OneHot generated features?


texts: Classification of phone numbers belonging to same client

texts:
I have the following task at hand:
Suppose, that there is a data on clients actions from the base stations of the mobile operator.

While being in the reach of the base station, client can make the following actions: Turn off the phone, change location, turn out of range, and etc. Hence, there are several data points for each phone number in the dataset, with attributes like coordinates of base station, type of the base station, type of device, client action, and so on. Also, there could be several base stations in one location, but pointing in different directions.

Also, there is a training set of phone numbers that belong to the same client. The task is to find all phone numbers that belong to one client (Each client can have 2 or 1 sim-cards).

As of now, my main idea is to somehow embed data points of each phone number into vector, and use some distance threshold small enough, to group these vectors into the pairs. But in this scenario i'm not using the training data at all.

Any suggestions would be appreciated.

P.S. After digging into the data a little bit, i found out that the data on each phone number is hugely varied in length. For instance, there could be 8 data points for one number, and 1000 for another. On this evidence, i'm not considering the idea with embedding viable anymore.


texts: Is applying pre-trained model on a different type of corpus called transfer learning?

texts:
I trained my classification model on corpus A and evaluated it on corpus B.
I do it, because for corpus A I have a lot more labeled sentences than for B. Nature of sentences used in A is different than sentences using in B. A has name of products from e-shop, B has names of products as they appear in shopping lists, with all slang, abbreviations, spelling errors and private notes.
Am I doing transfer learning?


texts: TensorFlow: What is wrong with my (generalized) dice loss implementation?

texts:
I use TensorFlow 1.12 for semantic (image) segmentation based on materials. With a multinomial cross-entropy loss function, this yields okay-ish results, especially considering the sparse amount of training data I´m working with, with mIoU of 0.44:

When I replace this with my dice loss implementation, however, the networks predicts way less smaller segmentation, which is contrary to my understanding of its theory. I thought it´s supposed to work better with imbalanced datasets and should be better at predicting the smaller classes:

I initially thought that this is the networks way of increasing mIoU (since my understanding is that dice loss optimizes dice loss directly). However, mIoU with dice loss is 0.33 compared to cross entropy´s 0.44 mIoU, so it has failed in that regard. I´m now wondering whether my implementation is correct:

def dice_loss(onehots_true, logits):
    probabilities = tf.nn.softmax(logits)
    #weights = 1.0 / ((tf.reduce_sum(onehots_true, axis=0)**2) + 1e-3)
    #weights = tf.clip_by_value(weights, 1e-17, 1.0 - 1e-7)
    numerator = tf.reduce_sum(onehots_true * probabilities, axis=0)
    #numerator = tf.reduce_sum(weights * numerator)
    denominator = tf.reduce_sum(onehots_true + probabilities, axis=0)
    #denominator = tf.reduce_sum(weights * denominator)
    loss = 1.0 - 2.0 * (numerator + 1) / (denominator + 1)
    return loss

Some implementations I found use weights, though I am not sure why, since mIoU isn´t weighted either. At any rate, training is prematurely stopped after one a few epochs with dreadful test results when I use weights, hence I commented them out.

Does anyone see anything wrong with my dice loss implementation? I pretty faithfully followed online examples.

In order to speed up the labeling process, I only annotated with parallelogram shaped polygons, and I copied some annotations from a larger dataset. This resulted in only a couple of ground truth segmentations per image:

(This image actually contains slightly more annotations than average.)


texts: I am trying simulate  data from a normal distribution but bias the sample by excluding all negative values and values divisible by 5 . To demonstrate the effect  of bias . I will probably calculate the statistics from the original vs biased sample. I want to generate the data for samples 100,100,1000  and show plots using python . I hope this clarifies the question. I'm not sure how to approach.


texts: Find value and return row in excel spreadsheet with over 1 mil rows

texts:
I have an excel spreadsheet that has over 1 million rows and about 120 columns. I frequently need to find a value or a part of a value (value is name or number) and return the row where the value is found. I only need values from about 20 columns so I could drop out the extra columns but that does not solve the problem of excel being very slow in finding the value and it is very difficult to read information in the row.

What would you recommend to use for this task? Ideally I would like to enter the value and get the row in some sort of a report but not necessary as long as it is well readable.

I was thinking of using Power BI but got kind of stuck and in the end felt that it wasn't the right tool for the task.

I am open for all suggestions!


texts: Relationship between Sigmoid and Gaussing Distribution

texts:
I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?":

The assumption of a dependent variable to follow a sigmoid function inherently assumes a Gaussian distribution for the independent variable which is a general distribution we see for a lot of randomly occurring events and this is a good generic distribution to start with.

Could someone elaborate on this relationship between the two?


texts: Logistic Regression outperforms MLPClassifier

texts:
I am new in ML and I am trying to train classifier. I have a tiny dataset, just 90 examples, I divided it 70/30 train/test set and started to train.

As I know MLP must outperform Logistic Regression, but I have an opposite situation in my case, firstly I thought that I give bad hyperparameters, so I tried to use GridSearchCV for finding best hyperparameters, but after it, again Logistic Regression was better, I am wondering maybe is it because of the small number of dataset or I missed something?


texts: Does anyone know of literature regarding a Neural Net boosted GBM?

texts:
For obvious reasons, most GBMs created in the private sector are tree boosted.
Occasionally, one might want a linear boosted GBM so that the residual models collapse into a simple linear combination. However, linear boosted GBMs take a lot longer to train.
So what I'm wondering is, has anyone done research on a Neural Net boosted GBM? The model itself would seem to be computationally implausible in almost any use case. But I'm still interested if there has been research on it.


texts: Detect different "sections" of stationarity in time-series data?

texts:
I'm working with some time-series data that is clearly non stationary. At first glance, it looks like there are 3 sections of my data. Within each section, the data have similar mean and variance, but between each section the variance of the data is quite different.

Is there a well known algorithm to detect and label different sections of stationarity in time-series data?


texts: Data Science education curriculum design and guidelines in Computer Science and other Disciplines

texts:
I am studying data science education curriculum design and guidelines. All resources I have up to now is about data science in statistics education and library and information science education.

Is there any work on data science in computer science/software engineering education (adding data science courses to a traditional software engineering curriculum) ,management and other disciplines?

An example of including the curriculum for statistics is this paper


texts: Is regression or ANOVA the right approach for determining RRP based on a set of RRPs associated with categorical variables

texts:
Basically I have scraped some retail site data and I would like to know if this is a regression problem:
If I have data from the website about shoes like RRP, what brand they are, what colour they are and what size and I collate that data to build a model around such that I can give the model known variables about a new pair of shoes (brand, size colour) to determine it's likely RRP based on the set of shoes i already know all the variables and RRP for, should I use ANOVA or regression. My reasonings for regression is that it seems like a classic problem in that price is a continuous variable. But I'm unsure to go about this, a lot of the regression examples i see use contiuous variables for both the predictor and outcome variables maybe ANOVA is better?


texts: Create a Predictive model to find users that would book an exam

texts:
I'm currently working for a health company in Brazil, and I want to create a Predictive model that would find users that are one step close to book an appointment.

We currently have 1 million visits per month, and 60% of this visits are not booking anything (exam, consult, checkups, etc). I believe that with ML, we could find which users would book if we make a call or send an email.

I have been a professional web development for the past 7 years, but I don't have the skills to create a Predictive model, so I'm looking for directions, a way to start.

We do have Google Analytics, but I'm thinking to create a simple software that would track:

Age
Gender
Which pages the user visited
How many time the user expended in each page
Name

Then, I could segregate the users in two groups: the ones that booked, and the users that didn't book, and then... I'm stuck.

Could someone give a direction? Should I create a software to track this information or use GA? I want create a demo to get financing, so I don't have much time :/

Thank you.


texts: Combining K-means clustering with Agglomerative clustering

texts:
I am trying to identify 6 clusters in a graph of authors. Authors are connected with an edge when they have co-authored on a paper. I have already created a clustering with K-means (6-Means) and with Agglomerative clustering. Now, I want to combine these two clustering approaches in order to get more accurate clusters.
I am able to find corresponding clusters, but I have no clue on how to decide to which cluster an author belongs when the algorithms do not agree.

Possibly I could use some form of a linear combination, but I do not know how to apply this properly. Other methods might be possible, but I am not sure on that.


texts: Understanding the XLNet model for a concrete case

texts:
I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it:
Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (only applying what the paper says), if not, why? I thought that if we combine a hierarchical classification and then apply the XLNet model it can lead to better results for many categories but I'm not sure because many examples in Kaggle and other websites use XLNet directly. Has some sense what am I saying or I didn't properly understand what XLNet does since I didn't see anyone applying this proposal for many categories?


texts: why do we need row sampling in random forests?

texts:
In random forests, where our estimators are decision trees, we do column (feature) sampling without replacement within an estimator, and with replacement  in between estimators. This is perfectly fine as we are trying to reduce the high variance of individual decision trees.

But what is the need to do row sampling?

Usually more the data, the better it is for a model to learn, and even if i dont have any computational resource limitation, why do we have to do row sampling in estimators for random forest classifier?


texts: How can I create a constant keras tensor of variable length?

texts:
I am trying to create a tensor of zeros of variable length for inputs using keras backend but unable to define the shape of the same. The sample code is shown below:

from keras import backend as K

def func():
  const = K.constant(0, dtype='float', shape=[?,1])
  return const

Of course such a syntax would not work. Can somebody tell me how I can rectify this?


texts: How to apply a different Loss function to one specific Label?

texts:
I got a recurrent neural network in Keras, which classifies on 14 labels.
The first label is the most important one and should be predicted with the highest accuracy. The other labels don't have to be very accurate, they should just help the network to generalize better.

I thought of using the "Mean absolute error" for the first and the "mean squared error" for the other labels.

How can I integrate this in Keras?


texts: Can overfitting occur in Advanced Optimization algorithms?

texts: while taking an online course on machine learning by Andrew Ng on coursera, I came across a topic called overfitting. I know it can occur when gradient descent is used in linear or logistic regression but can it occur when Advanced Optimization algorithms such as "Conjugate gradient", "BFGS", and "L-BFGS" are used ?


texts: Dealing with an apparently inseparable dataset

texts:
I'm attempting to build a model/suite of models to predict a binary target. The exact details of the models aren't important, but suffice to say that I've tried with half a dozen different types of models, with comparable results from all of them.

On looking at the predictions on various subsets of the training data, it appears that a certain subset of features is important for around 30% of the data, while a different subset is important for the remaining 70%. With the training data/holdout set. This separation is fairly easy to detect when the target is known (run a model with subset1, another with subset2, find the subset where one model does better than the other). Obviously, this is not possible with the test data, since the target is not known there.

There are clearly (at least) two regions in the data that are substantially different from each other, since a model trained on the whole dataset does worse than the combination of separate models trained on each of these regions.

However, the dataset seems spectacularly resistant to separation, with no major difference between principal components (linear and RBF kernel) and no clear/stable clusters after applying several different clustering algorithms (KMeans, agglomerative, mean shift).

Are there any popular methods that allow separation/clustering of data that are resistant to "normal" methods? The end goal here is to find out which model to use on which rows when target is unknown.


texts: Should I have "normal" sampled data in my dataset?

texts:
I am busy working on a project to find the reasons why kids in normal households are doing badly in school.

I have a dataset of which consists of kids that live in environments where the family is middle class, has access to necessary facilities and the kid is not suffering from any disorders but is failing grades in school.

It is understandable for kids that has a poor living condition to have problems at school but for kids that has all the necessities in life to do at least average in school, needs a bit more research.

Now that I have this dataset, does it make sense to add kids with the same living environments that are doing ok at school to the dataset?
I am planning to use SOM for data mining if that helps.


texts:
I'm working with domain-oriented documents in order to obtain synonyms using Word2Vec.
These documents are usually templates, so sentences are repeated a lot.

1k of the unique sentences represent 83% of the text corpus; while 41k of the unique sentences represent the remaining 17% of the corpus.

Can this unbalance in sentence frequency impact my results? Should I sub-sample the most frequent sentences?


texts: Ignoring symbols and select only numerical values with pandas

texts: In one field I have entries like 'U$ 192,0'. Working on pandas, how I ignore non numerical data and get only the numerical part?


texts: Is there an alternative to Google Maps to find GPS coordinates of a place in Germany?

texts:
For probably any place on Earth, you can search it in Google Maps, make right-click, pick the option "What's here" and you get GPS coordinates.

If I don't want to use Google Maps API, is there an alternative open API to get GPS coordinates by address? To be less broad, say in Germany?


texts: How to plot an histogram from a dictionary data?

texts:
This is my data which is recorded in a dictionary. A sample of my dictionary looks like this: (id-s:{id-des:value})

{'1': {'1': 2.5039}, '2': {'2': 8.993242, '1': 8.972473}, '7': {'5': 6.4600, '7': 9.16026}, '3': {'5': 6.46004, '3': 8.8472, '6': 8.8266, '1': 8.806035}, '5': {'5': 8.70111}, '6': {'6': 8.8889}, '4': {'4': 9.55662}}

Firstly,I want to plot them in histograms. Every figure presents values of an  id-s. Every bar presents values (2.5039 is the value in the first example).

I have tried with this code.

plt.subplots(len(param_dict), 1, sharex=True)
    plt.subplots_adjust(hspace=0.8)
    for key, src in param_dict.items():
        plt.title('Source : {}'.format(key), fontsize=9)
        print(key)
        for attribute, value in src.items():
            print('{} : {}'.format(attribute, value))
            plt.hist(value, bins=20, alpha=0.5, label='my label')

But it takes only the last data.

Secondly, I want to plot all of them in one histogram, all values whatever their id-s.


texts: Word2Vec / Doc2Vec - map change of words overtime

texts:
I have text data for 10 years and I want to run a Word2Vec model on the text data each year and calculate the distances between words and I would like to compare them over time.

Are there any papers in which the Word2Vec model is compared to previous years data? Or if I have a matrix of the Word2Vec vectors are there ways to compare these matrices year-on-year or from period t to t-1?

I can compute the cosine distance of the matrix but cannot think how to compare these "scores" for each word over time.


texts: Can LSTM be used to predict value as regression problem?

texts: I have time-series data as shown below. Which model is generally preferred if grig_id is needed to be predicted? Is it possible to use LSTM with a sigmoid activation function at the final layer for such a task?


texts: Graphs demonstrating the structure of neural networks are very unclear

texts:
This image looks pretty familiar to anyone getting acquainted with neural networks, and on first glance it makes a lot of intuitive sense.

But on second, third, fourth, etc glance, some questions pop up! I hope I'm not the only one with these concerns.

There are 3 circles representing Input nodes. Does this mean that the data sent into the NN has to be separated into multiple batches? I don't hear much talk of that, or any kind of emphasis / reasoning. Why don't we send the data through the network all at once? Can't the video card handle it? (Or is that the reason why right there, that the card can't handle it?).

There are 4 circles representing Hidden nodes. Each previous node points to
a Hidden one. The way this is show, it is confusing:

Does this mean that data coming in is somehow merged together? I see 3 arrows all coalescing together into the next layer. That's what it looks like visually.

Is there some order, or sequence here? There's no indication of this process happening in some order. It just looks like a mess of arrows pointing from Input to Hidden. Shouldn't be be concerned with the order in which these activations flow? After all, we are looking at a network graph.

There are 2 circles representing Output nodes. Is this showing that a NN can output 2 different sets of data?? Why do we want multiple outputs? We started with 3 blocks of data, now we have 2. How are 2 separate blocks of data going to help us make a binary choice on, say, image classification?

The more I look at this image, the more I dislike it. Yet I see it everywhere, in many introductory / intermediate discussions of NN. I'm hoping there is a reason why such an unclear image is so popular.


texts: What are the benefits and tradeoffs of a 1D conv vs a multi-input seq2seq LSTM model?

texts:
I have 6 sequences, s1,..,s6. Using all sequences I want to predict a binary vector q = [0,0,0,1,1,1,0,0,0,1,1,1,...], which is a mask of the activity of the 6 sequences.

I have looked at seq2seq lstm models, but am struggling with the multiple-sequence-input and single-sequence-output architecture. Am I headed down the right path, or should I shift my focus to a convnet with 6 non-spatial dimensions, and 1 spatial dimension?

Thanks,


texts: Convert Keras model to C++ using protobuf file and Tensorflow C API

texts:
I need to convert Keras model to C++ code. I found some implementations to do this: keras2cpp, another keras2cpp and frugalydeep. Only the third one was able to convert my model to cpp code. But the frugalydeep has no support for GPU. Therefore I am looking for another option to make this. My thoughts are to save Keras model to .pb(protobuf) file and then load this model with Tensorflow C API.
Do you think this will be possible? Or I will have to manually rewrite the Keras model to Tensorflow C API.
Are there any other ways to do this task?


texts: Meaning of subscript in min max value function

texts:
This possibly is a very stupid question, but i have not been able to find the answer on the internet and have got no clue which keywords to use while searching.

What's the meaning of $\mathbb{E}_{x \sim p_{data}(h)} [...]$
Where ... is some function.

in context like

$\mathbb{E}_{x \sim p_{data}}[...] + \mathbb{E}_{x \sim p_{z}} [...]$

and

$\mathbb{E}_{x \sim p_{data}(h)} [...] + \mathbb{E}_{x \sim p_{z}(z)} [...]$

This is written in a lot of papers, but what does it mean?


texts:
i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will be used as input for the final model (logistic regression too).

the default base model used a softmax function. i think transform a confident value into a probability will lose much information. so i have a plan to change the softmax into activation function.

what i learn in my CNN class, Relu is the best default activation function for an image. but my case is multi-class classification email.

which activation function should i choose?
Relu, sigmoid or other?

thanks


texts: How to organize data in workflows with user input between steps?

texts:
I am building a multistep workflow.

User provides input files input.csv, parameters_global.yml, parameters_step1.yml, parameters_step2.yml.
Step 1 loads the data and parameters: input.csv, parameters_global.yml, parameters_step1.yml. It processes the data and x, processes it and produces step1_results.csv.
User has to check the results, mark every line OK or not, and saves this as step1_checked.csv.
Step 2 loads parameters_global.yml, parameters_step2.yml and step1_checked.csv, processes everything, and produces some step2_results.csv.

Repeat ad infinitum. Are there best practices for how to organize the different steps (data and code) in data input, output etc folders?
Current suggestion is

input/01_step1name/, input/02_step2name/ for all user-provided extra input per step
output/01_step1name/, output/02_step2name/ for all generated output per step
scripts/01_step1name/01_firstscript.R, scripts/01_step1name/02_secondscript.R, scripts/01_step1name/03_thirdscript.py
all steps may use input and output from any preceding step
within one step, there shall be no additional user input (but potentially multiple scripts).

There are also cases where a user might need to take an output and go to a measurement device, measure something based on that output, come back and add the results as a new input for the next step. I could "symbolize" the measurement by adding an extra step where there is nothing to execute, and manually put the measurements in the output folder when done. Alternatively, I could treat the measurement as user input and use the measurement results as input for the next step...
An extra add-on is that some but not all steps share a code basis, e.g.

scripts/02_processingstep2/01_run.R sources a functions.R
scripts/03_processingstep3/01_run.R sources the same functions.R
scripts/04_processingstep4/01_run.R might use a completely different set of R libraries/environment/functions

I don't really have a good concept to handle this... perhaps an extra

library/mytoolset/ with functions.R
library/othertoolset/ with different codebase

I know about Nextflow, but I feel it doesn't solve the problem of organization, more the problem of execution which is not such a big deal for me.


texts: Probability Calibration : role of hidden layer in Neural Network

texts:
I try a simple Neural Network (Logistic Regression) to play with Keras.
In input I have 5,000 features (output of a simple tf-idf vectorizer) and in the output layer I just use a random uniform initialization and an $\alpha = 0.0001$ for $L_{2}$ regularization with a sigmoid activation function (so something pretty standard).

I use an Adam optimizer and the loss function is the binary cross-entropy.

When I display the probability for both classes I end up with something like this :

After that I try to add a hidden layer with a $Relu$ activation function, 64 nodes and again the same parameters for regularization & initialization.
-- EDIT --
Here for the output layer I keep exactly the same parameter (and $sigmoid$ activation function) as in the previous NN.
-- END OF EDIT --

But now when I plot the probability distribution for both classes I end up like this :

And cleary I didn't get why by adding a hidden layer the probability are push to 0 or 1 ?

Have you any references that I can read so I can understand the math behind ? It could be great!

Moreover sometimes (with a more "deeper" neural network for another application) I get the same plot as the second one but this time the predicted probabilities are between $[0.2; 0.8]$. The probabilities are push to some "values" but those values are more "centered". Not sure it is clear. And here again I don't understand what is the phenomena behind this ? How can I "debug" it to see in my architecture of my Neural Network what is the cause ?

In addition How can I "tweak" my ANN to have a "perfect" calibration plot (like in the scikit webpage : https://scikit-learn.org/stable/modules/calibration.html) ?

Thank you in advance for every answer that can enligthen me :)


texts: Cosine Similarity but with weighting for vector indexes

texts:
I am very new to NLP and although this seems like a basic question I don't know how to search for an answer online.
This is my problem: I have extracted and ranked keywords from 2 text sources:

A rank of 1, means this keyword is more important than a keyword with rank 5. Some keywords may not exist in one text but do in the other. In this case, where the keyword does not exist, there is no rank, therefore, Nan.
What method do I need to use to extract the similarity between the keyword ranks? I want to find out how similar the 2 texts are based on what keywords it contains and the ranks of these keywords.
I have tried cosine similarity by removing rows that contain Nan values and then treating text1Rank and text2Rank as vectors like this:

the 2 columns are the vectors I pass into the cosine similarity formula.
However, I do not think that this method weighs higher-ranked keywords more than lower-ranked keywords. Am I correct in thinking this?
If so, what method should I use to compare the 2 sets of ranks of keywords?


texts:
I know how to use LDA as a classifier.
But how to use Linear Discriminant Analysis as a dimensionality reducer to reduce the number of features and apply logistic regression on top of it.
I am using R language.


texts: Different estimation methods of Time series model

texts:
This question might be a huge topic, but let's talk about the simple case, say AR (auto-regressive) model. Usually people have various methods of estimating the AR model, like Burgs algorithm, Yule walker algorithm, or even OLS (ordinary least squares). Could some one tell me what is the motivation of using so many different skills to estimate the same model. Is there any literature of book discussing these topic?

Thanks a lot,


texts: Will unnecessary features harm the tree based model?

texts: Is it necessary to drop noisy features (eg column of random numbers) from tree features? I think it's not. sometimes it may benefit but will never cause any harm to the model. Because at each split model is checking which feature will reduce the impurity. Sometimes by chance random numbers may be the one.


texts: Creating an Object Detection model with images and coordinates of bounding boxes

texts:
I am trying to build a custom Object Detection model which can detect guns from a given image. The dataset is from here.

The dataset has a good number of images and each image has 4 coordinates of bounding boxes with it. I recently read about YOLO and the structure of its labels is as follows:

I need to get this label for every grid cell of the input image. The dataset which I have contains coordinates for the object and not a grid cell in the image.

  Also, I can manage to get bx, by, bh, bw and c1 , c2 , c3, but how
  do I extract pc for training from the dataset?

I will like to build a YOLO model from scratch but I don't know how to transform the dataset so that I can train a YOLO on it.


texts: how to create input file for collaborative filtering

texts:
http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html
https://github.com/apache/spark/blob/master/data/mllib/als/test.data

I want to find missing value in a matrix based on past matrices

But I find that the test.data are not matrices, instead a long list of 3 columns

How do I understand test.data and create my own data?

Which value represent missing value?

and how to understand the result, which value represent result?


texts: Predict the first observations of a time series when order of the model is higher

texts:
Suppose you have you have a time series with 365 observations, one for each day of the year, and you split the first 183 rows in training set and the latest 182 in test set.
Suppose you create an AR (autoregressive model), and you set the order of the model to 4. So you will have:
$y(t) = a1y(t-1)+a2y(t-2)+a3y(t-3)+a4y(t-4)$
In a situation like this, is it possible to do predictions on the first observation of the test set? That basically is the 184th row
I think no, because we do not have y(t-1),...,y(t-4) but we have only y(t)=value of 184th observation.
So, the first row we can predict is 188th, right? Because:
$y(t-1)=value$ of $187^{th}$ row
$y(t-2)=value$ of $186^{th}$ row
$y(t-3)=value$ of $185^{th}$ row
$y(t-4)=value$ of $184^{th}$ row
I think that until now is right. Correct me if I am wrong.
But if I want to predict the obs. 184, the first one of test set, is there any way? I mean, without decreasing the order of the model from 4 to 1 for example.


texts: I have multiple sequential data that lead up to an event. All I want to do if find the equivalent sequential data or a pattern that leads to the event. The data is also multivariate so I want to consider multiple features rather than just a column. Help will be greatly appreciated.


texts:
Suppose we have the 2 sets of time series data L1,M1 sampled at the same 1000 points of time t[t1,t2,...t1000]. Lets assume before hand that that the L1 can be given as
L1 = tX+a where X is the slope and a is the noise

and M1 can be give as
M1 = tY+b

Now I want to know how to formulate a parameter that measures the difference between slopes of L1 and M1 from their respective 1000 sampled points.

I am a novice when it comes to statistics, so any help or hints would be deeply appreciated.


texts:
I was reading about PAC framework and faced the definition of Generalization Error. The book defined it as:

  Given a hypothesis h ∈ H, a target concept c ∈ C, and an underlying distribution
  D, the generalization error or risk of h is defined by

  The generalization error of a hypothesis is not directly accessible to the learner
  since both the distribution D and the target concept c are unknown. However, the
  learner can measure the empirical error of a hypothesis on the labeled sample S.

I can not understand the equation. Can anyone please tell me how it can be interpreted? Also what is x~D?

Edit:
How do I formally write this term? Is something like $$\mathbb{E}_{x \sim D} [1_{h(x)\neq c(x)}] = \int_X 1_{h(\cdot) \neq c(\cdot)} (\omega) dD(\omega)$$
correct or do I need to define some random variable? Also, to show that the empirical error
$$ \hat{R}(h) = \frac{1}{m} \sum_{i =1}^m 1_{h(x_i)\neq c(x_i)} $$
is unbiased, we have
$$\mathbb{E}_{S \sim D^m} [\hat{R}(h)] = \frac{1}{m} \sum_{i =1}^m   \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x_i)\neq c(x_i)} \right] = \frac{1}{m} \sum_{i =1}^m   \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x)\neq c(x)} \right]$$,
but how do we formally get
$$ \mathbb{E}_{S \sim D^m} ~ \left[ 1_{h(x)\neq c(x)} \right]= \mathbb{E}_{X \sim D} ~ \left[ 1_{h(x)\neq c(x)} \right] = R(h)$$

I think that I understand it intuitionally, but I can't write it down formally. Any help is much appreciated!


texts: When does boosting overfit more than bagging?

texts:
If we consider two conditions:

Number of data is huge
Number of data is low

For what condition does boosting or bagging overfit more compared to the other one?


texts: Why is my PCA boomerang-shaped when normalizing?

texts:
Running an unsupervised plot of my data, I noticed a hyperbolic ('boomerang') shape:

dim=2

vectorizer = TfidfVectorizer(min_df=5, max_df = 0.4, stop_words = 'english')
train_tf_idf = vectorizer.fit_transform(bunch_train.data)
svd = TruncatedSVD(n_components=dim,random_state=42)
svd_train = svd.fit_transform(train_tf_idf)
svd_train = Normalizer().fit_transform(svd_train)

labels = {
        0:'alt.atheism',
        1:'comp.graphics',
        2:'sci.med'}
y = np.vectorize(labels.get)(bunch_train.target)

with plt.style.context('seaborn-whitegrid'):
    plt.figure(figsize=(12, 8))
    for lab, col in zip(('alt.atheism', 'comp.graphics', 'sci.med'),
                        ('blue', 'red', 'green')):
        plt.scatter(svd_train[y==lab, 0],
                    svd_train[y==lab, 1],
                    label=lab,
                    c=col)

    plt.title('2D SVD on TF-IDF - 3-NewsGroups',size=28)
    plt.xlabel('PC1')
    plt.ylabel('PC2')
    plt.legend(loc='upper left',prop={'size': 20})
    plt.tight_layout()
    plt.show()

I suspect it has something to do with the Normalizer - when deleting the following line:

svd_train = Normalizer().fit_transform(svd_train)

the data plots like this:


texts: How to rank documents using Bag of words approach

texts:
I want to cluster the documents I get for Google scholar search using the Bag of words model. I thought of using Java as the language.

Assume for the keyword k, Google scholar gives me 50 results. If I have a predefined set of words w1, w2, w3... wn, how can I rank the the documents which have the predefined set of words most? How can I apply Bag of words model for this? Do I need a clustering algorithm like k-means? And do I need to perform NLP techniques as well?

Say that the word w1 has several synonyms. How can I consider those synonyms as well for document ranking? Will I have to create a corpus containing all the abbreviations, synonym etc for that?

Are there any good tutorials available for this? Will choosing Java over Python will be an disadvantage since most of the resources (Ex - Scikit) were in Python?


texts: Are there any good sources that explain how decision trees can be implemented in a scalable way on a distributed computing system.  Where in a given source is this explained?


texts: How to compute precision and accuracy of a sequence that is not strictly binary?

texts:
Given a predicted sequence and actual sequence I want to compute it's precision and accuracy, for example: Note that these sequences will only contain 0, 1 or -1

predicted sequence: -1,0,1,1,-1,0,1,1,0,-1

actual sequence:     -1,1,0,1,-1,1,0,1,0,-1

I know that precision is computed using this tp/tp+fp and accuracy is computed using tp + tn /tp + tn + fp + fn. But because I have -1 in it I am unsure how I would compute true positives? My understanding that a true positive is if I predicted a 1 and it's corresponding actual value is a 1. A walk through of the computation for precision and accuracy would help.


texts: Keep word2vexc/fasttext model loaded in memory without using API

texts: I have to use Fasttext model to return word embeddings. In test I was calling it through API. Since there are too many words to compute embeddings, API call seems to be expensive. I would like to use fasttext without API. For that I need to load the model once and keep it in memory for further calls. How can this be done without using API. Any help is highly appreciated.


texts: Supervised multiclass classification : is ANN a good idea ? or use other classifiers?

texts:
I have a problem deciding what to use since i'm just beginning to creating predictive models.

Let's say I have a training dataset with 5 or 6 features and a testing dataset. (With around 50k rows in training / 5k in testing). My samples are people that I would like to assign to types of products they would buy. (more than 2 classes).

I'd like to know the whole process of what to use, and based on what exactly. Also, is there a serious difference between the results rendered by an ANN and other classifiers on this type of classification?

Note: I have 10 possible classes in the output

Thanks in advance.


texts: How to implement LSTM using Doc2Vec vectors to get representation?

texts: Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about step-by-step guide to build it. How LSTM can make representation with input is list vector of sentence trained by Doc2vec.


texts: GE Predix's machine learning (anomaly detection) capabilities

texts:
Background:
I am investigating time-series anomaly detection for industrial machine data, and have stumbled upon GE Predix. It seems like a promising tool, however, I am not familiar with their approaches for anomaly detection.

Question:
Does anyone have experience with GE Predix, in the anomaly detection space? How would one rate its anomaly detection capability? What technical approaches (algorithms) does it use? Any other literature about GE Predix that is openly available that I have missed (besides References below)? Are there any available benchmarks (e.g., versus Splunk + Prelert)? Who are its competitors?

References:
1) https://www.predix.com/sites/default/files/predixplatformbrief-march_2016.pdf
2) https://www.predix.com/sites/default/files/GE-Software-Modernizing-Machine-to-Machine-Interactions.pdf


texts: ValueError: Found input variables with inconsistent numbers of samples: [367, 24]

texts:
Working on a Time Series model that was 730 days with of revenue data and I have two questions.
First, I'm a little confused on how many days I should set my train and test split to. Currently, it's set to 365, but if any of you have advice on that, I'm all ears. I know on linear regression models, I usually don't do 50/50 split, but a couple of tutorials I've seen online are all over the place. That said, I also tried 100 days, etc.
Now for the error. I've pasted the full code below, but all the code works except for the last block and there I'm getting a ValueError: Found input variables with inconsistent numbers of samples: [366, 24]. I believe it's saying that my test and train set aren't the same lengths, but I'm confused as to why. My best guess would span = 12 and alpha = 2/(span+1) are causing it; however, as I've adjusted that code, that block gives me an error. Here's a sample of what my data looks like and the code will be below it.
Day       Revenue
1            1.1
2            1.3
3            1.7
4            1.4
5            1.5
6            1.2
7            1.7

train_data = data.iloc[:365]
test_data = data.iloc[365:]

span = 12
alpha = 2/(span+1)
simpleExpSmooth_model = SimpleExpSmoothing(train_data['Revenue']).fit(smoothing_level=alpha,optimized=False)
doubleExpSmooth_model = ExponentialSmoothing(train_data['Revenue'],trend='add',seasonal_periods=12).fit()
tripleExpSmooth_model = ExponentialSmoothing(train_data['Revenue'],trend='add',seasonal='add',seasonal_periods=12).fit()

predictions_simpleExpSmooth_model = simpleExpSmooth_model.forecast(24)
predictions_doubleExpSmooth_model = doubleExpSmooth_model.forecast(24)
predictions_tripleExpSmooth_model = tripleExpSmooth_model.forecast(24)

print('Simple Exponential Smoothing RMSE: {:.4f}'.format(np.sqrt(mean_squared_error(test_data,predictions_simpleExpSmooth_model))))
print('Double Exponential Smoothing RMSE: {:.4f}'.format(np.sqrt(mean_squared_error(test_data,predictions_doubleExpSmooth_model))))
print('Triple Exponential Smoothing RMSE: {:.4f}'.format(np.sqrt(mean_squared_error(test_data,predictions_tripleExpSmooth_model))))
````


texts:
I am going to perform A/B tests for ML models. However I am not sure how long should I run it online in order to see significant differnce. What would be the right time frame ? and what will be the reason behind the time frame ?
The A/B test will run againts the None ML systems. Usally we run for none ML features for 2 weeks max.
Thank you


texts:
I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After creating a dictionary, I'm running a script in Python to replace strings (words) with numbers that keras/theano will embed later.

The problem is that such a large dataset requires a lot of time for lookup. I would appreciate if anyone had suggestion on a tool for faster lookup or similar. Currently I just loop through every word in the corpus and replace it with the corresponding number from the dictionary (1-hot encoding essentially)

EDIT:

I'm doing roughly the following: each Python list is a sentence (before tokenization here):

['noble', 'interesting_superlatives',...,'the_idea']

which I want to conver to a list of integers, like:

[143599, 12387,...,7582]

I referred to it (probably incorrectly) as one-hot encoding because for each word there is exactly one number in the dictionary.


texts: Any research on relationship between the dimensions of a (word2Vec) space and how the human mind constructs meaning (or reality) through language?

texts: Neuroscience is still trying to "find" how the mind (and language) somehow "works". Is there any theory linking a (low-dimensionality) embedding space (like word2Vec) to a mind (linguistic) model? Any Cognitive Linguistics theory?


texts: I am currently studying Machine Learning course and now we are requested ( each student ) to find an Image dataset to work on for the course project. I am new to ML and it's now a week and I am trying to find a good dataset ( easy and not complicated ) to work on it , but all I found are ones for tough topics such as lung cancer detection, plant disease detection,...etc . So please someone help me to locate a good one for a beginner .


texts: Use KL divergence to label topics from LDA

texts:
I have used sklearn's sklearn.decomposition.LatentDirichletAllocation module to model 10 topics in a set of documents.
I have also used the same on a reference text (1 document) and obtained a 1 topic LDA for it.
I want to compare the 10 topics from the documents to the 1 reference topic and determine which one is most closely related to the referece.
I have been trying to figure out an implementation of KL divergence for this purpose but cannot seem to understand how to tie it in with the LDA outputs. Any help is appreciated.


texts: if new feature downgrade the score for xgboost what do I have to look at?

texts:
let say I'm predicting the housing price of Boston(kaggle).
if I got some score x

then I added new feature  y_K
if this new feature drop the score.
what is wrong with this feature and what do I have to look at?

if I know this process I can test all my features with this test


texts: Deep Learning: Feed Forward for Unbalanced Classes Using Tensor Flow

texts:
In theory, Deep Learning NN can predict a class with very few observations.  My problem, I have a class that happens less than 4% of the time.  Feeding the network data with distribution intact (96 one class and 4% the other class) results in the network predicting mostly the most common class.  Using under sampling result in a model that is able to predict 90% of the less common class, but still have a high False Positive.
I know that using F Score (with beta >= 2) in the training phase of the model will allow me to improve significantly the model performance. My question, how or where do I set my own performance metric in Tensor Flow?
Any suggestions are appreciated.
By the way, I did tune all the parameters (learning rate, momentum, etc..) performance increases but still not at the level I want.
My network has 20 inputs 5 hidden layers with 60 unites
My batch is 100 (tried up to 500), and I tried epoch from 100 to 1,200

The code is in Mathematica.

 netDrop1 =
     NetGraph[<| "i1" -> BatchNormalizationLayer[],
    "l1" -> CatenateLayer[], "l12" -> DropoutLayer[0.2], "l2" -> 60,
    "l21" -> DropoutLayer[0.5], "l3" -> Tanh, "l22" -> 60,
    "l221" -> DropoutLayer[0.5], "l33" -> Tanh , "l23" -> 60,
    "l234" -> BatchNormalizationLayer[0.5] , "l34" -> LogisticSigmoid,
    "l24" -> 60, "l2435" -> DropoutLayer[0.5], "l35" -> Ramp,
    "l45" -> 60, "l435" -> DropoutLayer[0.5], "l36" -> Tanh,
    "l55" -> 60, "l37" -> Tanh, "l4" -> 2,
    "l5" ->  SoftmaxLayer[]|>, {NetPort["Input1"] ->
     "i1" -> "l1" ->
      "l12" ->
       "l2" -> "l21" ->
         "l3" -> "l22" ->
           "l221" ->
            "l33" ->
             "l23" ->
              "l234" ->
               "l34" ->
                "l24" ->
                 "l2435" ->
                  "l35" ->
                   "l45" ->
                    "l435" ->
                    "l36" ->
                    "l55" ->  "l37" -> "l4" -> "l5", {{NetPort[
        "MaritalStatus"], NetPort["Gender"], NetPort["Input"]} ->
       "l1"}}, "Input" -> 11, "Input1" -> 2,
       "MaritalStatus" -> NetEncoder[{"Class", marital, "UnitVector"}],
       "Gender" -> NetEncoder[{"Class", {"M", "F"}, "UnitVector"}],
       "Output" -> NetDecoder[{"Class", {"no", "yes"}}]]

From reading publication about NN, I have decided to create a Dropout with = 0.2 at the input level, and at all subsequent levels to have Dropout at 0.5.
I have used both Tangent and ReLu (but found little differences in performance between them)
As you notice, a simple FeedForward Network

I do under sampling, the under sampling with great success:  the model is able to recall most of the yes, but still predict many negative samples as positive.

  <|{"yes", "no"} -> 972, {"no", "no"} -> 1082, {"yes", "yes"} ->
  81, {"no", "yes"} -> 3|>

By the way, I have used these same data,and used RandomForest with a utility function with great results http://femvestor.blogspot.com/search?q=geico


texts: Dealing with highly variable feature set size

texts:
I'm trying to use machine learning for security event classification. My goal is to predict the outcome (true positive or false positive) of a specific event. An event has a set of variables in it, called observables. These can be urls, ip addresses, file hashes etc. (8 types altogether). However, one event could contain a small number of observables or a huge number as well. Since I want to predict the outcome based on these observables, my feature size varies in length - between 1 and 2500.
This in an example of the data set:
['user1', '1.1.1.1', 'explorer.exe', NULL, NULL, NULL ...]
['google.com', 'msword.exe', NULL, NULL, NULL ...]
['user3', '1.1.1.9', 'explorer.exe', 'e0d123e5f316bef78bfdf5a008837577', 'http://google.com, NULL ...]

How can I handle this scenario? I'd like to try out a classification as well as neural networks as well for comparison.
Edit
I ended up using the Bag of Words approach, as the "observables" I mentioned could be interpreted as words in a document. From there my case is a relatively known text classification problem and I achieved good results with Naive Bayes algorithms and hash vectorization.


texts: Machine Learning Models for Non-Imaging Medical Clinical Data

texts:
I am a beginner in machine learning. Since machine learning is a wide area, I would like keep my focus on clinical data.

When I search for machine learning samples in healthcare, most of the examples are related to interpreting medical images like retina image recognizing retinopathy. What are the clinical relationships modeled in ML?


texts: How to fill an missing values in a column based on another column

texts:
I have a dataframe called shoes

Brand   Comment
Ugg       NaN
Prada     NaN
Clarks    NaN
Ugg       NaN
Clark     NaN
Prada     Made from horse leather
Prada     Made from pig leather
Prada     NaN
Ugg       Made from Australian cow leather
...

and another dataframe df_mode which was obtained by taking the mode of the comments for each shoe brand in the shoes dataframe for nonnull values

Brand  Comment
Ugg    Made from sheep
Prada  Made from pig leather
Clarks Made from Cow leather

How can I assign the missing values for each shoe brand in the shoes dataframe with its respective mode response shown in the df_mode dataframe.

This is basically what I'm trying to achieve

Brand   Comment
Ugg       Made from sheep
Prada     Made from pig leather
Clarks    Made from Cow leather
Ugg       Made from sheep
Clark     Made from Cow leather
Prada     Made from horse leather
Prada     Made from pig leather
Prada     Made from pig leather
Ugg       Made from Australian cow leather


texts: SVM prediction time increase with number of test cases

texts:
I am using scikit-learn's SVM for the MNIST digit classification dataset. In order to improve the performance I extended the dataset by adding rotated samples. I was aware that SVM takes O(N^3) time to train the data, where N is the number of training vectors.

However even prediction seems to take increase polynomially, the number of test vectors is the same. Is there any explanation for this or some equation that relates prediction time to the number of training samples?

I am using a 3rd degree polynomial as the kernel with C=100.0.

Note: I am doing a group project to compare the performance of various methods so I can't use any other method as my teammates would have used those. I referred to a paper by Decoste and Scholkoph which uses Virtual SVM. However I don't think I can run this on my current system if I can't run a simple extended training set.


texts: What are "VGG54" and "VGG22" derived from the VGG19 CNN?

texts:
In the paper Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network by Christian Ledig et al., the distance between images (used in the loss function) is calculated from feature maps extracted from the VGG19 network. The two used in the article are (a bit confusingly) called VGG22 and VGG54.

What are these feature maps?

What do the designations "22" and "54" mean?


texts: How to get feature importance from RandomForest using scikit-multilearn library?

texts:
I am working on multi-label classification problem, binary case. As a target variable there are five columns with 0-1 values.

For a model training I use scikit-multilearn library. Below is my code for training a model with Binary Relevance and RandomForest.

parameters = [
{
    'classifier': [RandomForestClassifier(random_state=42)],

    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_features': ['auto', 'sqrt', 'log2'],
    'classifier__max_depth' : [4, 6, 8, 10, 12],
    'classifier__criterion' :['gini', 'entropy']}]

clf = RandomizedSearchCV(BinaryRelevance(), parameters, scoring='f1_weighted', cv=5, \
n_jobs=-1, verbose=10, random_state=10)
clf.fit(X_train, y_train)

How could I extract feature importance from the model?
Should I take the best parameters from this model and then apply to data with each label separately using RandomForest from sklearn?


texts:
Imagine I'm collecting some training data. Lets say I collect a 30minute time series from 1000 people so I have 1000 observations (rows) in my feature matrix.  I train some model (lets say a neural net for this example) and I find my AUC is really poor and I believe the problem to be the fact that I only have 1000 observations - so I don't have enough data

However, I am now unable to collect anymore data. One thing I could do, is take that 30minute time series, and slice it up into 30 1-minute sections. Then use these 1 minute series as rows in my data. So I would end up with 30 observations per person, and 1000 people, giving me 30000 rows in my feature matrix. I've now increased the size of my training set by 30x

If I were to do this when doing statistical/inferential tests I would be violating the assumption of independence, and I'd be forced to model this with some multilevel model to correct

Is this the same as the IID assumption in machine learning, where each of your rows must be independent of each other?

For inferential tests, (one of) the reason why this assumption matters is because it affects the Type 1 error of your inference. However, in machine learning we're not doing any inferential tests, so what effect does violating IID actually have on results? In other words, why do rows need to be independent of each other? Especially in a case like above where I can drastically increase the size of my training set by reusing different parts of 1 person's data


texts: Accuracy goes straight for about 200 epochs then start increasing

texts:
Can anyone explain the following observation?
Why did the accuracies keep to be a straight line with a very smooth decrease of loss?
By the way, why is the loss lines so beautifully smooth for the first 400 epochs?
Is this because of the learning rate or other reasons?


texts: What does the descision boundary of a relu look like?

texts:
A single non activated neuron is just a linear combination of its inputs.

Thresholding this neuron's output as-is against 0 would create a hyperplane binary separator, whose parameters can be learned.

What would be the shape of this separator (decision boundary) in case we also take a relu on the output and only then threshold?

I keep thinking it has to be non linear, otherwise NNs wouldn't work, but if something was positive before, it would remain positive, and if it were negative or zero, it would stay, and so the relu actually did nothing for the decision boundary, which makes no sense to me.

If someone could make order of this for me I would be glad.

As a follow up, I would like to understand multi level descision boundaries  - say on a 2 level network. What would the boundary look like with 2 neurons per layer, on a 2 layer network, for, say, a XOR-like dataset?


texts: I've read about FaceNet but the main problem is still unclear for me. Does embedding work on the trained images only? Or once trained on a big dataset it will readily cluster unknown faces without re-training? I need an approach to add new faces automatically.


texts: Memory allocating error with coxph model

texts:
I'm working with cox models and as I am adding more variables, I am facing a memory issue. I tried to subset my dataframe by selecting columns which have variables of interest but even then I am facing this problem. How can I overcome that?

My data has around 451,557 observations with 270 variables. 70 Percent of variables are categorical (strings).

coxdf_2<- coxph(surv_df ~ NS_group + age_at_recruitment + Gender + Alcohol_drinking +
                              Smoking + Diabetes + BMI +  +Hypertention,
                method="breslow", data = DFMODEL)

  Error: cannot allocate vector of size 16.1 Gb

coxdf_3<- coxph(surv_df ~ NS_group + SOC+ towndep_I + Education+age_at_recruitment +
                          Gender + Alcohol_drinking  + Smoking + Diabetes + BMI +
                          +Hypertention,
                method="breslow", data = DFMODEL)

  Error: cannot allocate vector of size 24.4 Gb


texts: Confusion matrix. "How close I am to the diagonal?". Is there such metric?

texts:
I have a question regarding confusion matrices.
To start, we discuss the case of multi-class classification so the confusion matrix has dimension, for example 4 times 4, for classification task with 4 possible outcomes.
The correct predictions should bring all the answers on the diagonal. Is it possible to rate though how close are the results on the diagonal? For example, is much better to predict a class very close to the current one (so guess 3 instead of 4) in comparison to predict wrongly a further away class (predict 4 while it was 1).
Is there a metric that can take that into consideration? Rate kinder classes that are have missed the ideal target by a bit and penalize stronger misclassification that are further away from the real class?
I have been looking a lot about this on the internet but no luck so far.
Any suggestions on how to proceed?