texts: multivariate clustering, dimensionality reduction and data scalling for regression

texts:
I have a dataset with approximately 20000 observations consisting of 40 independent and 1 dependent variable. My initial objective is to develop a model that will predict the dependent variable. I have tried several models and applied linear regression and other algorithms such as Random Forests, of course by splitting the dataset into training and testing sets.

Unfortunatelly I cannot get any meaningful results; I have very large errors. I believe there is something "messy" with the dataset, so I have decided to do some clustering first and then apply regression within each cluster. Considering that my dependent variable may exhibit a lot of variation I believe I should do clustering with all variables (dependent and independent), so as each cluster will have similar values of my dependent variable. I have tried to apply Kmeans and I faced several problems. First of all, it seems I cannot identify the right number of clusters. The "elbow" method gives an unclear number and when I use it with less data (about 2000 observations) I get something like this:

I also had similar problems with hierarchical clustering. I have already tried to apply regressions within the clusters identified, but the results are still very poor.

Right now I believe I should possibly use some kind of "weight" to my data, in order to put more weight on the dependent variable when I do clustering, since I believe that this is the problem. Hence, my questions here are:

is there a way/algorithm where I adjust weights in the variables to be clustered?

Moreover I am confused with two more issues:

data scaling:
is it necessary to scale the data before clustering? does this guarantee more accurate results? when do we scale the data?
dimensionality reduction:
I have read a lot about principal component analysis and dimensionality reduction, but I am still confused. Again; is this necessary? how many variables are too many to consider applying PCA? are 10 variables too many? or maybe 20? or 50? when should we apply dimensionality reduction? a problem with PCA is that I would still need my original variables to extract the coefficients after regression, while to my understanding with PCA I cannot do that.

This question is more about discussion in order to understand some particular concepts and find a solution to my problem and does not refer to coding issues. Any example and/or references would be appreciated though. I am coding in R.


texts:
My dataset is comprised of vector sequences. Each vector has 50 real-valued dimensions. The number of vectors in a sequence range from 3-5 to 10-15. In other words, the length of a sequence is not fixed.

Some fair amount of the sequences (not vectors!) are annotated with a class label. My task is to learn a classifier that given a sequence of vectors, the class label for the whole sequence is computed.

I cannot tell the exact nature of the data but the nature of sequences is not temporal. Nevertheless, a vector $x_i$ cannot be interchanged with a vector $x_j$ without changing the label ($i \neq j$). In other words, the order of vectors is important. The vectors themselves are comparable, for example it makes sense to compute a dot product and use this similarity value.

My question is: what are the tools/algorithms that can help to classify such data?

UPDATE:
The data has such a property that one or very few vectors influence strongly the class label.

POSSIBLE SOLUTION:
After some research it looks like the Recurrent Neural Networks (RNN) fit the bill pretty naturally. The overarching idea is to pick a context size $k$, concatenate word vectors, do max pooling and feed that through classical NN. At each possible context window position in a sentence, a feature vector is built. The final feature vector is built using max pooling for example. The backpropagation is done to adjust the network's parameters. I already got some positive results (GPU is a must).


texts: Analysis of railway data - Detecting outliers

texts:
I would like some pointers about the following problem:

I would like to detect anomalies in a pretty huge collection of railway data. Or create a baseline model for detecting future anomalies. The data I have at my disposal exists out of coordinates and speed at the given coordinate (also the time of measurement). Could this perhaps be approached as a regression problem where there's a clear(?) connection between location and speed of a train. For instance a train suddenly moving at snail's pace on a track that from historical data is considered high velocity could be a potential anomaly. If this could indeed be approached in such a way would something like SVM be an option or should I look into other algorithms?


texts: xgboost cannot identify perfectly fitting regression line

texts:
For a dataset I want to use xgboost for the optimal ensembling of $n$ forecasts instead of just using their arithmetic mean for combination. I found that xgboost generates forecasts that are worse than many of the $n$ individual forecasts the moedl could choose for combination.

I do not know why this can be the case. For the illustration of my observation I created the toy dataset below. The artificial target variable is generated by $$y = \frac{x_1+x_2}{2} \, \,\mbox{with } x_1, x_2 \sim N(0,1) $$
Given the deterministic relationship between $y$ and the two explanatory variables $x_1$ and $x_2$, xgboost could make perfect forecasts, but it does not. The linear model easily does. Since this is the most simple multivariate linear regression model I can think of and xgboost fails I wondering about the implications.

Why is this the case? What are the limitations of tree models for regression?
Why is then xgboost used for stacking and ensembling of forecasts if it cannot reproduce the MSE minimizing arithmetic mean as optimal combination mechanism?

Note that the parameters of xgboost do not affect that. I tried many parameter settings and the results are never perfect.

Data Generation

library(tidyverse)
library(xgboost)
n <- 1000
param0 <- list("objective"  = "reg:linear", "eval_metric" = "rmse")
set.seed(1)
df <- tibble(x1 = rnorm(n), x2 = rnorm(n), y = (x1+x2)/2)

xgboost

xgtrain <- xgb.DMatrix(as.matrix(df[1:900,c("x1","x2")]), label = df$y[1:900], missing = NA)
xgtest <- xgb.DMatrix(as.matrix(df[901:1000,c("x1","x2")]), missing = NA)
#Crossvalidation just to illustrate that the algorithm
#learns something that is not correct since the test data
#cannot be forecasted with 0 error.
#xgb.cv(nrounds = 100,nfold = 10, params = param0, data = xgtrain)
#nrounds and other parameters do not not get you to the prefect forecast
model <- xgb.train(nrounds = 100, params = param0, data = xgtrain)
preds_xgb <- predict(model, xgtest)
#no perfect forecasts
sqrt(mean((preds_xgb-df$y[901:1000])^2))
0.04654448

Linear regression

model <- lm(y ~ x1+x2, data = df[1:900,])
#0.5 and 0.5 for x1 and x2 as expected
model$coefficients
preds_lm <- predict(model, df[901:1000,c("x1","x2")])
#perfect forecasts
sqrt(mean((preds_lm-df$y[901:1000])^2))
1.389314e-15


texts:
You split the data in k subsamples. Train it on k-1 subsamples, test it on kth subsample, record the performance with some error merric.

Do it k times for each of the k subsamples, record the error each time.
Then choose the model with the lowest error?
Is it the same as ensemble technique?


texts: Visualizing Portfolio Asset Allocation over Time

texts:
I've been trying to figure out a good visualization for seeing how the asset allocation within a portfolio changes over time (due to daily returns/losses or cuts to exposure of specific asset). Traditionally, I've seen stacked area charts used to illustrate this, but I find that the viewer is unable to quickly glance at such a chart and understand how the share of the portfolio has changed over time. See my image below to understand what I mean:

Some changes in asset allocation are easy to see (early 2017 and late 2018), but others are difficult to pinpoint (January 2019, for example).

Does anybody know of any alternative charts which can show such an asset allocation time series visualization more effectively?

Thanks in advance.


texts: What if irrelevant features impact the outputs?

texts: We know that weather conditions in China is not very useful to predict price of a house in Spain (as life experience). However, when we drop that China's weather condition, the accuracy is reduced largely. Will we keep it?


texts: Multi-Output Regression with neural network in Keras

texts:
I have got an .xlsx Excel file with an input an 2 output columns. And there are some coordinates and outputs in that file such as:
x= 10 y1=15 y2=20
x= 20 y1=14 y2=22 ...
I am trying to do that regression using tensorflow. But somehow i can't manage to do it. I am leaving my code here, I would appraciate it if someone could help! I also have test datas ready as well.

training_data = pd.read_excel(...\training_data.xlsx',sheet_name="i1-o2")

training_data_X = training_data['i1']

training_data_Y = training_data[['o1','o2']]

testing_data = data = pd.read_excel(....\testing_data.xlsx',sheet_name="i1-o2")

testing_data_X = testing_data['i1']

testing_data_Y = testing_data[['o1','o2']]

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(6, activation='linear')
])

model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])
model.fit(training_data_X,training_data_Y,epochs=10,batch_size=100)

val_loss,val_acc = model.evaluate(testing_data_X,testing_data_Y)
print(val_loss,val_acc)


texts: Ensembling vs clustering in machine learning

texts:
Following the raise of ensembling (e.g ensembling of xgboost learners) after its recurrent wins in Kaggle competitions, using it has become a tradition in machine learning. However, some argue that clustering is a form of ensembling where we apply clustering on a data set first, then applying a learning algorithm (to do classification or regression) on each of the resulting clusters.

I would like to understand how clustering is different from ensembling?

EDIT

I understand that clustering can be similar to the idea of ensembling if we simply consider "homogeneous ensembling" where we train the same model on each of the clusters. But I am looking for potential limitations of clustering where using heterogeneous ensembles (different learns for all the data) is the "only" way to go about the problem.


texts: Can Facebook's Prophet be considered a “type” of AI? Like and CNN or RNN

texts:
I'm facing a problem to determine if the model Prophet developed by Facebook team is some kind of Neural Network or similar.
Someone has an answer for it?

In Github repository has the following explanation: "Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality(...)".

But reading the Prophet paper, by Sean J. Taylor and Benjamin Letham. It really seems to be and Machine Learning algorithm.

Even the way that use it seems to be a Neural Network.

Any discussion or reference would be apreciate


texts: Why does my GridSearchCV always break up?

texts:
GridSearchCV for my Random Forest breaks up. I need to know the reason and the solution to make it work:
# Grid-Search for Random Forest
param_grid = {
    'bootstrap': [True],
    'n_estimators': [100, 200, 300, 400, 500],
    'max_depth': [50, 100, None],
    'max_features': ['auto', 200],
    'min_impurity_decrease':[0],
    'min_samples_split': [2, 5],
    'min_samples_leaf': [2, 5],
    'oob_score': [True],
    'warm_start': [True]
}

# Base-Model for improvement
rf_gridsearch = RandomForestRegressor(random_state=42)

# Grid-Search initiation
rf_gridsearch = GridSearchCV(estimator = rf_gridsearch, param_grid = param_grid,
                           scoring = 'neg_mean_absolute_error', cv = 5,
                           n_jobs = -1, verbose = 5)

# Perform the grid search for the model
rf_gridsearch.fit(X_train, y_train)
```


texts: Dealing with zeros when plotting log-scaled data

texts:
I have a non-negative variable and I'd like to plot it, log-scaled
I'm trying to understand how to deal with 0-values. One naive idea I had in mind is just to add 1 to all values (or some very low number greater than 1)
What other options are available?
Thanks


texts: how to create outputs for key points of bounding boxes on image in Neural network in Python

texts:
I've made my first convolutional neural network which can classify one object on image (220, 220). The objects classes are only two - dogs and cats. I have 3385 '.jpg' pictures of dogs and cats on my PC. I have information about class of each object (1 or 0). Also I have information about two corners coordinates (from 0.0 to 1.0) of bounding boxes (0.0 is left/top corner of image and 1.0 is right/bottom corner of image) around of the objects. Convolutional net must have 5 outputs (1 output for classes and 4 outputs for two corners coordinates: x1,y1,x2,y2).
I have 3047 train images 'x_train' shape (3047, 220, 220, 3) and labels information in 'y_train' shape (3047, 5). Also 338 test images in 'x_test' shape (338, 220, 220, 3) and test labels in 'y_test'. For example, y_test[0] has such elements array([1.        , 0.555     , 0.18      , 0.70833333, 0.395     ]).
But my neural network have such prediction results prediction[0] is array([276.47525 ,  51.961666,  20.63266 , 125.419945, 106.816345],dtype=float32). And np.sum(prediction) is 338.0 which is number of test images. During the training parameters increased from  'loss: 4.8534 - accuracy: 0.5938' to 'loss: 533785.6875 - accuracy: 1.0000'. Could anyone help me with an advice what do I do wrong?
#Summary:
#Model: "sequential_1"
#conv2d_2 (Conv2D)            (None, 220, 220, 32)      896
#max_pooling2d_2 (MaxPooling2 (None, 110, 110, 32)      0
#conv2d_3 (Conv2D)            (None, 110, 110, 64)      18496
#max_pooling2d_3 (MaxPooling2 (None, 55, 55, 64)        0
#flatten_1 (Flatten)          (None, 193600)            0
#dense_2 (Dense)              (None, 128)               24780928
#dense_3 (Dense)              (None, 5)                 645
#Total params: 24,800,965
#Trainable params: 24,800,965
#Non-trainable params: 0

import os
import cv2
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
import numpy as np
import matplotlib.pyplot as plt
from tensorflow import keras
from tensorflow.keras.layers import Dense, Flatten, Dropout, Conv2D, MaxPooling2D
import time

model = keras.Sequential([
    Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=(220, 220, 3)),
    MaxPooling2D((2, 2), strides=2),
    Conv2D(64, (3, 3), padding='same', activation='relu'),
    MaxPooling2D((2, 2), strides=2),
    Flatten(),
    Dense(128, activation='relu'),
    Dense(5, activation='relu')

])

print(model.summary())
model.compile(optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy'])
his = model.fit(x_train, y_train, batch_size=32, epochs=2, validation_split=0.5)
model.evaluate(x_test, y_test)


texts: How to track user given some guaranteed unique but deletable data and some possibly conflicting but non-deletable data?

texts:
I am trying to track users reliably on my website so that if they are abusive, they can be banned and not come back easily (obviously this can be bypassed with TOR and such, but most trolls don't care that much). I have some data that can be set from my server onto the browser and is guaranteed to be unique because I have set the data myself. However, this data is also easily deletable (clearing cache, clearing cookies, clearing other site data). I have some other data (fingerprinting data) that is extremely difficult to remove or change but can easily conflict with other devices which have the same fingerprint. Here is the data I could collect:

Storage/caching (guaranteed unique, deletable): Cookies, HTTP ETags,
localStorage, indexedDB, FileSystem API, Last-Modified date echoing

Fingerprinting (possible collisions between devices, difficult to delete):
canvas2d, WebGL,
font list, screen size,
IP address, plugins installed

And a bit more. All I know so far is:

Trust the unique identifiers first
    If unique identifiers conflict, choose the majority
    If I identifier a user based on his/her unique identifiers but their fingerprints are different, update their default fingerprints to match

Can you help me decide what to do in any  given situation, e.g., a user has deleted all of his/her storage/caching data but the fingerprints remain? What happens when fingerprints conflict in some ways but not in others? What are the probabilities that this is a new user or a returning user?


texts: Faster-RCNN how anchor work with slider in RPN layer?

texts:
I am trying to understand the whole Faster-RCNN,

From  https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work

  Then a sliding window is run spatially on these feature maps. The size of sliding window is n×n (here 3×3). For each sliding window, a set of 9 anchors are generated which all have the same center (xa,ya)(xa,ya) but with 3 different aspect ratios and 3 different scales as shown below. Note that all these coordinates are computed with respect to the original image.

It is much more clear than other articles for my opion, but still hard to understand how the feature map generate.

I saw another flow pics :

The problem, I write below steps for example:

If input is 600x1000x3 pic
Through VGG16 convnet , layer 13 output feature map is 40x60x512
Use a 3x3 sliding window, generate 1x1x512 feature map ???

Here, how 3x3 sliding window use a set of 9 anchors ???

Sorry, I am really new to object detection and image proccessing.

I only have a little understand about the steps, I known 9 anchor shapes(not the real anchor) are used to generate a lot of anchors(2400*9 in this case).

I can only imagine that use 9 anchor shape to slide the original image to get the all IoU . I don't understand how to use 3x3 sliding window in conv feature map here.

I know how anchors be selected,  2400*9 -> ignore cross-boundary -> 6000 -> apply NMS -> 2000 ,  in each minibatch, it randomly choose 512 anchors from 2000.

What I can't understand is 3x3 slide with 9 anchor shape . Because from original paper, anchors with is 16, height from 11 to 273 . I don't think it use the 13 layer conv output feature map to calculate IoU . Anchor must be apply in original image,  so what is the 3x3 sliding window doing ??


texts: How to change data source names in Google Data Studio

texts:
Google Data Studio has connectors to MySQL, PostgreSQL, etc. but these connections will come with default names.
Couldn't find out how to set the names for data sources in Google Data Studio. Is it even possible?


texts: Predicting household energy consumption?

texts:
I have a fairly simple dataset of energy consumption values generated every half hour. I want to train a model to predict the energy consumption at a particular time.

How do I model time values?


texts: estimate user's satisfaction of a video based on how much of it they watched - normalization

texts:
I am trying to estimate how much a user liked a video using how much of the video they watched.
Let's say, on the scale of 1 to 10, 1 means that the user didn't like it at all, and 10 means they enjoyed it a lot.
For instance, if a user watched 8 minutes from a 10-minute video, it means the score of 8. If they watch 18 minutes of a 20-minute video, it means the score of 9.
The problem is, the probability of a short video (say 1 minute) being completely watched is much higher than that of a long video (say 120 minutes). It doesn't necessarily mean that the user liked it more. It just was short.
I am looking for an equation to consider the length of the video in the process of making the estimated score.
I came up with this:
raw_score - (raw_score / log10(video_length))

"raw_score" is a the estimated score mention above (1 to 10) and "video_length" is the length of the video in seconds. "log10" is the base 10 logarithm.
However, this results in drastic penalties.
For instance, it would reduce the score from 10 to 5 for a short video.
I am looking for some way to normalize this penalty so that the amount that a score gets reduced can be limited to a specific range, for instance at most 2 points.
What is the best way to tackle this problem?


texts:
I am new to pytorch, i want start from a simple example-linear regression:
I created some random training and test sample.
here is my code:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as opt
from numpy import random
from util import *

np.random.seed(100)

class MLP(nn.Module):
  def __init__(self):
    super(MLP, self).__init__()
    self.l1 = nn.Linear(100, 100)
    self.l2 = nn.Linear(100, 1)

  def forward(self, x):
    x = (self.l1(x))
    x = (self.l2(x))
    return x

mlp = MLP().float()

target = nn.MSELoss()
o = opt.SGD(mlp.parameters(), lr=0.02, momentum=0.9)

w = torch.tensor(np.random.rand(100, 1) * 3).float()
x = torch.tensor(np.random.rand(100, 100) * 100).float()
y = torch.mm(x, w) + 2

test_x = torch.tensor(np.random.rand(100, 100) * 10).float()
test_y = torch.mm(test_x, w) + 2

for epoch in range(100):
  op = mlp(x)
  if epoch == 10:
    print(op)
    sys.exit(1)
  o.zero_grad()
  loss = target(op, y)
  loss.backward()
  o.step()
  test_pred = mlp(test_x)
  print(test_pred.shape)
  print(test_y.shape)
  print('%dth: loss=%.4f os_loss=%.4f'%(epoch, loss.item(), target(test_pred, test_y).item()))

I found it loss become nan, when several rounds passed.
i cant find out why, i think my netual network framework is correct, can you help on this?


texts: Scaling negative and positive variables when performing a k-means cluster analysis

texts:
I'm looking to perform a k-means cluster analysis on a set of data that contains variable ranges that contain both positive and negative values. Given the rangers vary so much the data will need to be scaled, but my concern is with the variables that contain negative value ranges. Should I perform some sort of log transformation on all the date so as to scale the data to positive values. For example:
Variable A: 3.4, 5.6,1.3,7.6,8.3
Variable B: 1,2,3,2,1
Variable C:-1.3, -1.4, -2.3, -4.2, -1.3


texts: Confusion around categorical cross entropy

texts:
I understand the binary cross entropy formula for a problem with a single label 0 or 1. If we have more than 2 labels we sum this binary cross entropy over all these classes.

$$ H_{y'}(y) := - \sum_{i} \sum_{c} ({y_{i, c}' \log(y_{i, c}) + (1-y_{i, c}') \log (1-y_{i, c})}) $$

I always thought the above was categorical cross entropy but it turns out its the following:

$$ H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i) $$

I'm pretty sure the two formulas are actually different but I think both would work. I realise the second formula wouldn't work on a problem where the labels are not mutually exclusive (label vector can have multiple 1's), but still I don't see why we can't just use the first formula for all problems whether the labels are mutually exclusive or not? What is the need for the second formula and why is it proffered over the first?


texts: XGBoost: Can the features in test data be a subset of the features used to train the model?

texts:
Is it a problem if the test data only has a subset of the features that are used to train the xgboost model? All my predictor variables (except 1) are factors, so one hot encoding is done before converting it into xgb.DMatrix. So the different levels of the factor variables become the features and my test doesn't have all of these features, only a subset of it.

At the moment, while running my model on test data in R, I 'm running into an error saying that "Features names stored in object and newdata are different!".

I'm new in the field, so any help would be much appreciated. Thanks!


texts: What is missing from the following Curriculum Learning implementation in a Deep Neural Net?

texts:
First of all we have a classification task.
So we use the typical softmax cross entropy to classify.

Current implementation of curriculum learning is as follows.

First we train our best version of the neural net
At the last epoch we get all of the cross entropies for all the instances. So smaller cross entropies are better classified and larger ones not so well.
Then we sort the instances based on the cross entropies.
Then we start training the instances from easier to harder as the curriculum learning theory suggests

Note that we have already experimented with various steps and repetitions. So in one example we took the first 200 batches and trained them two times before going to the next batch and so on until an epoch is completed.
In another example we took the first 10 batches and trained them only once  and then the next 10 and the next 10 and so on until the end of the epoch.

All of the experiments so far have concluded that the neural network is has a relatively ok accuracy at the beginning and this gets worsen as the more difficult instances come along.
The final accuracy is much worse than expected and in addition the maximum accuracy is still quite bad.

Why is this curriculum learning not working? Is anything missing?


texts: Is a good shuffle random state for training data really good for the model?

texts:
I'm using keras to train a binary classifier neural network. To shuffle the training data I am using shuffle function from scikit-learn.
I observe that for some shuffle_random_state (seed for shuffle()), the network gives really good results (~86% accuracy) while on others not so much (~75% accuracy). So i run the model for 1-20 shuffle_random_states and choose the random_state which gives the best accuracy for production model.
I was wondering if this is a good approach and with those good shuffle_random_state the network is actually learning better?


texts: Dealing with small number of examples in hierarchical text classification

texts:
I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict the super class and then the sub class. To achieve this i do the following:

fit one model to the data to predict the super class;
then in a loop fit a separate model for each of the super classes to predict the sub class.

Unfortunately, some super classes have extremely small number of examples - up to 1 example in a class! Surely i cannot build a model predicting sub class on one example. The only thing one can do is to drop such super class from the data set. One can try building a model on 2 examples (1 - for train, 1 - for test sets and predict always the same unique sub class for this super class). Restricting the number of examples to 3, 5, 10, ... the model for such low-populated super classes will start making at least some sense.

Dealing with this problem of small number of examples is a trade-off between model usefulness and model performance. On one hand, i do need these low populated super classes, because when i get next real data set - there for sure will be examples from such 'trashy' super classes and I do want to be able to predict a sub class for such examples. On the other hand, I simply cannot apply the classifier to the data set because I either cannot properly split it into train and test set (ex.: super class has 100 examples, out of which 1 example is of sub class A and 99 examples are of sub class B) or model performance is low due to its disability to learn good on small number of cases.

The question is - what are typical approaches to solve such problem?

are there any rules of thumb on the minimum (threshold) value for number of examples to have in a super class to built a more or less reliable model predicting its sub classes?
should i apply oversampling for low-populated super/sub classes? And will it help to at least deal with train/test split issue?
extra: suppose i drop examples with low-populated super classes. Then i predict probability on a new test set --> the model will give some (maybe high) probability that the example belongs to well-populated super class X, but in reality it belongs to some low-populated super class that i dropped on training stage. Should i just live with it? Or there is a way to determine from probability distributions across super classes that the model doing wrong?


texts: RDD of gziped files to "uncompressed" Dataframe

texts:
I have a bunch of compressed text files each line of which containing a JSON object. Simplified my workflow looks like this:

string_json = sc.textFile('/folder/with/gzip/textfiles/')
json_objects = string_json.map(make_a_json)
DataRDD = json_objects.map(extract_data_from_json)
DataDF = sqlContext.createDataFrame(DataRDD,schema).collect()
'''followed by some transformations to the dataframe'''

Now, the code works fine. The problem arises as soon as the number files can not be evenly divided between executors.

That is as far as I understand it, because spark is not extracting the files and then distributing the rows to the executors, but rather each executioner gets one file to work with.

e.g If i have 5 files and 4 executors, the first 4 files are processed in parallel and then the 5th file.

Because the 5th is not being processed in parallel with the other 4 and cannot be divided between the 4 executors, it takes the same amount of time as the first 4 together.

This happens at every stages of the program.

Is there a way to transform this kind compartmentalized RDD either into a RDD or Dataframe that is not ?


texts: Cross-validation for Timeseries Counterfactual Analysis

texts:
We are looking to predict counterfactual states from time-series data. In our problem we are looking to determine the energy savings from a grid-installed device that is varied on and off for many substations and feeders. Our methodology (open to critique) is to fit a model on hourly load for many substations and feeder's data of a years, then nudge the feature related to the device being on or off to get energy savings for a full timeperiod. So we

train the model on all the existing data (kW ~ x1 + x2 + ...)
make a prediction for the device being off at all timestamps (producing kW at every timestamp)
make a prediction with the device on for all timestamps (producing kW at every timestamp)
Subtract the "off" predictions with the "on" predictions.

While a lot of this is up in the air, my main question right now is related to lag terms and hyperparameter tuning. Typically if we are predicting the future we aim to have no information from future timesteps in the train data during hyperparamter tuning. We include lagged weather terms from the past. Some of us think that we would not be able to include lagged features in our training set unless the observations they refer to are actually in the training set, otherwise information is leaking. Is this concern sound?

Thoughts on how you would approach this and reasoning?

More info: we are using random forest and xgboost. Did this n the past with k-fold CV, but reassessing.


texts: How to train NLTK Sequence labeling algorithm for using custom labels/Train set?

texts:
I am trying to recognize and classify the entity types based on the IOB/Sequence labeling.

I was able to use nltk.ne_chunk() which is already trained to recognize named entities using their train set.

I was wondering if there is any way to train the algorithm using my on training set and labels using training data like the following,

(
send        O
sms         B-TASK
to
8714349616  B-MOB
how         B-MSG
are         I-MSG
you         I-MSG
)

(
sms        B-TASK
how        B-MSG
are        I-MSG
you        I-MSG
to         O
8714349616 B-MOB
)
etc..

is it possible using NLTK ? Any examples or Tutorials ?


texts: Dichotomising a Categorical Variable in R into 1-2 (not 0-1)

texts:
I wondered if someone could help regarding a set of commands for re-coding an ordinal categorical variable into a dichotomous variable, where the binary values are '1' and '2' rather than '0' and '1'.
Suppose I had a data set called 'trial' and within the data set there were five categorical variables. Call them: 'L', 'M', 'N', 'O', 'P'.
Each of these five variables, has six categories {0,1,2,3,4,5} representing health states (zero meaning no health issues - five representing poor health).
I'd like to take variables (L, M, N, O, P) within the data set 'trial', and dichotomise them so that 1 = {0,1,2}, and 2 = {3,4,5}.
Any help would be much appreciated. The 'sjmisc' package has several dichotomosing functions for splitting into groups; but again this is only zeros/ones.
https://cran.r-project.org/web/packages/sjmisc/vignettes/recodingvariables.html
All guidance would be appreciated.


texts: One Hot Encoding where all sequences don't have all values

texts: Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder and numpy's to_categorical only account for the values in the current sample so for example, encoding DNA sequences of 'AT' and 'CG' would both be [[1, 0], [0, 1]]. However, I want A, T, C, and G to be accounted for in all sequences so 'AT' should be [[1, 0, 0, 0], [0, 1, 0, 0]] and 'CG' should be [[0, 0, 1, 0], [0, 0, 0, 1]].


texts: Random Forest in R with only character variables

texts:
I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent variable (1 vs. 0) I have other variables which have different levels such as professional status (retired, employed, unemployed), searching for (myself, parent, other) and region (north, west, south), etc...

summary(data) provides me with the information that the class of my variables is character (dependent variable shows min, 1st Qu, Media - so I assume R reads it as continuous?) and I believe that a character variable needs to be factored before I can run the randomForest command. Is there a single command that transforms all character into factors?

My second questions is whether I should remove the id of the customer from my imported table, or whether it will affect the results if I keep it in the RF model?


texts: Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

texts:
I want to store JSON Data into MySQL Database using Python.
I used dataframe of pandas. I found that to_sql() can do this job easily.

Python Code:

jdata=json.loads(json_data)
df=pandas.DataFrame(jdata)
df.to_sql(con=con, name='crashTable', if_exists='replace', flavor='mysql')

Output:

TypeError: to_sql() got an unexpected keyword argument 'flavor'

I got this error message. How to resolve this error?


texts: How to describe the process of using existing weights for training a deep learning model?

texts:
I have weights which I use for initializing the training of a convolution neural net model as follows (in Keras):

model = load_model("my_weights.hdf5")

Now I want to rerun the entire model with further data augmentation and training using these loaded weights. I would like to know if this is a feasible approach to apply. And also, would this be considered as fine-tuning? or transfer learning?


texts: whether data science will cool down in the years to come

texts: Data science jobs are everywhere and popular. Laymen, leaders etc talk about it, without knowing it. Will data science cool down in the years to come?


texts: How to more simply see overlapping data for dozens of variables?

texts:
I'm trying to think of the best way to see how multiple variables (about 40) related to a very large userbase can be seen to interact with one another.

As an analogy, imagine I had survey data of superheros liked by thousands of school children, such that Adam Agner only likes Batman, Brian Agosto likes Superman and Batman, Cian Ailiff likes Wonderwoman, Superman and the Flash, etc. The index would be the list of children's names (tens of thousands), and the variables would be each superhero (from a list of 50) each with a True or False value.

Rather than just the amount of children who like each superhero, if I can see the overlap information in some easy way, I might find that an unusually large number only like Batman, or that most of those who like Superman are likely to also like the Flash.

The easiest would be to do it visually, but Venn diagrams wouldn't be practical with a large number (50 here) of variables. So it would seem like a correlation matrix with a heat-map base would be the way to go, with something like this:

I can imagine a heat map with superheros plotted on both axes could work for seeing interesting matches of one variable against all other ones, but it would still require some less ideal extra steps to see when three or more superheros matched, like seeing the colour overlap of superhero1 with superhero2 & superhero3 both be high, and that's kind of guesswork, because the actual children who like all 3 could still be low.

At the moment, the best solution I can think of is to try to reduce my variables down into categories, so the example could be Marvel, DC, male, female, but that loses some potential data that could be helpful from overlaps within those categories.

Maybe if I did something like the image above, the circle size would be the number for that overlap, and the colour could be number matches with other variables (without listing them, but I could further investigate). That kind of coding is a little out of my comfort zone, but I could try.

Ideas appreciated! I'd ideally do this in matplotlib or some other means within python, but if I have to use Matlab or another tool then I will consider that. Ideas or suggestions appreciated! Hopefully this was clear, thank you!


texts: How to improve accuracy of a binary classification in neural network

texts:
I tried to program a binary classifier to predict whether a customer belongs to one class or another.

I have over 200k feature vectors, consisting of numeric 12 features each and assume that should suffice. Of these 200k, 120k are features for the one label, 80k are features for the other label. My problem now is that I cannot get the accuracy of this classifier over 70%. Sure, this is better than chance, but I hoped for something a little more... accurate? Here is my code for the network:

# I have 200k feature vectors with values each;
# these features include age values between 20 and 80,
# but also boolean values, so 0 and 1. Therefore I apply a scaler.

minmaxscaler = MinMaxScaler(feature_range=(-1, 1))
minmax_features = minmaxscaler.fit_transform(features)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(5, input_shape=features[0].shape, activation='relu'))
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))

model.compile(optimizer='adam',
             loss='binary_crossentropy',
             metrics=['accuracy'])

BATCH_SIZE = 10000

early_stop = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=2, verbose=1)

hist = model.fit(minmax_features,
                 labels,
                 epochs = 100,
                 shuffle=True,
                 steps_per_epoch=len(features)/BATCH_SIZE,
                 validation_split=0.2,
                 validation_steps=len(features)*0.2/BATCH_SIZE,
                 callbacks=[early_stop],
                 verbose=1)

Adding another layer or changing the number of nodes on the layer did not change much about the outcome. Does anybody have any advice how I can raise the accuracy of my network?


texts: How word2vec can handle unseen / new words to bypass this for new classifications?

texts:
In simple terms, if my classification is based on word2vec as features, what I am supposed to do, if a new word comes, which does not have a word2vec?
I am trying to used word2vec or word vectors for classification based on entity.
For example:
I have to classify the following words in a sentence as:
"Google gives information about Nigeria"

Here, I would like to classify Nigeria as location.
Suppose I have good word2vec vectors for each of the words, based on some readings I came to know that, recurrent neural networks can be used for this. So, word2vec will capture most locations with a kind of similar word vectors.
But my questions are:
a) Suppose a new location is there. lets say, Russia . So, do I need to assign a new word vector for this location ?
b) If my input for training does not have grammatical sense. For example,
" Google information Nigeria " . Everything else Nigeria is associated with a non-location label. Will this condition work for find new location in non-grammatical sentences.


texts: High accuracy on test-set, what could go wrong?

texts:
You are given a pre-trained binary ML classification model with 99% accuracy on the test-set (assume the customer required 95% and that the test-set is balanced). We would like to deploy our model in production. What could go wrong? How would you check it?

My answer was that the model could be biased towards our test set and could fail to generalize on yet to be seen data. We can check this by running the model on multiple unrelated test-set that it haven't seen before.
Is this the right angle?


texts: Dynamically create dependent column based other column values in dash

texts:
Guys i have this table in dash the following columns are drop down columns Reason Code, Staging and Overwrite. Based on their values i want to create values under Final Staging column dynamically in dash

Here is my code

app.layout = html.Div([
dash_table.DataTable(
    id='table-dropdown',
    data=staging.to_dict('records'),
    columns=[
        {'id': 'customer_id', 'name': 'Customer ID'},
        {'id': 'booking_date', 'name': 'Booking Date'},
        {'id': 'oustanding_balance_(currency)',
         'name': 'Outstanding Balance(currency)'},
        {'id': 'booking_date', 'name': 'Booking Date'},
        {'id': 'past_due_days', 'name': 'Past Due Days'},
        {'id': 'segment', 'name': 'Segment'},
        {'id': 'contract_rate_%', 'name': 'Contract Rate(%)'},
        {'id': 'fee/commission_rate_%', 'name': 'Fee/Commission Rate(%)'},
        {'id': 'collateral_(force_sale_value_(currency)',
         'name': 'Collateral(currency)'},
        {'id': 'collateral_(type)', 'name': 'Collateral(Type)'},
        {'id': 'maturity_date', 'name': 'Maturity Date'},
        {'id': 'repayment_years', 'name': 'Repayment(Years)'},
        {'id': 'quantitative_assessment', 'name': 'Quantitative Assessment'},
        {'id': 'reason_code',
            'name': 'Reason Code', 'presentation': 'dropdown'},
        {'id': 'staging',
            'name': 'Staging', 'presentation': 'dropdown'},
        {'id': 'overwrite',
            'name': 'Overwrite', 'presentation': 'dropdown'},
        {'id': 'final_staging', 'name': 'Final Staging'},
    ],

    editable=True,
    dropdown={
        'reason_code': {
            'options': [
                {'label': i, 'value': i}
                for i in staging['reason_code'].unique()
            ]
        },
        'staging': {
            'options': [
                {'label': i, 'value': i}
                for i in staging['staging'].unique()
            ]
        },
        'overwrite': {
            'options': [
                {'label': i, 'value': i}
                for i in staging['overwrite'].unique()
            ]
        }
    }
),
html.Div(id='table-dropdown-container')

])


texts: From analysing their previos transactions how can I predict for what type of product is the customer more likely to take take an EMI

texts:
So basically I need a kind of product/ category affinity for EMI for all customers  eg - Customer A is more likely to take an EMI on her insurance premium.
One approach I had thought was to broadly categorize the the transactions into 3-4 categories and predict the amount( linear regression)  that a customer is going to spend on each of the category in that particular month.eg It is estimated that customer A will spend huge amount on category 1 in month of September, then this customer A will be targetted for a marketing campaign for taking EMI for products in category 1 in September
Another approach I had thought was to do a multi class logistic regression to classify each customer into 3-4 category and marketing can target customers for EMI of specific products
Or I could do clustering and check the conversion for EMI of different types of categories in each cluster
Please let me know if I'm going in the right direction or if there is any other approach I could take or any algorithm I can use.
Thank-you


texts: Do xgboost and random forests in general handle multiple splits of the same numeric feature in a single branch?

texts:
For example, let's say that the age (say $x$) of a person for $12< x< 25$ can be used to predict computer usage to a high degree of certainty. In a decision tree, this could be represented by a split of $x<25$ followed by a split of $x>12$.

Does xgboost, or other decision tree learning algorithms in general, handle multiple splits within the same branch?


texts: How to split temporal sequences to sub-sequences in a meaningful yet unsupervised manner?

texts:
I have a biological process that undergoes some cellular event which I am observing.
I have a series of events, with different temporal gaps between them.
For example

30, 0, 65, 0, 100, 0, 300, 25, 5, 5, 5, 235, 0, 5, 480, 0

That is, 30 seconds between the first event and the second one, the third even occurring together with the second one (activation events can occur in different locations in the cell and I classify them as distinct events), the forth even occurring 65 seconds after the third event, and so on.

I assume that one event can trigger the next one (spatial distance also has an effect but we have decided to ignore it at this point) - forming a sub-sequence of events. Or the events can be non-related, and thus belong to different sub-sequences. I assume that this can generally determined by temporal distance.

The question is - how?
I know that the temporal distances between different sub-sequences should be larger than those between events in the same sub-sequence, but I still need to decide what the threshold is.

Seeing as I don't know what a good separation would be, I think that an unsupervised approach might be useful here.

However beyond that I do not have any idea how to approach that.

Do any of you have any methodologies suitable for this endeavor, or any other insights and tips?

Many thanks.


texts:
I have some doubts regarding the approach to build a classifier such as Multinomial Naive Bayes or SVM. I will go through the steps to see if the approach is fine. I have not a lot of experience in model build, so any suggestions would be great!
My dataset has approx. 1115 obs having positive value (0) and 66 obs having negative value (1).
The distribution of the dependant variable is shown in the figure below.
I split the dataset into train (70) and test (30), using stratify (it should help in case of such discrepancy between classes, hopefully):
from sklearn.model_selection import train_test_split

y=df['Label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

Then I imported the sim model to create a SVM classifier:
from sklearn import svm

clf = svm.SVC(kernel='linear')

clf.fit(X_train, y_train)

And to Predict the response for test dataset I used the following:
y_pred = clf.predict(X_test)

For accuracy calculation, I used the following code:
from sklearn import metrics

print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

print("Precision:",metrics.precision_score(y_test, y_pred))

print("Recall:",metrics.recall_score(y_test, y_pred))

getting different values every time I re-run it:
Accuracy: 0.75

                precision    recall  f1-score   support

           0       0.95      0.79      0.86       316
           1       0.08      0.30      0.13        20

    accuracy                           0.76       336
   macro avg       0.51      0.54      0.49       336
weighted avg       0.90      0.76      0.82       336

2nd re-run
Accuracy:  0.8005952380952381

                precision    recall  f1-score   support

           0       0.94      0.84      0.89       316
           1       0.07      0.20      0.11        20

    accuracy                           0.80       336
   macro avg       0.51      0.52      0.50       336
weighted avg       0.89      0.80      0.84       336

Confusion Matrix:
 [[265  51]
 [ 16   4]]

3rd re-run
Accuracy:  0.7797619047619048

                precision    recall  f1-score   support

           0       0.94      0.81      0.87       316
           1       0.08      0.25      0.12        20

    accuracy                           0.78       336
   macro avg       0.51      0.53      0.50       336
weighted avg       0.89      0.78      0.83       336

Confusion Matrix:
 [[257  59]
 [ 15   5]]

I have a couple of questions on these results and I hope to find answers to them:

Which value should I take into account for saying that my model has accuracy of ...?
Does it make sense to run a model where there are so a few values = 1 for the dependent variable?


texts:
I'm currently working on a prediction for fiction. I have a database with fiction, which are each described with different story tags. My idea is to use a neural network that can tell you by processing a new story which tags are relevant.
The problem is, that the original data wasn't generated but added by users. A story in the woods could be tagged with trees, nature etc. Another story that also takes place in the woods might not be tagged with nature, even though the tag applies. This might confuse the neural network. Is there a way to prevent this form happening?

Thank you!


texts: Suitable metric choice for imbalanced multi-class dataset (classes have equal importance)

texts:
What type of metrics I should use to evaluate my classification models, given that I have two imbalanced multi-class datasets (21 and 16 classes, respectively) where all classes have equal importance?
I am somehow convinced with macro-averaged-based metrics choice such as macro F1 and macro TNR, ...etc. Are macro-averaged-based metrics suitable for my problem based on the aforementioned inputs?


texts: Order SparseVectors by the closest distance to given SparseVector

texts:
I have a Spark dataset containing a column of SparseVector types. Additionally, I have another SparseVector $X$ which is not a part of the dataset. I want to order my dataset according to the closest distance (or similarity) relative to $X$.
Can anyone help me with how to implement this?


texts: Convolution Neural Network Loss and performance

texts:
I have a set of about ~100,000 training examples. Ratio of positive to negative example is roughly 1:2. The true ratio is more like 1:100 so this represents a major downsampling of the negative class. It is also a very noisy dataset - the examples were automatically generated through distant supervision. Each example represents a set of sentences and has 700 columns. Number of rows may vary from 10 to 100 (maybe even more).
I used a Convolution Neural Network in Tensorflow to train my model (model architecture similar to the one described here)  with only 2 epochs and stored the loss, f-score , precision and recall every 10 steps. I evaluated the model on a validation set (which too was generated automatically through distant supervision with negative class downsampling resulting in pos:neg ratio of ~1:2) every 100 steps. Here are the hyperparameters:
batch size: 60 for train, 100 for validation
epochs: 2
convolution filter sizes: 700x5, 700x6, 700x7, 700x8, 700x9, 700x10
number of convolution filters per filter size: 125 (so total of 750 filters)
dropout: 0.5
l2reg: 0.001
lr: 0.001

I'm seeing some strange behavior with the model and I don't understand why.
My training precision, recall and f-score go over 0.95 in about a 100 steps (6000 examples) and then plateaus. The loss falls down from 0.8 to 0.2 in about 200 steps and then fluctuates between 0.1 and 0.4.
On the validation set my precision, recall and f-score are over 0.95 starting from the first time I evaluate it on the 100th step. Loss fall slightly from 0.3 to 0.2.
When I evaluated on a real-world test set (without downsampling negative class so it has the true ratio of pos:neg), the actual precision and recall were 0.37 and 0.85.
My results are not making any sense to me. I use tensorflow metrics for calculating training precision, recall and fscore and scikit-learn metrics for calculation validation precision, recall and fscore. I can't find anything wrong in the code but I don't understand why I should have such results unless there is a bug. I would have understood having low precision and recall all through - the class imbalance favors the negative class and my set is noisy. However, I am very confused about why I'm having such misleadingly high scores all through..
Given that my dev dataset is also noisy and generated in the same manner as the train set, the dev results might just be useless and it is possible that the model is overfitting the noisy set. But I still don't understand why the scores are so high so soon. Also, if overfitting is the issue, do you think I should make the dropout even higher?
I've attached a screenshot of the graphs and would really appreciate your thoughts on this. Blue is train and red is dev. Thanks a lot!


texts:
I'm working on a classification project. I have many rows, containing many binary attributes, some of which are often appearing together, exactly like what we can encounter in the Market Basket problem (in which you can, for example identify, that if you buy 'Milk' to a supermarket, you also have a more than random chance to buy 'Eggs').
My idea is then to take my target as an attribute, extract best Item-set containing my target (so having Target=1, exactly like my previous example if I wanted to make a model to predict whether the client will buy eggs or not).
Then, looking at the fit of each item-set, I'll have the info of which groups of products have best chances to make my target be 1.
Even though the method looks good, I have a problem. I'm wondering if there's a way to only extract item-sets with target on them.
All articles I found about making classification based on association rules explain that one of the big downside of the method is time taken, because it has to find all item-sets, and after removing all the ones that are not including target.
Is there a way, using item-set properties, to specify the algorithm (Apriori, FP-Growth, etc.) to only calculate Item-sets including Target, and not waste time calculating all item-sets?


texts: Selection of co related variables for ridge regression

texts:
I'm a newbie to machine learning. I'm working on a dataset to predict a target variable in terms of the independent variables. In the dataset the independent variables few are very highly correlated . correlation coefficient is 1.

I used ridge regression since it will take care of the correlated variables.

The algorithm is penalizing all the correlated variables and taking only one among them.

Can someone throw light on how the variable selection is done for ridge regression in presence of correlated variables.


texts: using simple autoencoder for feature selection

texts:
I am using a simple autoencoder to extract the informative features and I have multiple Q:

I know that the features extracted will be a linear combination of the original features so I consider that the feature that has a larger mean weight (has the highest percentage in the formation of new features) will be important so I will take that features but I don't know if this is true or not

the second things is that I want to apply the grid search to find the optimal hyperparameters for the model but I can't do that please if anyone can help me in this and save my life


texts: Distance between very large discrete probability distributions

texts:
I have 192 countries where each country has some value for 1 million attributes which sum up to 1 (a discrete probability distribution). For any one country most of the values for the attributes are 0.

Now I am trying to find the distance/similarity between those countries using these attributes. I know we can use  Jensen Shannon Divergence between two discrete probability distributions to get a distance measure, but the caveat is that all the values have to be non-zero.

Given that there are zero valued attributes for the countries, is there any other suitable statistical distance measure that can help me to cluster these countries using these 1 million attributes?


texts: how to programmatically introduce grammatical errors in sentences

texts:
I've a set of sentences in English language. I'm exploring ways to create a dataset of sentences with grammatical errors programmatically. The following options has been tried out randomly -

identify verbs, propositions etc. by POS tagging and change the tense or remove them
change the order of 2 or more words
remove commas, colons, semi-colons etc.

These are not always fool-proof. Are there any proven ways to approach this problem?


texts: Intuition of group parameter in Caffe Layer

texts:
I am confused about the group parameter in caffe convolution layer.

Image I have an input of 20x3x227x227 and filter size is 96x3x11x11 with stride=4 and padding=0, if I set the group=2. Then the setting becomes:

group1 : 20x1x227x227, filter1: 96x1x11x11
group2 : 20x2x227x227, filter2: 96x2x11x11

is that correct? If that is correct, how can you concatenate them to the output 20x96x55x55?


texts: What is the correct format of the test input for LSTM neural network?

texts:
I have learned some tutorials for LSTM time series prediction. According to that tutorials, My training data format would be like,

NOTE :
This data is only for example. this is not the real data.

x  y
x1 x2
x2 x3
x3 x4
.  .
.  .
x9 x10

I  am using the basic LSTM Model,

model = Sequential()
model.add(LSTM(len(input), input_shape=(1, look_back)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')

So after model.fit(). I am confused a bit how to test predict the next value of the series. I don't know How to find the next time series Data x11 ?

I also tried with shown below inputs to find the x11,

Methods:

[x10]
Or
[ x2,x3,x4,...x10]

As far as i learnt method 1 is right. But Method 2 provides better accuracy than Method 1. So I don't know which way is correct to predict the next time series data ?

Thanks in advance.


texts: How to create custom Keras metric using multiple functions with numpy arrays and matrices?

texts:
I'm training a model for predicting position and orientation using regression.
I want to implement custom metric using last 3 values of model output to calculate and minimize angular error defined by function like this:
def angular_error(data_true, data_pred):

  # Axes orientation in radians (X,Y,Z)
  axes_orientation_gt = np.transpose(data_true[2:5])
  axes_orientation_pred = np.transpose(data_pred[2:5])

  axes_size = 0.1  # Axes_size in meters

  # Convert euler angles to rotation matrix (XYZ)
  _, Rx_gt, Ry_gt, Rz_gt = eul2rotmat(axes_orientation_gt)

  # Rotation matrix XYZ
  R_gt = Rz_gt * Ry_gt * Rx_gt

  # Y-axis end
  y_a_gt = R_gt * np.transpose(np.matrix([0, 1, 0])) * axes_size

  # Predictions
  # Convert euler angles to rotation matrix (XYZ)
  _, Rx_dnn, Ry_dnn, Rz_dnn = eul2rotmat(axes_orientation_pred)

  # Rotation matrix XYZ
  R_dnn = Rz_dnn * Ry_dnn * Rx_dnn

  ....

  return gaze_ang_err

Is there a way to avoid using Keras/TF functions instead of every numpy function appearance/usage? This function also calls two more functions with numpy and math operations so it seems like an overkill.
Any input appreciated...


texts: Vector Autoregression forecasting with large dataset

texts:
I am trying to use VAR to forecast electricity price for a whole day and I have a dataset with over 20000 observations (price for every hour) from 2015-2017.

My first intuition was to select 19975 observations as train:
model = VAR(train) and model.select_order() (looking at the information criterion - AIC,BIC etc..) function to select the number of order. After training I would evaluate it by forecasting it with rolling forecasting.

However, I have read that when training your model it is better to train it with the most recent observations instead of using the entire dataset. But I have also read that the more past data you have the better your forecasting.

My question is should i use a small number of recent observations instead and if so how much ?

Thank you very much


texts: Need to prepare the data to Link Analysis project?

texts:
I've a dataset with following schema:

Customer_ID - Unique ID
Product - ID of purchased product
Department - ID of the department that sells the product
Product_Type - The purchased product type
Date - The date of purchase
Quantity - The number of units purchased

I need to do a link analysis project to analyze some consumption patterns of the products and answer the following questions:

"If product B is purchased then customer will also take product A"

I will use Scala/Python to make the link analysis over the datasets but the  examples I have seen are dataset with direct links to the project of "Flight Data" in the schema is:

ID
Origin
Destination

My question is: there I need to prepare my dataset to make the Link Analysis (there exists some best practices to do this?) or can I analyze the dataset with that structure?

Many thanks! Sorry my inexperience on this topic!


texts:
I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form.

First if we assume,

p('to go') = count('to go') / sum(all lemmas)

p('went') = count('went') / sum(all words)

Breakpoint here, since every word comes with its respective lemma, we have the condition that

sum(all lemmas) == sum(all words)

The joint probability is also a little hard to normalize

# count of "went" being lemmatize to "to go
p('went', 'to go') = count('went'->'to go') / sum(all words)

The catch here is since "went" always lemmatize to "to go", we have special condition of

count('went'->'to go')  == count('went')

The reverse is not true though since the "to go" can be realized as different surface forms.

In that case, it's really awkward, if we put it all together

 p('went', 'to go') = p('went') = count('went') / sum(all words)
 p('to go') = count('to go') / sum(all words)

 PMI('went', 'to go')
 =  p('went', 'to go') / (p('to go') * p('went')
 =  p('went') / (p('to go') * p('went')
 =  1 / p('to go')

But that would assign the same information value for all surface forms given the same lemma.

I think I've made some mistakes in how I'm accounting for the joint or independent probabilities. Could anyone advise how to approach an information score for lemma and its surface?

Or is the information score between lemma and its surface uninformative thus the conclusion of inverse probability of the lemma?

One possible solution is to change the normalization value for the joint probability where:

p('went', 'to go') = count('went') / count('to go')

Then the PMI equation would be:

PMI('went', 'to go')
=  p('went', 'to go') / (p('to go') * p('went')
=  (p('went') / (p('to go')) / (p('to go') * p('went')
=  1 / p('to go')**2

Still the final mutual information disregard the surface probability =(


texts:
I'm trying to implement an algorithm to find the minimal value of a function.

Before moving to sigmoid activation functions, i'm trying to understand linear regression.

Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.

Say I have two vectors:

x=[1,2,3,4,5,6,7,8,9,10,11,12]

y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]

Between these points, I would like to add a linear separator with least squares.

Say I have some imperfect linear function:

$f(x)=0.026x+2.3$

As I know, there are two ways to find this:

$w = (X^TX)^{-1}X^Ty$

and a gradient descent algorithm:

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

Although for linear models finding derivative is trivial, thus second method is not necessary.

Now i've used the first equation on the vectors, in Python:

w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y

Unfortunately, the output was irrelevant:

[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]

Then i've tried using the second method for 500 iterations, in Python:

for i in range(1,5000):
    x_old = x_new
    x_new = x_old - v*dydx
    print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))

However, i'm not sure how to know when it reaches a convergence point.

How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?


texts: Plotting ROC curve in cross validation using Matlab perfcurve

texts:
I have the following code for a binary classifying using SVM, and 10 cross-validation,

k=10;
cp = classperf(lables);
cvFolds = crossvalind('Kfold', lables, k);
 for i = 1:k
  testIdx = (cvFolds == i);                %# get indices of test instances
  trainIdx = ~testIdx;                     %# get indices training instances

   svmModel = fitcsvm(data_features(trainIdx,:), lables(trainIdx),
   'Standardize',true,'KernelFunction','RBF','KernelScale','auto');

  pred = predict(svmModel, data_features(testIdx,:));
  cp = classperf(cp, pred, testIdx);
end
acc= cp.CorrectRate;
conf= cp.CountingMatrix;

I want to Plot the ROC curve using the  perfcurve function  in Matlab, However, the input 'score' changes each fold and can't be used outside the k-fold loop.

[X,Y] = perfcurve(labels,scores,posclass)

Any suggestion on how to plot a ROC in such case?


texts: Recommender System for mostly unique user and items

texts:
I am trying to develop a recommender system for a job matching problem. My data consists of past matched candidate profiles and job profiles as well as if there was a success such that both, candidate and employer, accepted the offer.
Now, what I want is to build a recommendation system for future candidate and jobs which will recommend a fitting candidate to an open job posting based on past successes.
The big problem here is, that each candidate and each job is unique. So, if a matching was successful, both candidate and job won't be available anymore. As for recommender systems, this is a huge problem, since both, collaborative filtering and content-based filtering, rely on user ratings/interaction and similar users, who already rated items for which a recommendation should be calculated. But these items don't exist anymore. What I have here is basically a cold-start Problem for user and items.
One way I could tackle this problem would be to user a knowledge-based recommender system, since I have candidate and job profiles describing skill, position, experience and so on. I could construct common feature vectors and simply calculate similarity between them, recommending the top n most similar candidates to a job posting. But I want to use also the knowledge of the past job-matchings since they contain information about what kind of candidate was suitable for a specific job.
Does anybody have an idea how I could do this or had a similar problem?


texts:
I'd like to apply some of the more complex supervised machine learning techniques in python - deep learning, generalized addative models, proper implementation of regularization, other cool stuff I dont even know about, etc.

Any recommendations how I could find expert ML folks that would like to collaborate on projects?


texts: how to derive the steps of principal component analysis?

texts:
I'm trying to understand the theoretical reasoning behind the method, but I can't understand a particular step in the middle of this page.

"The constraint on the numbers in $v_1$ is that the sum of the squares of the coefficients equals $1$. Expressed mathematically, we wish to maximize
$$\frac1N\sum_{i=1}^NY_{1i}^2$$

where
$$y_{1i}=v_1'z_i,$$

and $$v_1'v_1=1$$

(this is called "normalizing" $v_1$).
Computation of first principal component from $R$ and $v_1$. Substituting the middle equation in the first yields

$$\frac1N\sum_{i=1}^NY_{1i}^2=v_1'Rv_1."$$

I don't understand how $R$ suddenly appeared in this equation. The right hand side "$v_1' R v_1$" seems to have appeared out of nowhere.

Please help


texts:
I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability boundary (The split between stable and unstable regions of a certain system) that can be extracted from the surrogate model.
I have used SVM (Regression) previously to create a surrogate model of a 'system' based on a Latin Hypercube Sampling plan to which I was able to extract the stability boundary, although this required many sample points (some even redundant) and thus a high computational cost.
Essentially, my previous work treated my SVM model as black box with no prescribed intelligence.
My aim is to use an adaptive sampling strategy, with the aim of focusing ONLY on the stability boundary as opposed to creating an accurate surrogate of the WHOLE system, to increase the 'intelligence' of my SVM model. Are there any suggestions to any strategies in research/literature relevant to SVM?
i.e. Contour location is of interest, contrast to a global surrogate.


texts:
I have certain data , in that I have labels and formed a groups , Also with the same data without using any labels I used clustering option that is available on tableau . So, Now I want to check the accuracy of the cluster that is formed without any labels. How to compare and find the accuracy percentage of the visualizations that is formed with labels and without labels (clusters) .??
Example : label | feature 1 | feature 2 | feature 3|
0 0.568 0.765 0.3454
1 0.123 1.34 2.567
Here the graph shows the visualizations with labels
And one more graph is plotted without labels but did clustering to see to what category that the features belong to .
How to compare these two graphs and get the accuracy rate of the clustered that are formed?


texts: How to Convolve a High-Res Image by a (fully convolutional) CNN kernel?

texts:
My CNN is an extremely simple neural network.
input_img = Input(shape=(80, 80, 1))

x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

decoder = Model(input_img, decoded)
decoder.compile(optimizer='adadelta', loss='binary_crossentropy')

As you can see, it is a fully convolutional neural network. Once it get trained, can I use this CNN as a kernel for convolution operated on a large, high-res image, meaning can I do something like this?
output = scipy.signal.convolve(target image (4k x 4k), CNN (trained on 80 x 80 pix), mode='same')

I need this to do image segmentation. I know there are U-nets to do segmenttion, but my problem doesn't require such a big network, and I need something extremely light and fast.


texts: How many images per class are sufficient for training a CNN

texts: I'm starting a project where the task is to identify sneaker types from images.  I'm currently reading into TensorFlow and Torch implementations. My question is: how many images per class are required to reach a reasonable classification performance?


texts: Estimating Propensity Score via Regression Trees (in R Using rpart)

texts:
I am trying to estimate propensity scores in R. By this I mean that I am trying to estimate probability that an individual selects into treatment, where the selection into treatment is a binary variable. Typically people use logit or probit to estimate these. I was thinking of using regression trees.

Is it okay to use rpart "anova" method to estimate propensities? I think I cannot use classification tree because that gives a binary outcome whereas I want the continuous propensity score function i.e. how does probability of treatment changes as the continuous explanatory variables changes.

I am using the following command where x and z are continuous explanatory variables and d is a binary treatment variable and all 3 are observed for each individual and I want to estimate P(D=1|X,Z).

  cart = rpart(d~x+z, data=data, method="anova")

  data[,pz_c:=predict(cart, type = "vector")]

where the first command estimates the propensity and the second adds the predicted propensities to the data set for each individual.

I originally posted this on stackoverflow and was told to post thee question here instead.


texts: Unevenly stretched sequences with LSTM/GRU

texts:
I am looking for a right NN architecture(probably, based on LSTM/GRU) for the classification problem I faced. I have an alphabet of events {A, B, C, D, ..., N} and sequences of these events for each training example. It is obviously that Embedding + LSTM/GRU gives me some baseline model, but my problem is a little bit more complex.

I also have timestamps attached to each event, so the same sequence in terms of order my represent different scenarios in reality for example {B, C, C,  <2 days after> A } and {B, <2 days after> C, C, A }. I tried to replace embedded vectors representing events with just events' timestamps(or duration from previous event) in the LSTM/GRU layer input and it gave me also some model with 75%-80% quality from the baseline model(and about ~6% of examples were correctly classified which were not correctly classified in baseline model). Now I am looking for an NN architecture to combine these two datasets(sequence of events and sequence of timestamps). I have already tried the following approaches:

Simply concatenate timestamp information into embedding vectors(with
and without training), for example if embedding size = 32, we would
have 33 dimensional vector for LSTM input - 97%-100% quality from
the baseline model
NN with 2 inputs for event sequence and for
timestamps sequence with concatenation before LSTM - ~90% quality
from the baseline model
NN with 2 inputs for event sequence and for
timestamp sequence with 2 LSTMs and concatenation after - ~95%
quality from the baseline model

Thanks for any comments or recommendations.


texts: Do i need to use hyperparamters from Gridsearch to train on WHOLE training set to get final model?

texts:
I just want to make sure i am on the right lines so please correct me if wrong. I am testing which hyperparmets are best for logisitic regession on my data X, y where X is featrues and y is target. X, y are made from my training set. I also have a test set.

from sklearn.linear_model import LogisticRegression

# split train into target and features
    y = Train['target']
    X = Train.drop(['target'], axis = 1)
    X = pd.get_dummies(X)
#split test data into target and features

y_test = Test['target']
X_test = Test.drop(['target'], axis = 1)
X_test = pd.get_dummies(X_test)

logistic = LogisticRegression()  # initialize the model
# Create regularization penalty space

    param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] }

clf=GridSearchCV(logistic,param_grid=param_grid,cv=5)

best_model = clf.fit(X, y)# View best hyperparameters
print('Best Penalty:', best_model.best_estimator_.get_params()['penalty'])
print('Best C:', best_model.best_estimator_.get_params()['C']) #

I will now use these hyper parameters and 'train' it on my training data. Just so i'm sure when we say train do i then take best_model and train on the whole X data. Then i use my X_test which is my hold out data on this new trained model:

bestLog=best_model.best_estimator_
trained_model=bestLog.fit(X,y)
predicted=trained_model.predict(X_test)

then use this output above as my final model to test?


texts: Concatenating Encoder hidden states in LSTM pytorch

texts:
I am implementing a seq2seq autoencoder in pytorch:
Q1) While it is true that we can keep the encoder as bidirectional, but can we keep the decoder as bidirectional as well(does it make any sense) if we are not using teacher forcing?
if we cant, then let us suppose the architecture is:
encoder = nn.LSTM(128, 128, layers = 2, bidirectional=True)
dencoder = nn.LSTM(128, 128, layers = 2, bidirectional=False)
here 128 is the input and output dim of both the LSTM. The encoder hidden output will be of size (4, 1, 128) following the convention(2(for bidirectional)*num_layers, batch_size = 1, 128)
Q2) Now I wanna know that among these 4 tensors of size (1, 128) which tensor is the hidden output of which layer and of which direction from the encoder.
considering the complete output of encoder being:
encoder_output, (encoder_hidden, encoder_state) = encoder(input_tensor)
My basic assumption is:
encoder_hidden[0, :, :] is from encoder layer 1 forward direction
encoder_hidden[1, :, :] is from encoder layer 1 reverse direction
encoder_hidden[2, :, :] is from encoder layer 2 forward direction
encoder_hidden[3, :, :] is from encoder layer 2 reverse direction
but I have no way to confirm this.
then, let us suppose I have Identified this and let us say that this assumption above of mine is correct. Even then I have to somehow pass these encoder hidden outputs as thought vector to the decoder(which has 2 layer but is unidirectional), so it expects only two tensors to be passed as the initialized hidden state inputs (one tensor for each layer).
Q3) Then how should I convert 4 tensors to 2. Some Ideas are:

pair up by concatenating any two vectors among these 4 to convert them to 2 tensors and make the decoder hidden dimension as double the size of the encoder hidden dimension while instantiating the decoder.
pair up by averaging or adding any two tensors among these 4(the decoder input dimension here stays the same)
follow 1 and then pass them to a linear layer to map them back to the original decoder hidden dimension.

I have confused about for a long time, any help is appreciated. Thanks.


texts: Why should I normalize also the output data?

texts:
I'm new to data science and Neural Networks in general.
Looking around many people say it is better to normalize the data between doing anything with the NN. I understand how normalizing the input data can be useful.

However I really don't see how normalizing the output data can help.

I've also tried both cases with a easy dataset, and I achieved the same results. The only difference is that in some weird problems, it is really hard to then re-convert the output back.

Can you give me some intuition on why we should also normalize the output?

Or maybe why it is indifferent?


texts:
Does this line in Python indicate that KNN is weighted?

clf = KNeighborsClassifier(n_neighbors=5, metric='euclidean', weights='distance')

Are the weights the inverse of the distance?

Can anyone also give an example of how weighted KNN works mathematically?


texts: Tensorflow Adjusting Cost Function for Imbalanced Data

texts:
I have a classification problem with highly imbalanced data. I have read that over and undersampling as well as changing the cost for underrepresented categorical outputs will lead to better fitting. Before this was done tensorflow would categorize each input as the majority group (and gain over 90% accuracy, as meaningless as that is).

I have noticed that the log of the inverse percentage of each group has made the best multiplier that I have tried. Is there a more standard manipulation for the cost function? Is this implemented correctly?

from collections import Counter
counts = Counter(category_train)
weightsArray =[]
for i in range(n_classes):
    weightsArray.append(math.log(category_train.shape[0]/max(counts[i],1))+1)

class_weight = tf.constant(weightsArray)
weighted_logits = tf.mul(pred, class_weight)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(weighted_logits, y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)


texts: Unable to immediately continue Jupyer Notebook after closing then reopening the project

texts: When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by rerunning all the code. Is there a better way?


texts:
Given an original probability distribution P, I want to measure how much an approximation Q differs from the initial distribution. For that I calculate the KL-divergence via scipy.stats.entropy, which returns infinity due to the large difference. However, as with time the approximation becomes better, I still want to quantify the divergence between the two sets.

The question is, is there any hack to avoid inf values or should I circumvent the behaviour by using some other distance measure?


texts:
I'm working with Mask R-CNN and want to understand the relationship between the produced losses Lmask, Lclass, Lbox which are eventually making up the muli-loss output and generally, multi-task learning.
Questions:

Are these two concepts related to each other in any sense?

Can we call Mask R-CNN as a multi-task learning framework?

Does multi-task loss only refer the addition of losses that are output or can we infer more than that?

Papers or books explaining these concepts will be helpful as well.


texts:
I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value.
I am wondering how I can get rid of this -infinity?
I have been advised to add a small value number inside the log to get rid of this like:
log10(dataset + 0.01)

This does get rid of the -infinity but I'm unsure does this skew the data or is it the wrong thing to do?
Also,if this is the right method, does anyone have any tips on how to decide what number to use e.g. 0.01, 0.1, or 0.001?


texts: developement of a CNN model for the detection of titles and important image boxes in a resume file

texts: I am trying to develop a CNN model with python in order to detect titles and boxes in a resume file, how can I fix this?


texts:
In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x)  \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\right] + \sigma_\epsilon^2 \\ = Bias(x)^2 + Variance(x) + Irreducible, $$
the first two terms depend on the argument $x$. So, in principle, if we choose a set of hyperparameters that yields the best tradeoff between the global bias and variance (somehow approximated, depending on our cross-validation strategy), these hyperparameters may lead to regions of $x$-es that have sub-optimal tradeoffs between their local biases and variances.

My questions:

In which scenarios may it be an actual problem? I mean: when are we better off training two classifiers on two separate sets of observations, with hyperparameters tweaked separately?
How could we identify such regions? I was expecting that a Boltzmann machine might be adapted for discovering highly volatile regions of $x$-es, but I didn't find anything of that sorts.


texts: Need some advice on approach to select only the informative emojis from the data set?

texts:
I have a giant data set from a local elections, which contains hashtags, emojis, and comments. I wanted to make a network analysis using only emojis.

So far I have a network analysis graph made in R which looks like this:

Sorry, you may have to zoom in to see the nodes. So, basically my goal is to see what people are talking about as a whole group. Currently there are lot of nodes which don't really say anything concrete about the main context or the hook, yet also creating clutter. I had extracted the data with political hashtags. Therefore, nodes such as milk bottle, cows, joy faces, toffees, flags, etc. doesn't really give me anything between people's comments and the context. My goal is to see a collective sentiment of people via their use of emojis in their context. I don't know if I make sense.

I don't know how should I approach the problem of only selecting informative emojis. Should I focus on the hashtags, and make a list of hashtags I am interested in, and only extract their associated comments and emojis? Or should I look at the sentimental values associated with emojis, and focus on the extreme positive and negative ones only?

I am pretty lost, any direction which method/ algorithm should I use to de-clutter this graph a bit yet also keep it in the context of politics & elections?


texts: recommendation system for eCommerce healthcare portal suggestion

texts:
I am trying to build a recommendation system. My system is basically a ecommerce application where our customers answers a bunch of questions related to healthcare (their basic health related question). Based on their ansers, we recommend some product. This process of recommendationis based of conventional rule base aapproach. Think of it as bunch of if-else condition. Now I am playing around with some machine learning technique nd want to see if this way will add any value in our healthcare system. I am at the very starting point and can use any suggestion from you guys. The suggestion could aim towards following:

Any product that you feel which leverages ML technique with respect to health(considering HIPPA constraints)
Any product that you feel which leverages ML technique with respect to health
What could be the first step towards building such a system.


texts: When a dataset is huge, what do you do to train with all the images on i t?

texts:
I'm using Python 3.7.7.
I'm trying to load a lot of NIFTI images using SimplyITK and Numpy from the [BraTS 2019 dataset][1].
This is the code I use to load the images into a numpy array.
import SimpleITK as sitk

def read_nifti_images(images_full_path):
    """
    Read nifti files from a gziped file.

    Read nifti files from a gziped file using SimpleITK library.

    Parameters:
    images_full_path (string): Full path to gziped file including file name.

    Returns:
    SimpleITK.SimpleITK.Image, numpy array: images read as image, images read as numpy array

    """
    # Reads images using SimpleITK.
    images = sitk.ReadImage(images_full_path)
    # Get a numpy array from a SimpleITK Image.
    images_array = sitk.GetArrayFromImage(images)

    # More info about SimpleITK images: http://simpleitk.github.io/SimpleITK-Notebooks/01_Image_Basics.html

    return images, images_array

This code works fine with smallest dataset but here I'm trying to load 518 nii.gz files with 155 images each file.
To run the code I'm using PyCharm latest version on a Windows 7.
How do you do it to train with all the images if all of them can't be in memory because memory limits?


texts: Questions for data analysis of a class room

texts:
I have Wireless Sensor Network setup in a classroom. I'm sensing data like temp, humidity, light, occupancy, etc. Classroom can accommodate 150 students and has 6 AC with light. From one side the classroom has big windows and that side is towards west and other 3 sides have thick walls and doors always closed. I have been looking for some questions that can be answered by analyzing data collected. Some of the questions I can think of are:

Predicting optimum time to turn on AC before class starts so that class is attains required temp with least energy consumed.
As of now I have 9 sensors on all walls and ceiling too. So to decide best position for sensor such that I can co-relate and predict other sensor values from only one or two sensor values.
Decide optimum amount of light needed by occupants to be comfortable in classroom.

Please suggest to me more questions that can asked about this situation. Also suggest if I can use somehow machine learning in WSN I have deployed.


texts: Intuition behind the number of output neurons for a neural network

texts:
I am reading Michael Nielsen's book on deep learning. In the first chapter, he gives the classic example of classifying 10 handwritten digits, and uses it to explain the intuition behind choosing the number of output neurons.

Initially, before reading the book, I thought it was intuitive that you would want 10 output neurons, each representing a given digit, but I'm starting to question why I thought so.

Anyways, here's what Nielsen explains:

  You might wonder why we use 10 output neurons. After all, the goal of the network is to tell us which digit (0,1,2,…,9) corresponds to the input image. A seemingly natural way of doing that is to use just 4 output neurons, treating each neuron as taking on a binary value, depending on whether the neuron's output is closer to 0 or to 1. Four neurons are enough to encode the answer, since 2^4=16 is more than the 10 possible values for the input digit. Why should our network use 10 neurons instead? Isn't that inefficient? The ultimate justification is empirical: we can try out both network designs, and it turns out that, for this particular problem, the network with 10 output neurons learns to recognize digits better than the network with 4 output neurons. But that leaves us wondering why using 10 output neurons works better. Is there some heuristic that would tell us in advance that we should use the 10-output encoding instead of the 4 -output encoding?
  To understand why we do this, it helps to think about what the neural network is doing from first principles. Consider first the case where we use 10 output neurons. Let's concentrate on the first output neuron, the one that's trying to decide whether or not the digit is a 0. It does this by weighing up evidence from the hidden layer of neurons. What are those hidden neurons doing? Well, just suppose for the sake of argument that the first neuron in the hidden layer detects whether or not an image like the following is present:

  It can do this by heavily weighting input pixels which overlap with
  the image, and only lightly weighting the other inputs. In a similar
  way, let's suppose for the sake of argument that the second, third,
  and fourth neurons in the hidden layer detect whether or not the
  following images are present:

  As you may have guessed, these four images together make up the 0
  image that we saw in the line of digits shown earlier:

  So if all four of these hidden neurons are firing then we can conclude
  that the digit is a 0. Of course, that's not the only sort of evidence
  we can use to conclude that the image was a 0 - we could legitimately
  get a 0 in many other ways (say, through translations of the above
  images, or slight distortions). But it seems safe to say that at least
  in this case we'd conclude that the input was a 0.

  Supposing the neural network functions in this way, we can give a
  plausible explanation for why it's better to have 10 outputs from the
  network, rather than 4. If we had 4 outputs, then the first output neuron would be trying to decide what the most significant bit of the digit was. And there's no easy way to relate that most significant bit to simple shapes like those shown above. It's hard to imagine that there's any good historical reason the component shapes of the digit will be closely related to (say) the most significant bit in the output.

  Now, with all that said, this is all just a heuristic. Nothing says
  that the three-layer neural network has to operate in the way I
  described, with the hidden neurons detecting simple component shapes.
  Maybe a clever learning algorithm will find some assignment of weights
  that lets us use only 4 output neurons. But as a heuristic the way of
  thinking I've described works pretty well, and can save you a lot of
  time in designing good neural network architectures.

I don't understand what he means in the second to last paragraph. I included the rest for context, but it's this second to last paragraph that's confusing me. Could someone clarify what he's talking about?


texts: Recommender system: Give a feature more significance than another

texts:
I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the number of reviews for said hotel. In my data set, some hotels have a rating of 5 for example, but with only one or two reviews. On the other hand, some hotels who are also rated at 5 have over 300 reviews. Is there a way to give priority or more influence to hotels that have been rated more frequently?
Thanks in advance!


texts: Extracting structured data from semi structured data

texts:
I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text file that looks like this :
Input :
2021565267MALL1ETAGE ZARA1st FLOOR 2345561
2022565267MALL2ETAGE ZARA1st FLOOR 2345561
2022565267ANFAPLACE2ETAGECOFEESHOP2345561
20225652634ANFAPLACE2ETAGE 2345561

Desired Output :
2021565267,MALL1ETAGE ZARA1st FLOOR,2345561
2022565267,MALL2ETAGE ZARA1st FLOOR,2345561
2022565267,ANFAPLACE2ETAGECOFEESHOP,2345561
20225652634,ANFAPLACE2ETAGE,2345561

The semi-structured files are not fixed-width so we can not just add col specification in pandas like this ( it can work for the first line for example ) :
col_specification =[(1, 10),.... ]

One of the approaches that I found online is to make a dictionary based on the occurrences of the words in the semi-structured file will that work in this situation if so how can I implement something like that?


texts: Performing 1D Depthwise conv using Keras 2D Depthwise conv

texts:
I would like to perform a 1D Depthwise convolution (ie the first step of the depthwise-separable convolution) for a machine learning model I am working on. This means that for an input activation space of h x w, there are h kernels, each 1xk, where k is the length of the kernel (see attached figure).
This is not currently offered by Keras, but one can use the available Depthwise 2D convolution (which operates on 3D inputs), and simply add an extra dimension to the 2D inputs, and permute such that the original 2D height will span the channels of the new 3D activation space (see figure). The model report, showing these dimensions and permutations is below (the format for the Depthwise layer is "channels first").
However, I do not seem to be able to mimic this behaviour in a matlab golden model which uses the same input activation space and kernels. Does this method of using a Depthwise Conv2D to operate like a Depthwise Conv 1D make fundamental sense, or practical sense with keras, or am I missing something with the channel permutations perhaps?


texts: Name Entity Linking with Naive Bayes Classifier

texts:
I was trying to absolve one name entity resolution problem.
Suppose, I have two sentences,

Cameron is the PM of England . He decided to step down .

Cameron was trying to keep England in EU . Mr. Cameron could not do it .

These two sentences have two different properties, first contains a pronoun, and second one alias against Cameron. My objective is to determine the entities(NE or otherwise) and link the entities appropriately, considering pronoun and alias are entities.

I have divided the task in two phrases.

In the first I am using one entity extraction algorithm, and in the second, trying to link them.

I am trying to use Supervised Learning algorithm for both of them.

I used one Hidden Markov Model(HMM) to extract the entities.

I had given user defined training set and extracted entities as,

Cameron/PERS is/NA the/NA PM/EXE of/NA England/LOC ./NA He/PRN decided/NA to/NA step/NA down/NA ./NA

and,

Cameron/PERS was/NA trying/NA to/NA keep/NA England/LOC in EU/ORG ./NA Mr./TIT Cameron/PERS could/NA not/NA do/NA it/NA ./NA

I am using one standard Naive Bayes(NB) classifier to classify these two sentences belonging to two groups.

The first one being 'PRN-STRING' and second one 'ALIAS'.

I am using a new training set in NB classifier, and classifying the extracted sentences.

The first is being labelled as 'PRN-STRING' and second 'ALIAS'.

My questions are

(i) Am I doing right? As much I read about entity linking I am going right, my friends and seniors also say so.

(ii) Classification and extraction tasks are independent, but am I right to combine in the same code?

In my code I am tagging with HMM and after some processing of the output, pushing as input to the classifier.

NB: EXE-> It is a User defined tag as Executive.


texts: Relationship between train and test error

texts:
I have some specific questions for which I could not extract answers from books. Therefore, I ask for help here and shall be extremely grateful for an intuitive explanation if possible.

In general, neural networks have a bias/variance tradeoff and thus we need to have a regularizer. Higher bias --> underfitting; Higher Variance--->overfitting.

To solve overfitting, we use regularization for contraining the weight. This is a hyperparameter and should be learned during training based on my understanding using cross-validation. Thus, the dataset is split into a train, validation and test set. The test set is independent and is unseen by the model during learning, but we have the labels available for it. We usually report the statistics such as false positives, confusion matrix, misclassification based on this test set.

Q1) Is this bias/variance problem encountered in other algorithms such as SVM, LSTM etc as well?

In convolutional neural network (Matlab toolbox) I have not seen any option for specifying the regularization constant. So, does this mean that CNN's don't need a regularizer?

Q2) What is the condition if training error and test error are both zero? Is this the ideal best situation?

Q3) What is the condition if training error > test error?

Q4) What is the condition if training error > validation error?

Please correct me where wrong. Thank you very much.