texts: How to perform upsampling (and NOT interpolation) process theoretically modelled?

texts:
As an example, I know that sampling a signal $s$ is modelled by multiplication of s by a dirac comb, which has the effect of convolving the Fourier Transform (FT) of $s$ by the FT of the dirac comb which is another dirac comb but with inverse spacing between peaks.
My question is what is the corresponding process for when one upsamples  (e.g., by a factor of 2),      , i.e., inserts zeros between samples,  and NOT yet applies interpolation since I know for interpolation you need to follow upsampling by a lowpassfilter whicg is a convolution a signal $s$?


texts: Which PC hardware would you recommend to invest in for movement ecology studies in R (x,y,t data analysis)?

texts:
I mean all possible work with tagging data: GIS, tagging data pre-processing, visualisation, different types of modelling, simulations and modern analysis.

I think it will be about 40 tagged animals and maximum 30-40 thousand total locations to work with.

The tools that I will probably use are described here: https://www.movebank.org/panel_software


texts: When should I use Gini Impurity as opposed to Information Gain (Entropy)?

texts:
Can someone practically explain the rationale behind Gini impurity vs Information gain (based on Entropy)?

Which metric is better to use in different scenarios while using decision trees?


texts: Number of features vs. number of samples : if small sample size is sufficient, why take large number of samples?

texts: As a newbie, I am a little confused. I have a dataset for binary classification with 11 features and 102 sample data. I have seen in most places (e.g., kaggle competions), the dataset may have hundreds of thousands of data samples for tens of features. On the other hand, this paper says (at least for LDA classifier) optimal number of features is n-1 for a sample size n. My question is, if small no. of samples is enough (or even optimal), why care for larger samples? What am I missing here?


texts: Performance metrics for balanced binary classification

texts:
From my understanding the reason we use Confusion matrix, Precision-recall, F1 score is because when our datasets consists of imbalanced class labels accuracy can be misleading.
However what if dataset if balanced(50/50) would problem arise when I use only accuracy as performance metrics?
If there exists a problem what performance metrics are preferred (score, no graphs) and why?
Thanks in advance!


texts: How to select samples for a trainings set

texts:
My dataset contains half a million unlabeled entries with over 100 binary features. A third of these features are present in less than 1000 samples.

I want to classify a few samples by hand (into one of two classes) and use them as trainings set for either an SVM or a neural network.

How do I select samples for classifying them by hand? Are there some preferred methods? I fear that by selecting them randomly not all possible features are present in my training set.

Do I have to select the training set and the dataset used for later evaluation in the same way? For example could I just use 10 examples of each feature usage for training and random samples for evaluation?

Is there a rule of thumb about how large my training set has to be?


texts: I am using numpy to implement some neural network tutorials. There is a constant("eta") used in codes. What does it means and what it stands for?


texts: How can you map the exceedance of a threshold into an activation function of a Neural Network?

texts: I am totally new to Artificial Neural Networks. Let’s say that the model you are trying to turn into an artificial neural network has an output that is triggered only by the exceedance of a threshold: $y\geq y_{1}. $Therefore, you need to find a way to use this inequality as an activation function. Is this feasible?


texts: with tf.device(DEVICE): model = modellib.MaskRCNN(mode = "inference", model_dir = LOGS_DIR, config = config)

texts:
ValueError Traceback (most recent call last)
/miniconda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
509 as_ref=input_arg.is_ref,
--> 510 preferred_dtype=default_dtype)
511 except TypeError as err:

/miniconda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)
1106 if ret is None:
-> 1107 ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
1108

/miniconda/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py in _autopacking_conversion_function(v, dtype, name, as_ref)
959 return NotImplemented
--> 960 return _autopacking_helper(v, inferred_dtype, name or "packed")
961

/miniconda/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py in _autopacking_helper(list_or_tuple, dtype, name)
921 elems_as_tensors.append(
--> 922 constant_op.constant(elem, dtype=dtype, name=str(i)))
923 return gen_array_ops.pack(elems_as_tensors, name=scope)

/miniconda/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
195 tensor_util.make_tensor_proto(
--> 196 value, dtype=dtype, shape=shape, verify_shape=verify_shape))
197 dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)

/miniconda/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
423 if values is None:
--> 424 raise ValueError("None values not supported.")
425 # if dtype is provided, forces numpy array to be the type

ValueError: None values not supported.

During handling of the above exception, another exception occurred:


texts: is a 4 output regression equivalent to 4 single output regression superposition?

texts:
I would like to generate a regression based on 6 different inputs and 4 output.

I use python and I compare sklearn and scipy. But in both cases, regression models are essentially focused on 1 output parameter.

My question is : can I considerer that all of my outputs are unlinked, and if yes, is it true to imagine that I can perform 4 regressions in parallel (one for each output) to make an equivalent 4 ouput model ?

Tank you for your tips !


texts: Visualize the predicted and actual class after training and testing

texts:
The data set X has 10 features with 50 instances labelled as 0 and 1. Considering only 6 instances as an example here, let YPred labels are [1,0,1,0,1,1] and the actual ground truth labels are YTest = [0,0,1,1,1,0]. I cannot draw a decision boundary after classification since the data set is multi-dimensional. In the following code, pred = predict(svmModel, X(testIdx,:)); runs k times.

I cannot understand which predicted class labels should be taken after the cross-validation ends to say that this is the final "good" prediction.


texts: Optimal graph search algorithm for probabilistic world

texts:
Greeting, i have the following world:

~500-600 states
~4-5 action per state.
4 of the states of a coin on them.
Some action are probabilistic
(for example "Jump from A to B,C" , has 20% of ending in state C  and 80% to end in B).

I have time to train on the graph, but when i am triggered i want to find the 4 coins as fast as possible.
Recommendations?


texts: Present results of the best or the last iteration on dev set?

texts:
Which is the correct way - presenting the results of the best or the last iteration on the dev set in a paper? In research papers I usually see only one value, is it the best iteration of all?

I'm asking because my model training last iteration wasn't the best one. Instead the best iteration was somewhere in the middle of the training.


texts: Why does the transformer positional encoding use both sine and cosine?

texts:
In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.

I am wondering why it needs to use both sine and cosine though instead of just one or the other?


texts: tensorflow tf.scatter_update over a matrix

texts:
i begin to work with tensorflow, and now i want to update a matrix (in the matrix, every row is one prediction, all the matrix is updated after a batch), i want to know what is the max value in the output of the mlp, and i have two problems, the first is how i can update a matrix, in the doc not says about update a matrix, i want to try update a simple array, and after update the complete row in the matrix (i'm not sure if allows that, or i will reassign all the row)

...MLP...

unbatched_states = tf.split(hidden, batch_size, 0) #hidden is the last layer from one MLP, the shape is 100,100, size of 100, and batch of 100
unbatched_sense_ids = tf.split(train_sense_one_hot, batch_size, 0) ##creates a list of tensors, every tensor is a case tagged
self.predictions = tf.Variable(tf.zeros([batch_size, embedding_size], dtype=tf.int64), trainable=False)#variable, where is stored the prediction)

for i in range(batch_size):
        predicted_sense_i = tf.argmax(unbatched_states[i], 1, name='prediction')  #get the index of the max value in the case i of the batch
        predicted_sense=tf.Variable(tf.zeros([embedding_size], dtype=tf.int64), trainable=False) #a tensor updated marking one the max value
        predicted_sense = tf.scatter_update(predicted_sense, predicted_sense_i), tf.constant(1, dtype=tf.int64))   #update with the index of max value
        self.predictions = tf.scatter_update(self.predictions[i], predicted_sense, tf.constant(1, dtype=tf.int64)) #update the matrix

Exist another way for update a matrix.


texts: Naive Bayes for Multi label text classification

texts:
How to use Naive Bayes for multi-label text classification in R.

I tried using naiveBayes() from e1071 library but it seems that while training, it doesn't accept multi-label class variable.

I created TermDocumentMatrix using the text document corpus and tried to create the model using this matrix and class variable(list of topics a particular document belongs to). Below is the code that I used.

trainvector <- as.vector(traindata$bodyText)

trainsource <- VectorSource(trainvector)

traincorpus <- Corpus(trainsource)

trainmatrix <- t(TermDocumentMatrix(traincorpus))

model <- naiveBayes(as.matrix(trainmatrix), as.factor(traindata$topics))

The last line gives below error:

Error in sort.list(y) : 'x' must be atomic for 'sort.list'

Have you called 'sort' on a list?

I tried using

model <- naiveBayes(as.matrix(trainmatrix), as.factor(unlist(traindata$topics)))

but got error:

Error in tapply(var, y, mean, na.rm = TRUE) :
  arguments must have same length


texts: Is Adam's optimization susceptible to Local Minima?

texts:
# Neural Network Architecture

no_hid_layers = 1
hid = 3
no_out = 1

# Xavier Ininitialization of weights w

w1 = np.random.randn(hid, n+1)*np.sqrt(2/(hid+n+1))
w2 = np.random.randn(no_out, hid+1)*np.sqrt(2/(no_out+hid+1))

# Sigmoid Activation Function
def g(x):
    sig = 1/(1+np.exp(-x))
    return sig

def frwrd_prop(X, w1, w2):
    z2 = w1 @ X.T
    z2 = norm(z2, axis=0)
    a2 = np.insert(g(z2), 0, 1, axis=0)
    h = g((w2@a2))
    return (h,a2)

# Calculating Cost and Gradient

def Cost(X, y, w1, w2, lmbda=0):
    # Initializing Cost J and Gradients dw
    J = 0
    dw1 = np.zeros(w1.shape)
    dw2 = np.zeros(w2.shape)
    # Forward Propagation to calculate the value of the output
    h, a2 = frwrd_prop(X, w1, w2)
    # Calculate the Cost Function J
    J = -(np.sum(y.T*np.log(h) + (1-y).T*np.log(1-h)) - lmbda/2*(np.sum(np.sum(w1[:,1:].T@w1[:,1:])) + np.sum(w2[:,1:].T@w2[:,1:])))/m
    # Applying Back Propagation to calculate the Gradients dw
    D3 = h-y
    D2 = (w2.T@D3)*a2*(1-a2)
    dw1[:,0] = (D2[1:]@X)[:,0]/m
    dw2[:,0] = (D3@a2.T)[:,0]/m
    dw1[:, 1:] = ((D2[1:]@X)[:,1:] + lmbda*w1[:,1:])/m
    dw2[:, 1:] = ((D3@a2.T)[:,1:] + lmbda*w2[:,1:])/m
    # Gradient clipping
    if(abs(np.linalg.norm(dw1))>4.5):
        dw1 = dw1*4.5/(np.linalg.norm(dw1))
    if(abs(np.linalg.norm(dw2))>4.5):
        dw1 = dw1*4.5/(np.linalg.norm(dw2))
    return (J, dw1, dw2)

# Adam's Optimization technique for training w

def Train(w1, w2, maxIter=50):
    # Algorithm
    a, b1, b2, e = 0.001, 0.9, 0.999, 10**(-8)
    V1 = np.zeros(w1.shape)
    V2 = np.zeros(w2.shape)
    S1 = np.zeros(w1.shape)
    S2 = np.zeros(w2.shape)
    for i in range(maxIter):
        J, dw1, dw2 = Cost(X, y, w1, w2)
        V1 = b1*V1 + (1-b1)*dw1
        S1 = b2*S1 + (1-b2)*(dw1**2)
        V2 = b1*V2 + (1-b1)*dw2
        S2 = b2*S2 + (1-b2)*(dw2**2)
        if i!=0:
            V1 = V1/(1-b1**i)
            S1 = S1/(1-b2**i)
            V2 = V2/(1-b1**i)
            S2 = S2/(1-b2**i)
        w1 = w1 - a*V1/(np.sqrt(S1)+e)*dw1
        w2 = w2 - a*V2/(np.sqrt(S2)+e)*dw2
        print("\t\t\tIteration : ", i+1, " \tCost : ", J)
    return (w1, w2)

# Training Neural Network

w1, w2 = Train(w1,w2)

I'm using Adam's Optimization to converge Gradient Descent to a global minima but the cost is becoming stagnant (not changing) after around 15 iterations(the number is not fixed). The initial cost due to random initialization of weights is changing very minutely before becoming constant. And this is giving training accuracy from 45% to 70% for different runs of the exact same code. Can you help me with the reason behind this?


texts: StratifiedKFold: ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead

texts:
Working with Sklearn stratified kfold split, and when I attempt to split using multi-class, I received on error (see below). When I tried and split using binary, it works no problem.

num_classes = len(np.unique(y_train))
y_train_categorical = keras.utils.to_categorical(y_train, num_classes)
kf=StratifiedKFold(n_splits=5, shuffle=True, random_state=999)

splitting data into different folds

for i, (train_index, val_index) in enumerate(kf.split(x_train, y_train_categorical)):
   x_train_kf, x_val_kf = x_train[train_index], x_train[val_index]
   y_train_kf, y_val_kf = y_train[train_index], y_train[val_index]

ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead.

Is there a way I can used KFold with multi-class?


texts: 'tuple' object is not callable while reshaping training data using python

texts:
I have data csv file with three inputs names temperature, humidity, wind. Here I want to predict temperature value in every 60 minute using LSTM model.

Here I write the code to reshape the train . But I got an error tuple' object is not callable

My code:

data = pd.read_csv('data6.csv' )
data['date'] = pd.to_datetime(data['date'] + " " + data['time'], format='%m/%d/%Y %H:%M:%S')
data.set_index('date', inplace=True)
data = data.values

scaler = MinMaxScaler(feature_range=(0, 1))
data = scaler.fit_transform(data)

train_size = int(len(data) * 0.67)
test_size = len(data) - train_size
train, test = data[0:train_size,:], data[train_size:len(data),:]

X = 1
n_out = 1

x,y=[],[]
start =0
data = train.reshape(train.shape(train.shape[0] ,3, train.shape[1]))
for _ in range(len(data)):
  in_end = start+X
  out_end= in_end + n_out
  if out_end < len(data):
    x_input = data[start:in_end]
    x.append(x_input)
    y.append(data[in_end:out_end,0])
start +=1

x = np.asanyarray(x)
y = np.asanyarray(y)

My csv file

Error:

  data = train.reshape(train.shape(train.shape[0] ,3, train.shape1))

Error image:

Can anyone help me to solve this problem?

Error:


texts: Drowsiness Detection issue understanding LSTM input shape

texts:
I’ve difficulty in understanding LSTM input shape. For example. I’ve 2 videos out of these 1 is categorized as Awake (0) and 1 as Drowsy (1). I preprocessed them to extract Eye Aspect Ratio and Mouth Aspect Ratio as features from every second image frame using dlib 68 facial landmark detection.
Now my data has ( VideoFileName, Time Series, EAR, MAR, Label )
VideoFileName   TimeSeries   EAR      MAR       Label
Video1          1            0.30     0.25      0
Video1          2            0.31     0.27      0
Video1          3            0.35     0.25      0
Video2          1            0.30     0.25      1
Video2          2            0.27     0.28      1
Video2          3            0.31     0.29      1
Video2          4            0.33     0.30      1

I extracted above data from first 3 and 4 seconds of two videos respectively as the length of videos may be different. (Sample Data here)
How should I feed this data to LSTM? Any code example would be a great help. I know input shape should be [Batch Size, Time Step, Features] but I’m confused how to feed this to LSTM should I feed each video’s (Sample) data in a loop.
Loop for No. of Videos (Samples)
   LSTM( ...., input_shape( batch_size, No. of TimeSteps of Video, 2 ) )

batch_size  ->  Can be defined from 1 to max no. of time series. I'm taking 3 for Video1 and 4 for Video2.
Time Steps  ->  3 for Video1 and 4 for Video2. (For above sample data)
Features    ->  2 (As I will be only feeding EAR and MAR as features)
Now I've next query that how my Label dataset should look like because I'm taking full time series of each video as single batch. Should it look like below?
y_train = [0,1]

Please help me to clear my doubt.


texts: Help getting corresponding dataframe values

texts:
I have two dataframes:

self.thisSession_df:
    id   exerciseId  sets
    1       1         12
    2       1         14
    2       2         15
    2       2         15

self.exercises_df:
    id   exerciseName
    1      Squat
    2      Pullup

I would like to find a way to replace the exerciseId in self.thisSession_df with the corresponding name from self.exercises_df

Hopeful result:

self.thisSession_df:
    id   exerciseId    sets
    1       Squat       12
    2       Squat       14
    2       Pullup      15
    2       Pullup      15

I tried a solution that I found on here and modified it to come up with:
self.thisSession_df['exerciseId'] = self.thisSession_df['exerciseId'].map(df1.set_index('id')['exerciseName'])

This gives me error: string indices must be integers

I would appreciate a nudge in right direction!


texts: Low leves of probability observed after modelling.Is it right to scale the probability

texts:
I have done modelling on imbalanced class , without any sampling methods.
Event rate is around 0.1 ,After modelling I am getting probalities more at the lower side close to zero.I have tried differnt models and an ensemble as well using caretlist in R.I am getting AUC around 79.I have rank ordered the probability and the maximum probability predicted is less than .5.

My question is that

1)Is there any thing i can try for getting the predicted probabilities ranging from 1(or in the higher level close to 1 eg : .7,.8 etc)

2)Is it right to transform the predicted probabilities between 0 and 1 by by applying min - max normalization and considering all probabilities above .5 have chance of winning and less than .5 have a chance of losing

Note:I have tried sampling methods like smote, rose and up sampling but resulted in low AUC because of the small number of observations.

Please let me know if any clarification is needed.
Any assistance would be really helpful.


texts: How to classify features into two classes without labels?

texts:
I have a big dataset with nearly 200 features. However, I do not have class labels for these data. I want to divide these data into two classes based on these features. I know, when we do not have class labels we have to use some clustering method. However, since I do not have any labels, I am just wondering how to measure the accuracy of the models.

Please let me know the most suitable approach that I should follow?

I am happy to provide more details about my featureset if needed :)


texts: When should we start using stacking of models?

texts:
I am solving a Kaggle contest and my single model has reached score of 0.121, I'd like to know when to start using ensembling/stacking to improve the score.

I used lasso and xgboost and there obviously must be variance associated with those two algorithms. So stacking should theoretically give me better output than my individual algorithms.

But how to idenfity if stacking is worth it and that we've reached dead end to accuracy of a particular model?


texts:
I'm trying to build a model of Machine Learning for the first time, and I'm looking for some guidelines.
My final purpose is to identify customers who are planning to flight abroad.
From my existing data, I identified customers who flew in the past, and I've flagged some potential variables (features) that might indicate that one is about to flight abroad.

My questions are:
1. What techniques should I use, in order to choose the right features for my model?
2. What approaches (models) would you use in such a problem?

because this is the first time I'm experiencing ML, I'll be happy to hear any further suggestions you have to give me.

Your help would be appreciated.


texts:
I hope this question is relevant to this community as Theano is increasingly a standard software to do machine learning.

So I have been trying to follow the instructions here,
http://deeplearning.net/software/theano/install_windows.html#install-windows to install Theano on my Windows machine but as expected I am stuck with various parts.
So I have created a folder called "Theano" and in that I have 3 folders which have installed ``Theano-0.8.2" , GCC and Anaconda.

I am following the "Alternative:Anaconda" route and there I am instructed to type into my Windows command prompt, "conda install mingw libpython". But on doing that I get this message saying that mingw" and python 3.5*" specifications are in conflict.
What does this mean? What is ``mingw"!?

In the "Configuring The Environment" section what is this "PATH" that is being talked about? Do I have to write this ``shell script"? (What is this?) Where does one write this and save this file?
Can someone show a generic version of this script in which I can maybe fill in the blanks corresponding to how my folders are organized? (The .bat file that they have written seems to correspond to their file structure and its hardly clear as to which parts change!)

Then further down one sees this section, "Configuring Theano" it asks one to open a "Python Shell" and run the command "python setup.py develop".
So what is the "Python Shell"? Where do I find it?
Is it that black "Anaconda Prompt" terminal?

To be clear now when I open my "Anaconda Prompt" window it says,
< C:\Users(my_user_name)\Theano\Anaconda > C:\Users(my_user_name)>
Seeing this I thought that Theano is now installed.
But when I open the Jupyter notebook and type "import theano", I get an error!


texts: Choose threshold to get 90% precision classifier - ML Binary Classification problem

texts:
I have chosen threshold value with below code to get 90% precision classifier
from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(sgd_clf, X_train, y_train, cv=3)

z_scores = cross_val_predict(sgd_clf, X_train, y_train, method='decision_function')

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_pred, z_scores)

threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]

y_train_pred_90percent_precision = (z_scores >= threshold_90_precision)
print(precision_score(y_train, y_train_pred_90percent_precision))

I'm expecting precision_score to be 90% but it returned 95%. Is this expected? Anything incorrect with my code? If it's expected, can you please explain the reason?


texts: Convolutional Neural Networks layer sizes

texts:
I am trying to understand an article Backpropagation In Convolutional Neural Networks

But I can not wrap my head around that diagram:

The first layer has 3 feature maps with dimensions 32x32. The second layer has 32 feature maps with dimensions 18x18. How is that even possible ? If a convolution with a kernel 5x5 applied for 32x32 input, the dimension of the output should be $(32-5+1)$ by $(32-5+1)$ = $28$ by $28$.

Also, if the first layer has only 3 feature maps, the second layer should have multiple of 3 feature maps, but 32 is not multiple of 3.

Also, why is the size of the third layer is 10x10 ? Should it be 9x9 instead ? The dimension of the previouse layer is 18x18, so 2x2 max pooling should reduce it to 9x9, not 10x10.


texts: What's the principal difference between ANN,RNN,DNN and CNN?

texts:
I'm newer to deep learning domain.

I would like to know what is the principal difference between RNN,ANN,DNN and CNN?

How to implement those neural networks using the TensorFlow library?


texts: What are some good models to complement XGBOOST in stacking?

texts: What are some good models to complement XGBOOST via stacking in typical Kaggle datascience competition? I realize XGBoost with well-tuned hyperparamters are generally quite good already.


texts: How to display graph with MathJax in jupyter notebook?

texts:
I am trying to create a jupyter-notebook with a graphical depiction of a neural network.  For starters, I am trying to display a simple network that evaluates a boolean NOT operator.

I am using the %%dot cell magic that is documented at https://metakernel.readthedocs.io/en/latest/source/README.html?v=20190725125340#id13 and I have a code cell in my notebook with the following contents:

%%dot
digraph NOT {
  rankdir = LR
  node [shape="circle"]
  x0 [label="+1"]
  x1 [label="$x_1$"]
  h [shape="plain"]
  y1 [color="coral" label=""]
  x0 -> y1 [label="10"]
  x1 -> y1 [label="-20"]
  y1 -> h [label="g()"]
}

This is rendered into an SVG that is displayed as the output of the code cell.
However, for the label on the x1 node I've used $x_1$, which I'd like to render as $x_1$ (presumably by MathJax, as it does in the Markdown cells).  Unfortunately, I just get a blank node with no label, like this:

Any idea what I can do to fix this?


texts: Reduce multiclass classification targets to binary classification targets in scikit-learn

texts:
I would like to reduce multiclass classification targets to binary classification targets. Ideally, this mapping would happen within scikit-learn so the same transformation applies during both training and prediction.
I looked at transforming the prediction target (y) documentation but did not see something that would work. Ideally, it would be a classifier version of TransformedTargetRegressor.
Something like this mapping:
targets_multi  = {'A', 'B', 'C', 'D'}
targets_binary = {0: {'A', 'B'},
                  1: {'C', 'D'}}


texts:
I have more then 100 columns with the values of 1-0. But the two features at the end as seen in the below image, have different values then the rest. Should I rescale the values in the last two columns into a range of [0,1]?
Note : I use K-Means algorithm.


texts:
I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip.

We create pairs between data sets  and calculate cosine similarity and then pass cosine values for all features pairs to Gaussian Mixture model.We started with 2 cluster, with expectation of one match cluster and one no match cluster.But ML does not build one match cluster and  matches are in both the clusters.

Before passing to ML, i use Standard scaler and minmax scaler , but still don't get a clear nomatch and match cluster.If we increase the cluster same thing happens.

Match could be High cosine similarity  in Name,Address,State,City & zip or Name ,address ,zip or any other combinations.We are dealing with huge volume , so we are using Spark ML.

How can we achieve optimal clustering?


texts: Does convergence equal learning in Deep Q-learning?

texts:
In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted from N levels of a given game. Then, I want to use the trained model to solve M new levels of the same game, i.e., I want to test the generalization ability of the agent on new levels of the same game.
Currently, I have managed to find a complex (deep) CNN architecture which is able to converge. This means that after training it for a large number of iterations (by the way, I'm using prioritized experience replay), the training error (square difference between Q-values and Q-targets) is very low.
Since I want the agent to be able to generalize to new, unseen levels, I thought about finding the simplest possible CNN architecture which is able to converge on the training levels, since that model would generalize better (be less prone to overfitting on the training levels).
In supervised learning, my reasoning would be correct. However, I don't know if it holds true in Reinforcement Learning. Given a model which is able to converge (minimize error between Q-targets and Q-values), is that model always learning to solve the levels optimally, i.e., does the model find the optimal policy which maximizes reward? In other words, is it possible for a Deep Q-learning agent to find a non-optimal policy which converges to a very small error between Q-targets and Q-values?
From what I have read, I think that as long as Deep Q-learning converges, the policy found is always the optimal one. Please, correct me if I'm wrong.


texts:
I am trying to do some clustering. I have a dataset that is very
sparse - with the majority of features only occurring in a single vector.

Here is a list of our features:
https://gist.github.com/scrooloose/5963725dc88e5d15d74dcae522bebf82

I am looking for any suggestions/hints/pointers as to how we can merge some of
these isolated features together. This should hopefully make my clustering
experiments more successful.

For example, from a manual inspection of the data, I can see this group of
features that could be all merged into a feature like "health" or perhaps
"mental health" + "general health" or similar.

618: Mental Health Research
619: Mental disorder
1616: mental health
1617: mental illness
1618: men’s health
410: Genital wart
402: Genomic Medicine
476: Hygiene

Another example is this set of features that could be merged into something
like "education":

536: Kiir Primary School
591: Makonzi Boarding School
609: Mathematics
670: New York University
300: Education
301: Educational psychology
349: Female education

Any thoughts would be very welcome, thanks :)

Side note: These features are keywords as returned from alchemy
(http://www.alchemyapi.com/). Resulting from keyword searches for a set of
URLS. The intention is to cluster the URLs (and hence they companies they
represent) by these keywords.


texts: Model Validation accuracy stuck at 0.65671 Keras

texts:
I am using conv1d to classify EEG signals, but my val_accuracy stuck at 0.65671. No matter what changes i do, it never go beyond 0.65671.
Here is the architecture

model=Sequential()
model.add(Conv1D(filters=4,kernel_size=5,strides=1,padding='valid',kernel_initializer='RandomUniform',input_shape=X_train.shape[1::]))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Conv1D(filters=6,kernel_size=3,strides=1,padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Conv1D(filters=8,kernel_size=3,strides=1,padding='valid',activation='relu'))
#model.add(Conv1D(filters=24,kernel_size=7,strides=1,padding='same',activation='relu'))

model.add(Flatten())
model.add(Dense(12,activation='relu'))
model.add(Dense(1,activation='sigmoid'))

Shape of training data is (5073,3072,7) and for test data it is (1908,3072,7).

I have tried reducing the number of neurons in each layer, changing activation function, and add more layers. But this upper limit has not changed mostly.

I have tried one hot encoding of binary class, using keras.utils.to_categorical(y_train,num_classes=2) but this issue does not resolve.

I have tried learning rate of 0.0001, but it does not work. I have tried some kernel_initializer and optimizers but nothing help

Results

   Train on 5073 samples, validate on 1908 samples
Epoch 1/8
 - 23s - loss: 0.6865 - acc: 0.5757 - val_loss: 0.6709 - val_acc: 0.6564

Epoch 00001: val_acc improved from -inf to 0.65645, saving model to weights.hdf5
Epoch 2/8
 - 22s - loss: 0.6760 - acc: 0.5837 - val_loss: 0.6569 - val_acc: 0.6567

Epoch 00002: val_acc improved from 0.65645 to 0.65671, saving model to weights.hdf5
Epoch 3/8
 - 21s - loss: 0.6661 - acc: 0.5843 - val_loss: 0.6669 - val_acc: 0.6111

Epoch 00003: val_acc did not improve from 0.65671
Epoch 4/8
 - 21s - loss: 0.6622 - acc: 0.5915 - val_loss: 0.6579 - val_acc: 0.6253

Epoch 00004: val_acc did not improve from 0.65671
Epoch 5/8
 - 22s - loss: 0.6575 - acc: 0.5939 - val_loss: 0.6540 - val_acc: 0.6255

Epoch 00005: val_acc did not improve from 0.65671
Epoch 6/8
 - 21s - loss: 0.6554 - acc: 0.5940 - val_loss: 0.6448 - val_acc: 0.6399

Epoch 00006: val_acc did not improve from 0.65671
Epoch 7/8
 - 21s - loss: 0.6511 - acc: 0.6042 - val_loss: 0.6584 - val_acc: 0.6195

Epoch 00007: val_acc did not improve from 0.65671
Epoch 8/8
 - 21s - loss: 0.6487 - acc: 0.6059 - val_loss: 0.6647 - val_acc: 0.6030

Epoch 00008: val_acc did not impr

ove from 0.65671


texts: How to group various similar search keywords and find top 100 keywords from big dataset

texts:
I have search keywords in one of my database table. These are the keywords searched by users on a website.
My requirement is to find the top 100 search keywords after consolidating various similar search keywords.

e.g. COVID19, Covid 19, Corona, Corona virus, covid-19 are all related to corona and it should be grouped under Corona.

OR

blockchain, BLOCKCHAIN, Blockchain business, B L O C K C H A I N are all related to Blockchain so it should be grouped under Blockchain category.

I tried to do this using fuzzywuzzy python library but fuzzywuzzy needs a list to be compared with the search keywords.
And unfortunately I don't have that list because it can be anything based on user's search.

Any suggestions on the approach would really help.


texts: LSTM data scaling formula based on range

texts:
I am planning to run LSTM for a prediction project which has the following input

Presure Temp  DateTime              Strain  Output
123     515   2018-01-11 00:00:10   0.02    1
145     515   2018-01-11 00:00:20   0.03    1
145     515   2018-01-11 00:00:30   0.04    1

My issue is that for the scaling can I base let say for the pressure maximum value is 1000 so I scale based on 123/1000 and I do the same for temp but do I need to scale date time?


texts: Are there any algorithms that learn to learn a function mapping?

texts:
Typical algorithms involve learning and applying a single mapping e.g. $ f: X \mapsto Y$

Are there any algorithms that learn multiple mappings given an extra variable e.g. $ f(Z): X \mapsto Y$
(This feels like an abuse of notation.).

Here we learn a mapping, given $Z$, that is then applied to $X$ to produce $Y$

One solution (although intractable) would be to train an algorithm (e.g. a neural network) whose output is then used as the weights of a second network.

Any suggestions would be much appreciated and feel free to ask for more details. Thanks.


texts:
I am trying to compile a simple model and the following works just fine:

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
               loss='categorical_crossentropy',
                metrics=['accuracy'])

However, I would like to use this metric: tf.metrics.precision_at_k and I am not sure how I can get it to work. It requires two arguments and that is causing an issue as I don't know how to pass predictions as an argument. I tried a

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
               loss='categorical_crossentropy',
                metrics=[tf.metrics.precision_at_k])

And certain variations but did not help.

Minimal working code:

import tensorflow as tf
from tensorflow.keras import layers
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (3,3), padding='same', activation=tf.nn.relu, input_shape=(5, 5, 1)),
    tf.keras.layers.MaxPooling2D((2, 2), strides=2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(2,  activation=tf.nn.softmax)
])


texts: Scaling of ordinal data before both hierarchical and KMeans clustering

texts: I am new to data analytics. As part of my assignment I have to perform both hierarchical and Kmeans clustering on a data set wherein all applicable variables are ordinal (1-5 rating scale). Do I need to scale the data before performing clustering?


texts: wrong prediction from graphlab.recommender.item_similarity_recommender

texts:
I have a question about basic understanding of how item-item collaborative filtering of "Graphlab" library works.
I run this code:

f = graphlab.SFrame({'user_id': ['0', '0', '0', '1', '1', '2', '2'],
                     'item_id': ['a', 'b', 'c', 'a', 'b', 'b', 'c'],
                     'rating': [ 1,   3,   2,   5,   4,   1,   4]})
m = graphlab.item_similarity_recommender.create(f, target="rating", similarity_type='cosine')

print m.get_similar_items(verbose=True)
print m.recommend()

The output is:

The matrix form of data:

According to the model, the recommendation for (1,c) is 1.096 but according to my calculation it is 4.16!
My calculation:

(1,c) = (sim(c,a)*5 + sim(c,b)*4)/(sim(c,a) + sim(c,b))
      = (0.0877*5 + 0.4385*4)/(0.0877+0.4385)
      = 4.16

Please help, what am I missing?


texts: ML algorithm to predict the state of state of a device (ON/ OFF) based on day, time and location

texts:
I require to develop an machine learning algorithm to predict that if a secondary car battery should be connected to load at any given time of day based on the automobile usage profile of the user. (the output is either 1 or 0 , ON or OFF)

Some Background : My project car is equipped with a secondary battery which has to run a car Computer, LTE modem etc. So it is important that the battery is not connected to these loads all the time so as to extend the run time of the battery when the car is not in use( primary car battery charges the secondary battery when the car is running.) . These electronic parts are used for IOT operation, running alexa voice interaction etc.

So it is necessary to predict at what times user is using his car and switch the circuit ON before the driver actually gets in the car, So that it can start alexa voice interaction  with user as soon as he enters the car without any delay( if it starts after user gets in , it will take some time before all the internet connections are up and running.).

Any suggestions on which existing machine learning algorithm I can use here? For the start I’m planning to predict the state based on the day, time and location only.  For example if the time is 7.am, weekday, and the location is home,  the car is usually ON to travel to office., if the time is 10.00am and the location is Office , the car is usually OFF.

Please help me with any  ideas/ suggestions that can also  improve the concept.


texts: Dealing with foreign film titles on IMDb

texts:
I've managed to scrape the data from the Cinemetrics website, but as it is user-submitted, some of the data needed cleaning. I've been running the titles through the search function of the OMDb API, however this struggles with certain results.

For example, the film Meghe Dhaka Tara. When searching with this as the query on the OMDb API I get no results. However a quick Google search, or even IMDb search, will find the film with the English title The Cloud-Capped Star.

I was wondering if anybody else has ever run into a similar problem and how they go about working around it?


texts: How do I right feature selection for DBSCAN?

texts:
I want to use DBSCAN to recognize any clusters within all text elements from the DOM tree of any webpage. For example all menu items shall be clustered separatey to all main content or footer elements.

My features for now are:
- DOM path to the text element
- visual rendered CSS properties (used PhantomJS) of  the text element
- the middle x and y position of the text element
- the width of  the  text element

Before using DBSCAN I use StandardScaler for my feature vector.

But I have no good results even after trying channging eps.

Now the question:
Can DBSCAN handle att all such different kind of features like OneHotEncoding for DOM path combined (!) with features like positions or widths in pixel? Should I weight some features before using StandardScaler that all have a similar value?


texts: How to get predictions with predict_generator on streaming test data in Keras?

texts: In the Keras blog on training convnets from scratch, the code shows only the network running on training and validation data. What about test data? Is the validation data the same as test data (I think not). If there was a separate test folder on similar lines as the train and validation folders, how do we get a confusion matrix for the test data. I know that we have to use scikit learn or some other package to do this, but how do I get something along the lines of class wise probabilities for test data? I am hoping to use this for the confusion matrix.


texts: Probability that ensemble model is correct based on accuracies of its classifiers

texts:
I'm trying to understand what I did wrong when trying to answer this question.  The exact question is:

Assume that we have 3 trained prediction models, and each model outputs either
-1 or 1. We then tested the accuracies of these models and obtained the following outcomes:

Model
Accuracy

m1
0.60

m2
0.55

m3
0.45

Let M be the ensemble model that outputs a plurality vote of these three models. If we assume that the errors of the models m1, m2, and m3 are independent, what is the probability that M(x) would be correct on a test instance x?

I thought that because this was a plurality vote, and the classification errors are independent of each other, I could simply take the weighted average of the accuracy of the three classifiers:
$
\begin{align*}
P(X) &= \sum_{all\ models\ M_j} P(C_i|x,M_j)P(M_j) \\
 &= \frac{1}{L} \sum_{all\ models\ M_j} P(C_i|x,M_j) \\
 &= \frac{1}{3}(0.60+0.55+0.45)\\
 &= 0.53
\end{align*}
$
But I was told that this is incorrect (with no context as to why).
Can someone explain why this is incorrect?  If this is a plurality vote (which to me assumes that the votes of each classifier are equal), why can I not simply take the weighted average?


texts: Handling actions with delayed effect (Reinforcement learning)

texts:
I am working on a problem where the action that I learn (using DQN) can be executed 'now' but it's effect on the environment is delayed by 'T' units of time.

The environment however is active in that time and there are other conditions based on which rewards are computed and returned. How is this handled?

I believe the Q value function (with gamma) handles 'delayed' effects of rewards but not actions.

This is similar to the inventory management use cases. As an analogy consider that I sell cakes. As customers walk into my shop I consume cakes off the shelf. I must reorder to stock my shelf BUT this reordering can take time to take effect.

I thought of just adding the quantity reordered to the shelf at a later time and let the agent learn it's effects. Will this suffice?

As another approach I thought of Experience and Replay as a mechanism to handle this delayed effect.

Appreciate the help.


texts: What's the most robust way to predict the value of some noisy time series data?

texts:
I have some time series data (time, value) such as:

281,97.0
284,98.0
287,98.0
290,98.0
293,98.0
297,99.0
309,100.0
320,101.0
333,101.0
346,103.0
359,104.0
372,105.0
385,107.0
396,108.0
405,108.0

I am trying to predict when the value will be 160.

The data is somewhat noisy. The values go mostly up, but could go down sharply every now and then, or stay the same for a short time. When these distortions occur, the prediction is far off for a period of time.

What algorithms or heuristics would work the best to deliver a robust prediction that doesn't vary to much?

I am currently using a linear fit over a time window of 60 data points. The results are not stable enough.


texts: Which GUI library to use with Deep Learning

texts:
I have completed basics Deep Learning course from coursera using Tensorflow and Keras.
Now I want to apply GUI to it.
So which library should i learn:
1.PyQt
2.Kivy
3.Tkinter
Are there libraries which can help to easily create deep learning projects.


texts: How to train my CNN using AMD GPU on Windows?

texts:
I only have a laptop running Windows 10 with AMD Radeon Graphics Card at my disposal right now. I am aware that this is likely to make things tougher but I need to stick with this as long as I can't buy a better laptop with an NVIDIA Graphics Card in it.
I'm aware of the PlaidML framework. I downloaded it following the instructions under Windows on this page, and added os.environ["KERAS_BACKEND"] = "plaidml.keras.backend" to my script right after the import statements.
I started training my CNN but when opening the Task Manager it seems no process is using the GPU. The training process is estimated to last 15+ hours with a single epoch only. I'm pretty sure it is running it on the CPU.
My CNN is structured as:
model.add(Conv2D(32, kernel_size=(3,3), activation='relu', padding='same', input_shape=(90,60,3)))
model.add(Conv2D(64, kernel_size=(3,3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(4,4)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(1, activation='sigmoid'))


texts: Simulation - identify the input parameters that impact the most the output

texts:
I am studying (via simulation) a system that has several input parameters. The output of the system is influenced by the input parameters. My goal is to identify the parameters that have the most impact on the output of the system. I ran a large amount of simulations and for each one of them I changed the value of each parameters using a random algorithm. To identify the parameters that have the most impact on the output I was planning on doing a basic multivariate linear regression including all the parameters, however, it turned out to have a low R2 and low F-stat.

I am working a little bit out of my element here so I am looking for suggestions regarding what I should investigate next to achieve my goal.

The context of the study is building energy efficiency and the main difficulty is that some input parameters have multiple effect on the system. For example: if I lower the total lighting power used in the simulation it will have for effect to decrease the cooling load of my building but if at the same time I was increasing the cooling efficiency of my cooling system the energy saved by increasing the efficiency would be less than if the lighting power was not decreased. In this example I would like to show which has more impact in regards to the energy consumption. However, in what I am trying to do I have a lot more input parameters.


texts: I have a dataset of 39 medical MR images, and I have to build a model to classify the tumor type. so is it suitable to use k-fold cross validation for validating the model? if so, what would be the number of K?


texts: What is difference between transductive and inductive in GNN?

texts:
It seems in GNN(graph neural network), in transductive situation, we input the whole graph and we mask the label of valid data and predict the label for the valid data.

But is seems in inductive situation, we also input the whole graph(but sample to batch) and mask the label of the valid data and predict the label for the valid data.


texts: What is fractionally-strided convolution layer?

texts:
In paper Generating High-Quality Crowd Density Maps using Contextual Pyramid CNNs, in Section 3.4, it said

  Since, the aim of this work is to estimate high-resolution and
  high-quality density maps, F-CNN is constructed using a set of
  convolutional and fractionally-strided convolutional layers. The set
  of fractionally-strided convolutional layers help us to restore
  details in the output density maps. The following structure is used
  for F-CNN: CR(64,9)-CR(32,7)- TR(32)-CR(16,5)-TR(16)-C(1,1), where, C
  is convolutional layer, R is ReLU layer, T is fractionally-strided
  convolution layer and the first number inside every brace indicates
  the number of filters while the second number indicates filter size.
  Every fractionally-strided convolution layer increases the input
  resolution by a factor of 2, thereby ensuring that the output
  resolution is the same as that of input.

I would like to know the detail of fractionally-strided convolution layer.


texts: Determining Important Atrributes with Feature Selection

texts:
I have a labeled dataset of two classes. Let’s say sick and healthy patients. My features are patient data as well as diagnostic data such as blood-test results. It’s categorical as well as continuous data. In total several thousand features.

I want to identify the features that best differentiate the sick patients from the healthy ones.

It is not really an anomaly detection problem, because the sickness is not directly indicated by anything in my features. Meaning that for none of my features the sick patients would be indicated by outliers. For example a result could be that the risk of sickness is higher for patients over the age of 50. But being over 50 is not generally uncommon.

The obvious approach would be to use a dedicated feature selection technique. For example a chi-square test, variable importance in decision trees or backward elimination.

The problem about all of these approaches is that they are usually preprocessing steps before another learning algorithm is deployed. I on the other hand don’t want to use another learning algorithm. The entire task is to figure out the most important features.

Are feature selection algorithms really the best way to proceed, or are there any better approaches? Maybe some techniques especially designed for my kind of problems?

Many thanks in advance!


texts: Spacy - Ignored (essential) entities during labeling - how important is this aspect?

texts:
I'm training a custom Spacy model. Does it affect the results whether I leave entities unlabeled? For example, Germany is LOC. In one example (1) I label this. In the other example (2) I ignore it and label another entity:
Example 1: Germany (LOC) is a country in Central and Western Europe.
Example 2: Germany borders Denmark (LOC) to the north.
It's not about one entity that is occasionally ignored, but about several.


texts:
I am trying to learn some EDA skills and I get a scatter like this way:

I don't see that there are some model relationships or group-related patterns between these two data. I mean what I thought was right? I am not pretty sure about it.

If not mind, could anyone help me and give me some advice on multivariate plotting.

Thanks in advance.


texts: How to apply Kalman Filter for Cleaning Timeseries Data effectively without much optimization?

texts:
Someone gave me a tip to use kalman filter for my dataset. How time intensive is it to get a good kalman filter running, compared to simple interpolation methods like
df.fillna(method="")

which takes basically no effort.
If one or two iterations is enough to get useful results, which come very near the real missing value, then I am willing to take the effort to implement it. (Dataset length 100.000 up to 200mio rows)
If it needs to be optimized like a Neural Network itself which can be costly in terms of time, isnt it better to simply use an LSTM?


texts:
Here is my code:

ctrl <- trainControl(method = "cv")
fm <- train(y=train.df$Price,x=train.df[,-1], method = "leapSeq", trControl = ctrl)
fm$results
coef(fm$finalModel)

What I get back is this:

      nvmax RMSE Rsquared  MAE RMSESD RsquaredSD MAESD
    1     2 1888    0.760 1315    266     0.0703   130
    2     3 1530    0.842 1171    221     0.0458   157
    3     4 1707    0.809 1231    244     0.0315   132
> coef(fm$finalModel)
Error in coef.regsubsets(fm$finalModel) :
  argument "id" is missing, with no default

There is no Coefficients tab under finalModel.  If I use "lm" for method instead of "leapSeq" the coefficients tab is there in finalModel.

Any guidance is appreciated.


texts: How can I fix my classifier only predicting two classes, and do my metrics show that it is overfitting?

texts:
I have a relatively simple 16 feature neural network attempting to predict the outcome of a sports event as win, loss, draw, however regardless of the number of layers, or the number of nodes in said layers, or the learning rate, etc; it still only predicts home and away wins, it never predicts a draw, and even then seems to disproportionately predict home wins. I have a good amount of data, c300,000 matches. I currently have 4 layers in my network, with v.low learning rate (0.000001), 0.001 L2 reg, 0.05 dropout between layers, and test between 60 and 2000 nodes per layer, but no configuration I can find will predict draws in the output. The labels are approx 45% home wins, 25% draws, and 30% away wins. I can show code as required.
Secondly, I'm also unsure if these accuracy and loss metrics show that I'm overfitting, or if the network is still learning. It appears to get to c48% accuracy relatively quickly then make tiny improvements, but I'm really not sure if those improvements are just overfitting.
May be worth noting that I have tried training various network iterations on single class data, and it appeared to train to each class pretty quickly. Otherwise the only things I've tried is changing the number of layers, nodes, hyperparameters, etc.
The first bar chart shows the classes output by the model on the basis of the test_x data - note it only ever predicts home win or away win, never a draw. The second bar chart shows the frequency of the test_y classes - I worry that fact that I'm nowhere near these suggests I have more fundamental issues? Any clarifications or uncertainty just let me know and I'll respond as quickly as possible.
This first bar chart shows the classes predicted by my model on the test set; the second shows the actual class balance in the test set.

Regarding the possible overfitting, here are my graphs of loss and accuracy:

Any help is great appreciated!!


texts:
I have been tasked with creating a pipeline chart with the live data and the budgeted numbers.

I know what probability of each phase of reaching the next.  The problem is I have no Idea what to do about the pipeline budgeting with regards to time.  For instance what period of time should I have closed sales in the chart.

I have honestly been working on trying to figure it out.  Each successive revision gets me farther from the answer.


texts: How fit pairwise ranking models in xgBoost?

texts:
As far as I know, to train learning to rank models, you need to have three things in the dataset:

label or relevance
group or query id
feature vector

For example, the Microsoft Learning to Rank dataset uses this format (label, group id and features).

1 qid:10 1:0.031310 2:0.666667 ...
0 qid:10 1:0.078682 2:0.166667 ...

I am trying out xgBoost that utilizes GBMs to do pairwise ranking. They have an example for a ranking task that uses the C++ program to learn on the Microsoft dataset like above.

However, I am using their Python wrapper and cannot seem to find where I can input the group id (qid above). I can train the model using just the features and relevance scores but I feel like I am missing something.

Here is a sample script.

gbm = XGBRegressor(objective="rank:pairwise")

X =  np.random.normal(0, 1, 1000).reshape(100, 10)
y = np.random.randint(0, 5, 100)

gbm.fit(X, y) ### --- no group id needed???

print gbm.predict(X)

# should be in reverse order of relevance score
print y[gbm.predict_proba(X)[:, 1].argsort()][::-1]


texts: Classifying Data in Orange: What's the difference between 'Features', 'Target Variables', and 'Meta Attributes'

texts:
I've been looking for a more visual way to calculate statics on my spectroscopy data.  Python's Orange Canvas seems like it might be a great alternative to Pipeline Pilot, but I'm having trouble getting started with more simplistic data analyses.

Most notably, when I import data, I'm unclear as to what the difference between a 'Target Variable', 'Feature', and 'Meta Attribute' is.

How do I decide what type of data I have?


texts: Public dataset for news articles with their associated categories

texts:
I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc.

I want to use such dataset for topic detection of various sentences or paragraphs. I was planning to train a classifier with such a dataset and use it for predictions. However, I couldn't find any. Are there any such known datasets available?


texts: New to data Science. Which techniques best to use Large data set in insurance company?

texts:
Dataset Features

Insurance underwriting dataset for 8 years.

age
location
amount insured
some other features...(medical evidence)

Not all feature will be available to all applicants.

Target Variable

Decision on whether the applicant can be insured

Question

What techniques can be used and which ones would work best?
Outline a high-level overview - I do not think I have to go into too much detail as I do not have any data.

Things I have considered

I am thinking to first slice the data and analyse it in parts and see can I find  a pattern.

Regression analysis could be carried out. Possible Logistic regression?

I could take a sample and perform hypothesis tests?

I know that this is an ideal machine learning situation. I don't have any experience in this field and I think I am better to stick with methods I have some knowledge of.

I know this is very ambiguous, but a little nod in the right direction and I would be very appreciative.


texts: Scalable test-to-control matching method needed for Amazon cloud environment

texts:
I'm working on a project to port some of my employer's processes from our local Unix servers to the Amazon cloud. One process matches records from a test group to records from a control group using propensity score matching. Local process is implemented with some messy SAS scripts. I'm looking to replace the SAS scripts with something that can run in the Amazon cloud and that satisfies these conditions:

can handle large data sets (e.g., matching 10 million test records to 30 million control records)
not SAS (since we have little SAS expertise in-house)
leverages open source languages/frameworks (because cost matters)

At the moment, we're evaluating Python. We're able to use LogisticRegression in sklearn.linear_model to compute propensity scores, but we're struggling to find a scalable method to do the matching task. We tried using linear_sum_assignment() from scipy.optimize since it's intended to find optimal bipartite matching, but it ran for nearly five hours on a just small subset of the records before we terminated the job. I feel fairly sure this kind of assignment/matching problem is common enough that there might be some generalized solutions that we could leverage, but we've yet to find one.
To elaborate, the "matching task" is a process to match Test and Control records with very similar propensity score values. It's a fairly common technique for observational studies. Propensity scores are computed and assigned to all Test and Control records. Then Test records are matched 1-to-1 with Control records, provided the records have propensity scores within a specified difference or "caliper." Test records are matched to a single Control record, and vice versa. Some Test records might not be matched if they don't have a suitable partner in the Control group.


texts: How many minima does the residual sum of squares have for the logistic curve?

texts: Suppose we had some data points $\{(x_i,y_i)\}$ where $x_i$ is a real number and $y_i$ is zero or one. We want to fit the logistic function to the data where the logistic function is $\hat{y}=\frac{1}{1+e^{-(\beta x +\beta_0)}}$. To do this we would like to choose $\beta_1$ and $\beta_0$ in order to minimize the residual sum of squares, $RSS = \sum_i (y_i-\hat{y}_i)^2$.[nevermind that cross-entropy is a better cost function.] I believe there is only a single minimum. How can I know? Does anyone know of a proof for this or a counter example?


texts: How to adaptively sample n-dimensional data and build an optimum training set

texts:
My input space is at least 10 dimensional (after reducing it by various component analysis such as PCA) and the output space is 4 dimensional. I am building a neural network that works something like a function approximator which takes the above 10D data as input (10 neurons in input) and spits out 4D data as output (4 neurons at the output). In between there are hidden layers.

I need to build a good training set that covers all the possible values of input and output. Although it may seem like there is a very large combination possible for 10D input + 4D output, in reality, the input/output is restricted by real world constraints. The problem I'm facing is this: In some part of the training sets, I need to sample the data with higher resolution, in other parts, I can go away by sampling with lower resolution. I can obviously sample the entire dataset with high resolution, but then the number of samples is more than 10 trillion and I know that most part of the dataset is pretty slowly varying and sampling this part with lower resolution is enough. My guess-estimate is, if I could sample it properly, I could get away with 4-5 orders of magnitude less numbers of sample.

My questions is, what should I do/build in order to have something that will adaptively sample a high dimensional dataset (change resolution whenever necessary) and give me an optimum training set?

NB: one of my colleagues advised me to use Markov Chain Monte Carlo method. I am not sure whether it is the best way to go. Please share your opinions.


texts: What ML techniques work on imbalanced datasets

texts:
I have some specific questions for which I could not find answers in textbooks/research articles. Shall be grateful for an answer. These are:

Are there ML techniques that can be directly applied on class imbalanced datasets? OR is it a practice to balance the dataset either by using some weighted approach or SMOTE methods? What is the standard way for real datasets/industries? I am referring to fraud detection, anomaly and water leak detection where inherently the dataset would always be imbalanced.

Let's say I do class balancing by some weighted loss function. This loss function would be calculated on some amount of data say 100 examples of streaming data. Then during deployment phase, I may not have 100 examples of data coming; it could be more or less than what was used in training. The weighting approach is say the inverse class frequency approach which depends on the number of examples. So if the weighting approach was used on 100 examples during training, then during deployment phase I should always have 100 examples to work on to make some prediction on all those examples? Or there is no dependency between number of training examples and number of examples during deployment when doing class balancing?


texts: Can random forest algorithm provide customer churn prediction probability at each customer instead at class level?

texts:
I have customer training data set from telecom industry along with its test data set containing churn values 0 & 1 for each customer. I also have customer data set whose churn value is to be predicted ie 0 & 1. It is also required to get the churn prediction probability at individual customer level so that they can be arranged in descending order of the propensity to churn

For brevity, showing limited features
cust_train.xls

cust_id Account Length  VMail Message   Day Mins    Eve Mins    Night Mins  Intl Mins   CustServ Calls

cust_train_output.xls

cust_id churn (0/1)

I want to know if it is possible to get the churn prediction probability at individual customer level & how by random forest algorithm rather than class level provided by:
predict_proba(X) => Predict class probabilities for X.

Goal is to arrange the customer in descending order of the propensity to churn.

Alterntively, is this possible with Logistic Regression Model?

Thanks


texts: Understanding additive function approximation or Understanding matching pursuit

texts:
I am trying to read Greedy function approximation: A gradient boosting machine.
On page 4 (it is marked as page 1192) under 3. Finite data the author tells how the function approximation approach breaks down when we have finite data and some way to impose smoothness is needed to get a function that can be used at points other than the ones provided in the training dataset. One way it suggests is to use parametric base functions (like in neural networks) and in case that is infeasible, we can try a 'greedy stagewise' approach. This is called 'matching pursuit' in signal processing.
I understand how, in a non-parametric approach, you cannot make a prediction at points other than the ones provided in the training dataset. I understand why a parametric approach would work to solve that issue. I don't understand :
What are the implications and significance of 'greedy' and 'stagewise' in the 'greedy stagewise' approach? Can you please explain what is being said in that paragraph.


texts: Is it possible to generate syllogisms using an NLP algorithm?

texts:
I want to build a tool that generates sensible syllogisms. An example of a syllogisms is: all A are B. all C are A. all C are B). I want the triplet (A, B, C) to be semantically related to each other in the way described by the syllogisms. That is, I would like my tool to generate 'a=humans; b=mortal; c=greeks' and not 'a=chickens; b=burgers; c=frogs'.

There's a syllogism generator online (http://krypton.mnsu.edu/~jp5985fj/courses/609/Logic/Silly%20Syllogisms.htm) but it doesn't generate syllogisms that are semantically related, it generates random terms for A, B, and C which may or may not be related.

My question is, in NLP, are there any research papers for generating semantically valid syllogisms? What topics would I need to research to build this tool?


texts:
I am writing a thesis and don't know if the charts below should have grid lines or not. Do they need the grid lines?

An example without the grid lines:


texts: Pandas dataframe, create columns depending on the row value

texts:
I get a csv that if you read it, looks like:

import pandas as pd
df = pd.DataFrame([['de,ch,fr', '1,2,3'],['fr,ch,dk', '3,4,5']], columns=['countries', 'numbers'], index=['abc', 'bcd'])

I want to make it look like this:

df = pd.DataFrame([[1,2,3,0], [0,4,3,5]], columns=['de_number', 'ch_number', 'fr_number', 'dk_number'], index=['abc', 'bcd'])

Meaning exploding the countries column and getting for every value in index the number for every country in a separate column. I have the list of all the countries for this dataframe beforehand (meaning that I knew beforehand that I'm going to have the values ['de', 'ch', 'fr', 'dk'])

Is there a nice clean way of doing it? Everything that comes into my mind is quite messy.


texts: Training on derived features that won't be present in a test set

texts:
So I have an odd real-world problem in which the data that's going to be fed to a categorical predictive model has (a few) certain features, and when building a model I can bolt on additional datasets that add context to some of those features, but when it's going to be running live I won't be able to use any of that additional data - the model itself needs to already "know" what's important about specific values of the original data and be able to operate just on those attributes.

The major features here are time(/date/DOW) and location (lat/long) within a city.  So, the additional datasets are context about that location (zoning, demographics, transit) or time (traffic, commercial activity, etc).  But for performance reasons (join cost, etc) we can't use them live, we only have the time and location numbers (plus easily parse-able derivatives).

I'm imagining a process whereby the model uses those additional features to learn which subsets or demarcations of time-and-location actually matter, and can then continue to use those weightings and rules or whatever even in the absence of the additional features, just relying on the numerical values.  But it's also possible that there's no such approach.  Can anyone suggest an algorithm or approach that could let us retain some of that value?


texts: Tensorflow dense layers worse than keras sequential

texts:
I try to train an agent on the inverse-pendulum (similar to cart-pole) problem, which is a benchmark of reinforcement learning. I use neural-fitted-Q-iteration algorithm which uses a multi-layer neural network to evaluate the Q function.

I use Keras.Sequential and tf.layers.dense to build the neural network repectively, and leave all other things to be the same. However, Keras gives me a good results and tensorflow does not. In fact, tensorflow doesn't work at all with its loss being increasing and the agent learns nothing from the training.

Here I present the code for Keras as follows

def build_model():
    model = Sequential()
    model.add(Dense(5, input_dim=3))
    model.add(Activation('sigmoid'))
    model.add(Dense(5))
    model.add(Activation('sigmoid'))
    model.add(Dense(1))
    model.add(Activation('sigmoid'))
    adam = Adam(lr=1E-3)
    model.compile(loss='mean_squared_error', optimizer=adam)
    return model

and the tensorflow version is

class NFQ_fit(object):
"""
neural network approximator for NFQ iteration
"""

def __init__(self, sess, N_feature, learning_rate=1E-3, batch_size=100):

    self.sess = sess
    self.N_feature = N_feature
    self.learning_rate = learning_rate
    self.batch_size = batch_size

    # DNN structure
    self.inputs = tf.placeholder(tf.float32, [None, N_feature], 'inputs')
    self.labels = tf.placeholder(tf.float32, [None, 1], 'labels')
    self.l1 = tf.layers.dense(inputs=self.inputs,
                         units=5,
                         activation=tf.sigmoid,
                         use_bias=True,
                         kernel_initializer=tf.truncated_normal_initializer(0.0, 1E-2),
                         bias_initializer=tf.constant_initializer(0.0),
                         kernel_regularizer=tf.contrib.layers.l2_regularizer(1E-4),
                         name='hidden-layer-1')
    self.l2 = tf.layers.dense(inputs=self.l1,
                         units=5,
                         activation=tf.sigmoid,
                         use_bias=True,
                         kernel_initializer=tf.truncated_normal_initializer(0.0, 1E-2),
                         bias_initializer=tf.constant_initializer(0.0),
                         kernel_regularizer=tf.contrib.layers.l2_regularizer(1E-4),
                         name='hidden-layer-2')
    self.outputs = tf.layers.dense(inputs=self.l2,
                                   units=1,
                                   activation=tf.sigmoid,
                                   use_bias=True,
                                   kernel_initializer=tf.truncated_normal_initializer(0.0, 1E-2),
                                   bias_initializer=tf.constant_initializer(0.0),
                                   kernel_regularizer=tf.contrib.layers.l2_regularizer(1E-4),
                                   name='outputs')
    # optimization
    # self.mean_loss = tf.losses.mean_squared_error(self.labels, self.outputs)
    self.mean_loss = tf.reduce_mean(tf.square(self.labels-self.outputs))
    self.regularization_loss = tf.losses.get_regularization_loss()
    self.loss = self.mean_loss # + self.regularization_loss
    self.train_op = tf.train.AdamOptimizer(learning_rate=self.learning_rate).minimize(self.loss)

The two models are the same. Both of them has two hidden layers with the same dimension. I expect that the problems may come from the kernel initialization but I don't know how to fix it.


texts: Python Convert indexes starting from 1 to 365 into months names and display them into the x axis without changing the plot structure?

texts:
I need to display months on x axis of a plot instead of the indexes number of a data frame that goes from 1 to 365, which they represent the number of day of the year. So instead of an x-axis which goes from 1 to 365, I want to display it as "Jan", "Feb" and so on, without losing the structure of the plot.
Here is the main structure of my data frame:
    Month   Day Max_Data    Min_Data    MonthDay
1   1       1   1.1        -13.3        1-1
2   1       2   3.9        -12.2        1-2
3   1       3   3.9        -6.7         1-3
4   1       4   4.4        -8.8         1-4
5   1      5    2.8        -15.5        1-5

I am currently plotting using:
plt.scatter(data_2015.index, data_2015['Max_Data'], marker='^', color='green',s=40, alpha=1.0)

And if I changed data_2015.index to Month the graph will plot a different a completely wrong values, as they 28, 30 or 31 rows for each month.
So what is the way to convert indexes into month and display them into the x axis of a plot?


texts: Fields list of YearPredictionMSD (identify year of music from Million Song Dataset)

texts:
The YearPredictionMSD dataset, is a machine learning practice dataset extracted from the Million Song Dataset and found in the UCI Machine Learning Repository. I couldn't find the field list that describes the features. Does anyone know where I can find them?

Also, I'm confused why there are only 54 features in the original data from LabRosa but 90 in the subset of the data on UCI.


texts: Is sampling a valid way to reduce complexity?

texts:
I'm facing an issue where I have a massive amount of data that I need to cluster. As we know, clustering algorithms can have a very high O complexity, and I'm looking for ways to reduce the time my algorithm is running.
I want to try a few different approaches, like pre-clustering (canopy clustering) or subspace clustering, correlation clustering etc.
However, something that I haven't heard about, and I wonder why - Is it viable to simply get a representative sample from my dataset, run the clustering on that, and generalize this model to the whole dataset? Why/why not is this a viable approach? Thank you!


texts: What is considered a large (medium, small) input for simple feedforward neural networks?

texts:
What would be considered a large (or medium or small) number of input neurons for a feedforward neural network?

While I am trying to do phoneme detection using inputs of 3200 sound-samples, I became curious how to categorize input sizes of a simple 3 layered ANN (input - hidden - output).


texts: Classification with two 1-NN classifiers

texts:
I have two images, let's say old and new. In the old one there are 1:M objects, in the new one 1:N. I need to label the objects in the new one based on some metric (yes, I realize object tracking is a whole complicated area, but I need some   entry point right now). So I've obtained a metric based on the distance to the objects in the previous image, sort of 1-NN. I've also obtained a different metric, also sort of 1-NN, based on pixel intensity (color). Of course, I want to use both of these features somehow. For example, based on the distance metric object 3 in the new is closest to object 5 in the old, and based on the pixel intensity metric - closer to object 9 in the old image.

Any suggestions on how to combine these two? Perhaps some kernel, but I'm not sure.


texts: Open source interactive dashboards for python

texts: I want to create a dashboard in Python for some graphical reporting. But the website should be deployable. I want something similar to Dash. I want to display plotly plots with interactive widgets. But deploying Dash requires 'Dash enterprise' which is not free/open-source. What is a good alternative?


texts: How to use SVM in Orange for sentiments analysis? We will generate sentiments using sentiments analysis widget but now how to use SVM and how to teach SVM in Orange? i am use xlsx file format then it work good below example ... but i want to use text file (csv) format in which generate error target variable set how to remove this error


texts: how can i leverage NLP features like SRL, LSA, POS, NER, entity type , relation type with deep learning to find semantic similarity of texts?

texts:
I have texts similar to the ones below, and I want to find semantic similarity between these texts and the intent.

  1) what are the different steps to resign ?

  1) procedure to resign ?

  1) what are the process do i have to follow to resign ?

  1) what are the common schemes followed for resignation by any company !!

  2) to whom i need to contact after i resign ?

  3) i did resign today.

  4) if you misbehave again , i'll resign !!

  4) your rude behavior made me to resign.

  5) different company have different policy on resignation !!

To tackle above problem i found " Quoras Model to handle Duplicate Questions " interesting . so i thought let's try this but with small changes. and changes is instead doing binary classification , let's do multi class classification. so i was trying to do multi class classification on a "Quora Data Set". my intention was to classify each question with their duplicate questions into one class on a vector space using LSTM, CNN models. LSTM worked if sequence remain unchanged. but as you know questions structure can changes keeping intent same (or we can call them as duplicate questions.). so to handle such variation of question i tried with CNN. CNN worked for some small data set but for large data set it become very sensitive or overlapping.

So i am thinking " can i leverage NLP with deep learning to find intent and semantic relation ?"

I have POS tags, NER, SRL, LST, entity type, relation type, etc features. How can i leverage these features of NLP in deep learning to achieve a state-of-the-art result?

There is a paper, "When Are Tree Structures Necessary for Deep Learning of Representations?", that uses parse tree as input for recursive neural network model. Is there any similar work? Can any one give me data sets used in this paper?


texts: Difference between a parameter and a weight?

texts:
Is there any pedantic difference between referring to a specific "parameter" of a model as $\theta_1$ or referring to a specific "weight" of that model as $w_1$?

Are they always the exact same thing?


texts:
As far as I know, we need to use training data to find out the relation between the features, also known as input values, and labels, that are output values, in supervised learning. After that, by using this relation, our learning system tries to predict labels of data samples in next data sets.

However, there is no need to find out such a relation in unsupervised learning because the data samples do not have labels; they only consist of features. In this case, do we need a training set in unsupervised learning ?


texts: Are there any libraries in Python Pandas or statmodels to distinguish between addictive and multiplicative time series data

texts:
When I did research on identifying addictive models and multiplicative models in time series data, many sources are stressing on visual identification by looking at the graph.

There are more reliable methods than this however all explained by Statistical calculations. Additive vs Multiplicative model in Time Series Data

Are there any python Pandas or statsmodels or any other libraries that can do the identification part


texts:
LeNet accepts 32X32 image. So, to use LeNet for MNIST dataset,we have to change the size from 28X28 to 32X32. I came across This implementation. i am confused about how the following line of code work.

np.pad(X_train, ((0,0),(2,2),(2,2),(0,0)), 'constant')

Above line of code pad 28X28 pixed image to become a 32X32 image. Can anyone help me understand how exactly its done.


texts: Choosing between two object detection model checkpoints

texts:
I have the mAP scores for an object detection model evaluated at different model checkpoints. I want to choose the model that has the maximum recall on the test dataset, meaning the model which missed the least number of objects. However, the model that achieves this doesn't have the highest mAP.

mAP is the most common metric for evaluating an object detection model. How do you interpret this metric?

For the best real world performance, is it better to:

Choose a model checkpoint based on the least number of false negatives, even though the mAP isn't the highest?
Choose a model checkpoint with the highest mAP, even if it means the number of false negatives is higher?


texts: Gradient descent formula implementation in python

texts:
So I recently started with Andrew Ng's ML Course and this is the formula that Andrew lays out for calculating gradient descent on a linear model.

$$ \theta_j = \theta_j - \alpha \frac{1}{m} \sum_{i=1}^m \left( h_\theta(x^{(i)}) - y^{(i)}\right)x_j^{(i)} \qquad \text{simultaneously update } \theta_j \text{ for all } j$$

As we see, the formula asks us to the sum over all the rows in data.

However, the below code doesn't work if I apply np.sum()

def gradientDescent(X, y, theta, alpha, num_iters):

    # Initialize some useful values
    m = y.shape[0]  # number of training examples

    # make a copy of theta, to avoid changing the original array, since numpy arrays
    # are passed by reference to functions
    theta = theta.copy()

    J_history = [] # Use a python list to save cost in every iteration

    for i in range(num_iters):
        temp = np.dot(X, theta) - y
        temp = np.dot(X.T, temp)
        theta = theta - ((alpha / m) * np.sum(temp))
        # save the cost J in every iteration
        J_history.append(computeCost(X, y, theta))

    return theta, J_history

On the other hand, if I get rid of the np.sum(), the formula works perfectly.

def gradientDescent(X, y, theta, alpha, num_iters):

# Initialize some useful values
m = y.shape[0]  # number of training examples

# make a copy of theta, to avoid changing the original array, since numpy arrays
# are passed by reference to functions
theta = theta.copy()

J_history = [] # Use a python list to save cost in every iteration

for i in range(num_iters):
    temp = np.dot(X, theta) - y
    temp = np.dot(X.T, temp)
    theta = theta - ((alpha / m) * temp)
    # save the cost J in every iteration
    J_history.append(computeCost(X, y, theta))

return theta, J_history

Can someone please explain this?


texts: Machine Learning: Predicting target based on a feature

texts:
I have a df looks as follow:

-It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times.
-type values are in the range of 1..4.
I want to train a model, that for each feature1Xfeature2Xfeature3 combination as input, will predict a type s.t this type will match to the lowest price. e.g, if there are two rows with the same feature1Xfeature2Xfeature3 = [5.0, 3.6, 1.4] combination, one has price of 5000 with type 3, the other has price of 2000 with type 1, then inputting the model with feature1Xfeature2Xfeature3 = [5.0, 3.6, 1.42] (a combination which is very close to the above one) will likely output 1 since it has lower price.
I've got some experience with models and learning using python and sklearn, but it sums to training a model with one target, which is not based on a specific feature.
The final goal is to try and fit several algorithms on this problem(KMeans, DecisinTree, NN..), and check who did the best.
Any help will be appreciated.
Thanks


texts: Optimizing co-occurrence matrix computation

texts:
I am computing co-occurence matrix for a fixed windows size in python using scipy's lil_matrix for storing the counts and computing the counts by sliding the context window over each word and then counting in the window.

Now the code is taking too much time for relatively small corpus size also (100 MB Wikipedia dump). The code is :

def gen_coocur(window_size=5):
        '''
        Generates coocurrence matrix
        '''
        # vocab is precomputed.
        coocur_matrix = lil_matrix((len(vocab)+1, len(vocab)+1), dtype=np.float64)
        for page in self.wiki_extract.get_page():
            # word_tokenize is tokenizer from nltk
            doc_tokens = word_tokenize(page.decode('utf-8'))
            N = len(doc_tokens)
            for token in self.vocab:
                for i in xrange(0,window_size):
                    if (token in doc_tokens[0:i] or token in doc_tokens[i:(i+window_size+1)]) and token != doc_tokens[i]:
                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1
                for i in xrange(window_size, (N-window_size)):
                    if token in doc_tokens[(i-window_size):(i+window_size+1)] and token != doc_tokens[i]:
                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1
                for i in xrange(N-window_size, N):
                    if (token in doc_tokens[i:N] or token in doc_tokens[i-window_size:N]) and token != doc_tokens[i]:
                        coocur_matrix[self.vocab[doc_tokens[i]],self.vocab[token]] +=1

vocab is a dictionary which maps words -> wordId. How can I optimize this code to run faster?


texts: Scaling continuous data to discrete range

texts:
Edit: the context is as follows: I've trained some ML model that predicts some feature vector. Thats a. But I know that a can take some values from discrete range, but the model outputs continuous values. After applying predict I want to enforce the outputs to take the discrete values they are allowed to be in. I've also edited my example to be more informative.
I have this continuous data:
a = [0.003 0.994 1.334 3.2 1.464 2.454 2.941 999.999] (outputs)
How can I easily convert it to the following discrete array:
b = [0 1 2 3 3] (allowed values)
so that every element of a will be mapped to its closest counterpart in b:
a_scaled = [0 1 1 3 1 2 3 3] (what I need)
I know how to implement this, but I want to stick with "don't invent the wheel". So is there any nice function from some library that does this?


texts:
I have been reading through Chapter 9 of www.deeplearningbbook.org, where convolutional networks are being described.

The following image represents the output of a 2D convolution, without kernel flipping.

The book goes on to describe this matrix as a Toeplitz matrix where,

  for univariate discrete convolution, each row of the matrix is constrained to be
  equal to the row above shifted by one element.

I fully understand this statement since w, x, y and z are constants in their respective columns with shifting elements.

However, there is no mention of diagonal-constants which are a key feature of such matrices. As per Wikipedia (link above) and several other sources:

  a Toeplitz matrix or diagonal-constant matrix, named after Otto Toeplitz, is a matrix in which each descending diagonal from left to right is constant.

Going back to the image above, does this mean that, for instance, aw == cx == gy? How can this be ensured when all elements are different?


texts: Excel All Possible Combinations for MacOS

texts:
Is there a formula in Excel to calculate all possible combinations for multiple lists for MacOS. For instance, I am trying to draft a fantasy team and I have the following combinations:

(harden, westbrook, antetokounmpo)
(davis, leonard, irving, george)
(vucevic, holiday, beal, walker, conley)
(randle, collins, makkanen, williams, harrell, conley)
(paul, levine, bledsoe, gordon, young, turner, jackson, faried, horford)
(siakam, ibaka, dunn, young, collison, adams)

Basically, I want to figure out how to calculate all possible combinations for these 6 groups.


texts: Sequential Modelling: Multiple Sequence to One or Sequence to Sequence

texts:
Suppose I have a single sequence of $x_1, x_2, ..., x_n$ and corresponding labels $y_1, y_2, ..., y_n$.

An example would be a person makes website visits $x_i$ and the label $y_i$ tells us if there was a purchase in any of the previous visits.

Is there a difference between training multiple sequence to one or a single sequence to sequence?

Multiple sequence to one

This approach gives us $n$ data points. Additionally it seems that the model will see $x_1$ n times, while $x_n$ will be only fed in once.

$[x_1] , [y_1]$

$[x_1, x_2] , [y_2]$

...

$[x_1, x_2, ..., x_n], [y_n]$

Single Sequence to Sequence

This approach gives us just 1 data point.

$[x_1, x_2, ..., x_n], [y_1, y_2, ..., y_n]$, with no repetition in data.

Do these yield the same outcome in RNN (e.g. LSTM) training? It seems that when your data is imbalanced, you would greatly benefit from the 1st approach, since you get more observations.

My understanding is that, since even a single data point is technically a valid data point in my case, I should be using the multiple sequence to one approach. But in cases of NLP (where the first word alone has no context), obviously we need the whole sequence.


texts: What is a subspace and what is a shared subspace?

texts: I'm totally new in machine learning. The first confusing concept is subspace. In multi label classification we have to share the subspace. What does one mean by that shared sub spaces?


texts: how to scale a dataset contains a b&w and Grayscale images

texts:
I have a dataset that contain both black and white images and a grayscale images (some of them are scnned by printer and other by camera and changes into gray)
how can make or scale my dataset so I can use it to feed a model ?


texts: Deep advantage learning: how to predict the value

texts:
I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym
For deep q-learning, you need to calculate the q-values that should be predicted by your network. There are many strategies: monte-carlo, temporal-difference, TD(lambda), ...
Basically you get a tradeoff, between the number of steps you look into the future, and the variance of your observations.
Instead of predicting the q values, it is also possible to predict advantages. Where A(s, a) = Q(s, a) - V(s). So the advantage describes how much more than expected you get. It is discussed and motivated here.
For predicting the q values, you have to balance variance against the number of steps to look into the future. For the advantages, there is a method called generalized advantage estimation (GAE) which does this in a very neat way:
https://arxiv.org/abs/1506.02438
I would like to predict those advantages, instead of q values. That is by no means a new idea, and apparently, advantage learning can outperform q-learning: http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/glossary/advantage.html
The above link is a very small abstract on advantage learning. The important part is:

Advantage learning [...] requires only that the A(x,u) advantages be stored

But how do I do that ?
The GAE paper assumes that I can predict the value for every state. I need the values to calculate the advantages.
As far as I can see, I have to options:

Predict only advantages and then somehow calculate the value from the advantage. As far as I can see, we need two out of: q-values, advantages, values. So if I don't want to predict the values, I have to predict the q-values. Which is the original problem. I read somewhere that the maximum advantage is the value, but that makes no sense to me and I can no longer find the link.

Predict both advantage and value. If I do this, implementing GAE and training the network to predict the advantages correctly would be simple. But what would I use as training target for the value ? If I use the GAE formulation for the advantages, it looks many steps into the future. It seems nonsensical to base the calculation of those advantages on a value function approximation that I train on one step lookaheads.

My question is not how to set up a function approximator, or what network types would be well suited for this. My question is: What are the target values for the value function, that I can feed to my function approximator.
How do I actually train it ?


texts: I create a Random Forest and Gradient Boosting Regressor by using GridSearchCV. For the Gradient Boosting Regressor, it takes too long for me. But I need to know which are the best parameters for the models. So I am thinking if there is a GridSearch without CV because the OOB score is sufficient to evaluate the models. I hope I could explain what I mean.


texts: Implementing experience replay in reinforcement learning

texts:
I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay. My question is that whether we update the parameters $\theta$ of function $Q$ once for all the samples of the minibatch or we do that for each sample of the minibatch separately?

According to the following code from this paper, it performs the gradient descent on loss term for the $j$-th sample. However, I have seen other papers (referring to this paper) that say that we first calculate the sum of loss terms for all samples of the minibatch and then perform the gradient descent on this sum of losses.


texts:
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.

I have just started with Data Science and Machine learning. I am looking for long term project ideas which I can work on for like 8 months.

A mix of Data Science and Machine learning would be great.

A project big enough to help me understand the core concepts and also implement them at the same time would be very beneficial.


texts: Sudden drop in loss at epoch start, then gradual raise

texts:
I'm building an autoencoder for natural language words.
The question: Why do I see at the start of each epoch a sudden drop in the loss and then gradual bounce back (loss raises but slowly) followed by a gradual decrease in the loss until another epoch?
The graph of the loss looks a bit like this:

(it's a mock i've made in paint brush:)
Here is the model setup:
#input is a (35,35) matrix of one-hot vectors (one vector for each char in a word)
inputs = layers.Input(shape=(35,35), name='input')
encoded = layers.Masking(mask_value=0., input_shape=(35, 35))(inputs)
encoded = layers.Bidirectional(layers.GRU(256,return_sequences=True))(encoded)
encoded = layers.GRU(32)(encoded)
decoded = layers.RepeatVector(35)(encoded)
decoded = layers.GRU(32,return_sequences=True)(decoded)
decoded = layers.Bidirectional(layers.GRU(256,return_sequences=True))(decoded)
decoded = layers.TimeDistributed(layers.Dense(35,activation='softmax'))(decoded)
model = keras.Model(inputs, decoded, name="toy_model")
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_crossentropy'])
model.fit(x_train, y_train, shuffle=True, batch_size=32, epochs=10)

p.s. As far as I understand I do shuffle data on each epoch (see the model.fit(shuffle=True))


texts: How to cross-validate a deep learning model for highly imbalanced datasets?

texts: I am working with a multi-modality classification problem (with Keras). I have 1000, 5000 and 10000 samples for three different classes. I would like to do a five fold cross-validation to select the best pre-trained deep learning model for deployment. I'm including class weights during model training to give more weightage for less-pronounced classes. For a given fold, I would be validating with 200, 1000 and 2000 samples from these three classes. Is accuracy a good prediction measure to be used in this case? Or do I have to measure the F1-score and Matthews correlation coefficients as well? Am I doing it right?


texts:
Hey guys I'm currently reading about AUC-ROC and I have understood the binary case and I think that I understand the multi-classification case. Now I'm a bit confused on how to generalize it to the multi-label case, and I can't find any intuitive explanatory texts on the matter.

I want to clarify if my intuition is correct with an example, let's assume that we have some scenario with three classes (c1, c2, c3).

Let's start with multi-classification:

When we're considering the multi-classification setting we look at each label separately.

So if we're looking at the ROC for label c1, we can bunch together c2 and c3 as "negatives".

I.e, when we have a sample that belongs to c1, we only look at the predictive score of c1, and build a predictive score distribution of the positive samples. Then we look at samples that belongs to either c2 and c3,i.e the bunched together negative samples and we look at their prediction scores and build a distribution of those scores as well. This results in something like the following:

Based on those distributions we can get the TPRs and FPRs based of some thresholds and calculate the ROC for c1. Then we can do the same for c2 and c3, and if we want we can average over the three ROC curves, to get an aggregated score for the problem.

That's my intuition so far anyways.

But what about the multi-labeled scenario?

Here's when things get confusing for me. Do we calculate it in the exact same manner? I understand that we still calculate the ROC for every class individually, but I'm not sure of how to think of it. Let's say that we're looking at it from the perspective of class c1. For every sample that is regarded as c1 (and possibly c2 and c3 as well), we add the model's predictive score for c1 the the distribution. But what if we encounter for example a sample that is regarded as c2 AND c3 (this can't happen in the multi-class scenario), do we think of this as TWO negative samples and add two predictive scores to the distribution?

Am I thinking in the right tracks here?


texts: How to automatically mount my Google Drive to Google Colab

texts:
I have recently discovered Google Colab and I am wondering if there is an option to permanently authorize Google Colab to access and mount my Google Drive.

from google.colab import drive
drive.mount('/content/drive')
Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=xxx
Enter your authorization code:
··········
Mounted at /content/drive

Right now, I need to all the time insert manually the authorization code and I want when I start a new Google Colab to have my drive mounted.

Is there an option to do that or I should execute this all the time manually?