texts:
The use case :

Everyday, we have metrics that are established daily to check that our systems are doing fine. From time to times, bugs occur in the workflow building these metrics, and I have to build an algorithm that will alert us when it seems there is a problem, so that we can check where it is coming from.

What I'm trying :

I don't know much about time series. The data I have displays a weekly seasonality, and the volume isn't constant (it's growing). See image : http://www.blabala.com (I don't have enough rep to post multiple images)
Therefore, from what I read, ARIMA seems to be a relevant algorithm.

I stumbled upon the tsoutlier package in R, and its functions locate.outliers.oloop() and discard.outliers(), so I had the following idea: instead of trying to train something I don't master, let's have the automatic parts of the algorithm train on their own, and use that: if it detects outliers, then throw an alert.

First question: does it seem to be a good or a bad idea?

If it's a good approach :

I'm having trouble getting something that works well. I've explored two ways of doing things :

1) train the whole thing, and run it on itself.

cval <- 3
mo2 <- locate.outliers.oloop(TS_train,
                             arima_train,
                             maxit.iloop = 100,
                             maxit.oloop = 100,
                             cval = cval)
d2<-discard.outliers(mo2,TS_train, method = "bottom-up",
                     tsmethod.call = arima_train$call,
                     cval = cval)

OTL <-  TS_train[sort(d2$outliers$ind)]

plot(1:length(TS_train), TS_train, type = 'p')
lines(1:length(TS_train), TS_train, col = "grey")
points(sort(d2$outliers$ind), OTL, col = "red")

This hasn't given very good results. See image 1.

2) have a sliding window: train the algorithm with day 1 to 60 as training data, evaluate day 61, slide one day, train the algorithm with day 2 to 61 as training data, evaluate day 62, slide one day, and so on.

plage <- 60
cval <- 3
ylim <-  range(DT$col1) + c(-1,1) * .05 * diff(range(DT$col1))

for( i in (plage + 1):65 ) {
  TS_tmp <- ts(DT[(i-plage):i]$col1, frequency = 7)
  aa <- auto.arima(TS_tmp)
  mo_tmp <- locate.outliers.oloop(TS_tmp, aa,
                                  maxit.iloop = 100, maxit.oloop = 100,
                                  cval = cval)
  d_tmp <- discard.outliers(mo_tmp, TS_tmp, method = "bottom-up",
                            tsmethod.call = aa$call, cval = cval)

  Outl <- TS_tmp[sort(d_tmp$outliers$ind)]
  print(plot(DT[(i-plage):i]$col1, type = "b", xlim = c(1, nrow(DT)), ylim = ylim))
  print(points(sort(d_tmp$outliers$ind), Outl , col = "red"))
}

It seems to perform worse... (See image 2)

Then again, I don't know much about time series and don't really know how to evaluate how well a model performs.

My questions :

Are any of the two approaches a good idea ? Which one is best ? Any better idea ?
I'm using the 9 months of data I have in approach n1. In approach n2 (if it's relevant), how much data should be included in my training set ? (currently : 60 days)
Any advice on how to best calibrate the model(s) ?


texts: How to determine the inputshape of a ANN in Keras

texts:
I started to use Keras for ANN and something that I do not really understand is which values to choose for the input_shape parameter of the first layer in an ANN? I know that the number should be equal to the inputs but how can I determine the order and the other value in the vector. For example, here is the code of a Multilayer Perceptron that takes 3 inputs and calculates 1 output
model.add(keras.layers.Flatten(input_shape=[3,])),
model.add(Dense(20,  activation='relu'))
model.add(Dense(40,  activation='relu'))
model.add(Dense(20,  activation='relu'))
model.add(Dense(3, activation='linear'))

In this case I use 'input_shape=[3,]'. The 3 comes from the 3 inputs but why do I have a ",]" as the second argument. Here you see on the other hand the code for a recurrent neural network that is used for time series forecasting and that has 6 input features for every timeslot of the time series and calculates 24 outputs:
model6 = keras.models.Sequential([
    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 6]),
    keras.layers.SimpleRNN(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(24))
])

Why do I have here another order with "input_shape=[None, 6]" and not "input_shape=[6, None]" and why do I need here "None" as the first argument and not like in the multilayer perceptron just ",]". Has this something to do with the recurrent neural network such that I always have to use the number of input features as the second argument and 'None' as the first argument.
I read about this in the keras documentation but it is still confusing for me. Can you tell me how to choose those input_shape argument`I'd appreciate every comment.


texts: Multivariate Time series analysis: When is a CNN vs. LSTM appropriate?

texts:
I have multiple features in a time series and want to predict the values of the same features for the next time step. I have already trained an LSTM which is working okay, but takes a bit long to train.
So now my question: is it reasonable to use a CNN instead of an LSTM, even though it is a time series? Are there any indicators for when you should never switch to a CNN?


texts: Gaussian Mixture Classification Implementation with multidimensional trainning data

texts:
I'm trying to implement the gaussian mixture classification (GMC) implementation from scratch using python. The training dataset consists of 10 folds each of size $\left[100x64\right]$. In addition, each fold has it own train label of size $\left[100x1\right]$ with classes $C=\left\{ 5,6\right\} $.
I'm stuck on the parameter estimations $\mu_{i}$, which according to the book "Bishop - Pattern Recognition And Machine Learning - Springer 2006", should be $\mu_{1},\mu_{2}$ but I have been told that both should be a vector of $\left[1x64\right]$
I first need to estimate optimal parameters before maximizing likelihood but I don't know-how.


texts: How to inform tree based methods about sequence of data

texts: I am using tree-based methods for predictions, my series are such that they are usually high but then they slowly decrease to a lower level, I am giving the month and exogenous things as feature, but how would machine learning differentiate between pattern after some year, for example, pattern cj=hange occur after 2019. giving year as a feature in tree-based models runs into a problem also, when I predict new year, tree-based model would not have been trained on that new year so the problem would still persist.


texts: I was looking at the original implementation of LeNet-5 and I noticed a disparity in different sources. Wikipedia suggests that the non linearity used is the same sigmoid in each layer, some blog posts use a combination of Tanh and sigmoid while Andrew NG said it used some crude non linearity which no one uses today without naming it. I looked at the original paper but it's like 50 pages long and the diagram does not mention the activation functions used explicitly. I searched a bit and the sigmoid function was there and mentioned in context of activations while the tanh function is taken as a squashing function. I'm not sure if that is the same or different as then it used other terms when referring to the sigmoid ones. Anyone knows what's up with this?


texts: Time Series Data Multi-Class Classification

texts:
This is a very general question, as I'm still very much in the learning phase with machine learning.  I have some utility data around problematic meters.  Even tho the data is "time series", I believe I can perform a multi-class classification (looking at 3 labels) towards the data, but would like some opinions before I pursue down that road.

I have been doing some feature engineering to derive other data points to help with the classification process (examples below are columns "Error1" and "Error2").

The meters come in 2 classes, those that are estimated issues ="1", and does that are non-estimated issues ="0".

My dataset roughly looks like below (I have several other Error features):

 Estimated     Meter ID          Date             DaysInDuration    Error1  Error2
     0            BBA         11/19/2019               31              0       0
     0            BBA         12/19/2019               62              1       0
     0            BBA         12/19/2019               92              1       0
     1            JJL         11/2/2019               120              1       0
     1            JJL         12/2/20019              150              1       1
     1            JJL         1/20/2020               180              2       2

What I would like to attempt is to use a classification model that can handle multi-class classification (possibly a decision tree), and produce a output such as below:

 Estimated     Meter ID          Date             DaysInDuration    Error1  Error2   Classification Label
     0            BBA         11/19/2019               31              0       0            1
     0            BBA         12/19/2019               62              1       0            1
     0            BBA         12/19/2019               92              1       0            2
     1            BBA         11/2/2019               120              1       0            3
     1            JJL         12/2/2020                30              1       1            1
     1            JJL         1/20/2020                60              2       2            1

Labels Meaning = "1" = low risk issue/ "2" = medium risk issue/ "3" = high risk issue

The model would classify the either "1","2", or "3" depending on the length of days the meter has been in the "DaysInDuration" column, and the number of counted errors in the "Error1" and "Error2" columns.

In my thoughts it feels like classification would still work, including with train test splits, as the classification is moreso from other data points versus the actual order dependency in a typical time series problem.


texts: How to store a trained Random Forest model in python or matlab as a matrix

texts:
Is it possible to store a random forest model in python (or matlab). Then use that trained model in a C program?

I am trying to do this because I am making a myoelectric prosthetic, and you can only write to the chip we are using (PIC32MK0512MCF064) with C code.

My first thought on how to attempt this was to store the model as a matrix but I dont know if that is even possible?

Any clues? anything that could help would be greatly appreciated


texts: Gumbel-Softmax trick vs Softmax with temperature

texts:
From what I understand, the Gumbel-Softmax trick is a technique that enables us to sample discrete random variables, in a way that is differentiable (and therefore suited for end-to-end deep learning).

Many papers and articles describe it as a way of selecting instances in the input (i.e. 'pointers') without using the non-differentiable argmax-function. The thing that confuses me is that this effect can be achieved without randomness by just using Softmax with temperature:

Softmax with temperature
$$y_=\frac{exp(\frac{_}{\tau})}{\sum_{}exp(\frac{_}{\tau})}$$

Gumbel-Softmax
$$y_=\frac{exp(\frac{log(\pi_)+g_i}{\tau})}{\sum_{}exp(\frac{log(\pi_j)+g_j}{\tau})}$$

My question

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature?

A couple of observations:

When the temperature is low, both Softmax with temperature and the Gumbel-Softmax functions will approximate a one-hot vector. However, before convergence, the Gumbel-Softmax may more suddenly 'change' its decision because of the noise.
When the temperature is higher, the Gumbel noise will get a larger significance and the distribution will become more uniform. Why is this desired?

My best guess is that the introduction of the Gumbel noise enforces stronger exploration before convergence, but I can't recall reading any papers that use this as a motivation to bring in the extra randomness.

Does anyone have any experience or insights on this? Maybe I've completely missed the key point of Gumbel-Softmax :)


texts:
I have been using this library for basic neural network construction and analysis.

However, it does not have support for building multi-layered neural networks, etc.

So, I would like to know of any nice libraries for doing advanced neural networks and Deep Learning in Julia.


texts: Is color information only extracted in the first input layer of a convolutional neural network?

texts:
In a convolutional neural network (CNN), since the RGB values get multiplied in the first convolutional layer, does this mean that color is essentially only extracted in the very first layer?

Snippets from Stanford CS231n Chapter on CNN:

  [...] One dangerous pitfall that can be easily noticed with this
  visualization is that some activation maps may be all zero for many
  different inputs, which can indicate dead filters, and can be a
  symptom of high learning rates [...] Typical-looking activations on
  the first CONV layer (left), and the 5th CONV layer (right) of a
  trained AlexNet looking at a picture of a cat. Every box shows an
  activation map corresponding to some filter. Notice that the
  activations are sparse (most values are zero, in this visualization
  shown in black) and mostly local.


texts: Regression model to predict probability of rare event

texts:
I have a dataset with around 900.000 records, around 1000 of which are marked as positive (the studied event occurred).

The probability of the event occurring is always low (i.e. < 0.1), and I would like to create a regression model to predict the probability of the event occurring.

My first thought was to use logistic regression, but I am not sure if I could directly interpret the output as the probability of the event happening. The same doubt arises when using other models, such as SVM or RF.

Another doubt would be whether usual evaluation metrics (e.g. RMSE) would work well on such a model, since even a predictor that always outputs 0 would have a very good score.


texts: Keras/TF: Making sure image training data shape is accurate for Time Distributed CNN+LSTM

texts:
The comprehensible data shape to me is like:
(9186, 120, 120, 1)
this means 9186 entry, of 120 by 120 pixel grey images. I learnt that using Time Distributed to design a CNN combined with an LSTM model, could learn more from images, knowing they are sequenced.

Following some tutorials, I found out I should expend another dimension which describes moving frames, in my case, I have 6 hours image, this would be good as a moving sequence, (probably I should try other lengths because time-series images are not of regular lengths, example:

8:00 img1 target1
9:00 img2 target2
10:00 img3 target3
11:00 img4 target4
12:00 img5 target5
13:00 img6 target6
19:00 img7 target7
   ...

My question is, when I expend another dimension of lets say length 6, how could such a model here output for every entrie exactly one prediction, but using a frame of length 6 to learn the next ?

train_images.reshape((1531, 6, 120, 120, 1)).shape

=> This gives me the impression, that such a model would output a prediction for a sequence of 6, which means 1531 result.

Am I understanding it wrong ?

[Edit following wind's answer]

I think my problem persists but I was not aware, I declared the first layer as

TimeDistributed(
        Conv2D(64, (3, 3), activation='relu'),
        input_shape=(6, 120, 120, 1)
    )

for x_train as

train_images_ = train_images.reshape((1531, 6, 120, 120, 1))

while y_train declared as

y_train_ = y_train.reshape((1531, 6, 8, 1))

as you can see x_train and y_train have the same length, with a frame of 6, image_length of 120, while, y_train consists of 8 target columns.

To my understanding, this is the way to follow with tensorflow, unfortuanatly, this is the error I am getting...

  ValueError: Error when checking target: expected dense_2 to have 2
  dimensions, but got array with shape (1531, 6, 8, 1)


texts: Using RNN (LSTM) for predicting one future value of a time series

texts:
I have been reading several papers, articles and blog posts about RNNs (LSTM specifically) and how we can use them to do time series prediction. In almost all examples and codes I have found, the problem is defined as finding the next x values of a time series based on previous data. What I am trying to solve is the following:

Assuming We have t values of a time series, what would be its value at time t+1?

So using different LSTM packages (deeplearning4j, keras, ...) that are out there, here is what I am doing right now:

Create a LSTM network and fit it to t samples. My network has one input and one output. So as for input I will have the following patterns and I call them train data:

  t_1,t_2

  t_2,t_3

  t_3,t_4

The next step is to use for example t_4 as input and expect t_5 as output then use t_5 as input and expect t_6 as output and so on.
When done with prediction, I use t_5,t_6 to update my model.

My question: Is this the correct way of doing it? If yes, then I have no idea what does batch_size mean and why it is useful.

Note: An alternative that comes to my mind is something similar to examples which generate a sequence of characters, one character at a time. In that case, batch_size would be a series of numbers and I am expecting the next series with the same size and the one value that I'm looking for would be the last number in that series. I am not sure which of the above mentioned approaches are correct and would really appreciate any help in this regard.

Thanks


texts:
I'm conjecturing that with Complete-linkage clustering two elements from the same cluster will always be closer to each other some other element from another cluster.

In more formal terms:

Let $C$ be a clustering.
 $\not\exists z \in C_j$ s.t. $\bigtriangleup(x, z) < \bigtriangleup(x, y)$ where $x,y \in C_i$, $C_i \neq C_j$ and $C_i, C_j \in C$.

I haven't been able to prove the conjecture yet, thus I'm wondering whether I'm right or wrong. If this is indeed the case, I would much appreciate a sketch a proof. I'm pretty sure I can work my way from there.

On a side note (not that I think it makes a difference), I'll be applying the clustering algorithm on a one-dimensinal dataset.

Your input is much appreciated.


texts: ValueError: Error when checking input: expected dense_7_input to have shape (11,) but got array with shape (469,)

texts:
Running my model produces the following error:

> ### ValueError: Error when checking input: expected dense_24_input to have shape (11,) but got array with shape (470,)

My Code:

# Initalizign the ANN
classifier = Sequential()

# Adding the input layer and the first hidden layer
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))

# Adding the second hidden layer
classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))

# Adding the output layer
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))

#Compiling the ANN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

### fitting the classifier to the training set
classifier.fit(X_train, Y_train, batch_size = 10, epochs = 100)

In the last step, I am getting the error. I am new in Data Science & Neural Networks. Any suggestions?


texts:
I am pursuing a course on Edx on data analysis for Social Sciences. Actually I am stuck on this assertion of the teacher Mrs. E. Duflo which states:

Correlation is not causality.
A causal story is not causality either

While she explained this concepts quite well, I feel like I am not understanding this second sentence quite well. What is a causal story in fact?


texts: Building a multiclass classifier that can handle classes it has never seen?

texts:
I am given a dataset that has free-form text and a category associated with it. There are 100 different categories and 3000 records for each category. The goal is build a multiclass classification model. I have created a simple neural network with 10,000 input features/words and the results were fairly good (~88%).
The issue I am facing is with an unlabeled dataset that I have, that is missing the category label. This dataset is very large, and has much more than 100 categories. I am only interested in being able to classify unlabeled data for the 100 categories that I have, but I am not sure how to approach this.
One thought I had was to build a Word Embedding model for the labeled data. This model could be used to calculate a document vector for the unlabeled data and find a similar document from the labeled dataset. This would allow me to assign labels to some of the data in the unlabeled dataset. Is there a better way to approaching this problem?


texts: Stanford NER - Increase probability for a certain class

texts: I'm new to machine learning so I apologize if this question is silly. I'm using Stanford NER's english 4class classifier with good results. However, since my dataset is mostly focused on organizations, I think the results could be improved if I could boost the probability for an entity to be an organization to the detriment of other classes.(Ex: I would prefer "Carl Zeis" to be identified as an organization rather than a person). Is my supposition correct? If so, can it be achieved in an easier way than retraining the model?


texts:
I've model with two output layers, age and gender prediction layers. I want to assign different  weight values for each output layer's loss. I've the following line of code to do so.

model.compile(loss=[losses.mean_squared_error,losses.categorical_crossentropy], optimizer='sgd',loss_weights=[1,10])

My question is what is the effect of loss weights on performance of a model? How can I configure the loss weights so that the model can perform better on age prediction?


texts:
I'm looking for references on training time-series regression RNN models. For learning purposes I want to implement myself using autograd (or JAX) rather than a high level library. I cannot find really good references on how to train this class of model.

Specifically I have 1 years data at 15 minute resolution. The model I'm fitting is

x[t+1] = f(x[t],u[t])
y[t] = g(x[t])

where x is the latent state, u the input and y the output at time t.

Assuming no gaps in the data, I need to learn the parameters of f and g, and optionally the initial state (or provide a guess). One way of doing this is to compute derivatives and the loss on the entire sequence from t0 to T. In this case I can either initialise x[0]=0 and hope it's influence decays to zero or learn it.

Another way is to batch into say daily subsequences. This seems to be the more common method (since it aligns with stochastic gradient descent), but how then do you deal with the initial state for each subsequence? One way is to set x[0]=0, update the derivatives using the first subsequence, predict the subsequence and roll the final state from subsquence i to the initial state for subsequences i+1? Another is to attempt to learn the average initial state for each subsequence - it feels like this might work but only if the subsequences are long enough that the error in the intitial state estimation decays relatively quickly relative to the length of the subsequence.

I've not really found any really good papers or even blogs on this though, most blogs are aimed at showing how to implement an RNN using a high level api and don't cover the low level details or are talking about independent sequences such as sentences rather than a time-series.


texts: Are ontologies and the Semantic Web dead?

texts:
Is the Semantic Web dead? Are ontologies dead?

I am developing a work plan for my thesis about "A knowledge base through a set ontology for interest groups around wetlands". I have been researching and developing ontologies for it but I am still unclear about many things. What is the modeling language for ontologies?

Which methodology for ontologies is better? OTK or METHONTOLOGY?

Is there any program that does  as does

Cratilo is a software for analyzing of textual corpora and for extraction of specific terms of the domain of study (it is developed by professors Jorge Antonio Mejia, Francisco Javier Alvarez and John Albeiro Sánchez, Institute of Philosophy the University of Antioquia). It enables lexical analysis of texts, identifying the words that appear their frequency and location in the text. Through a process of recognition, Cratylus identifies all the words in the text and builds a database becomes the draft analysis of the work. Are there other similar tools?

Can the terms found by Cratilo be used to create a knowledge base?

What are the existing open semantic frameworks that can be used for such things?

Is there software that automatically creates RDF, OWL, and XML? How does Tails work? Jena? Sesame?


texts: What's the Seminal Book on data science?

texts: Can people provide the best textbook they have used for learning data science?  I know this is somewhat difficult because you have no basis of comparison but if you can just give your opinion on a book and why, I'd greatly appreciate it.


texts:
I am learning about SVMs in particular linear SVMs through many questions here. However, one problem i faced is that there seems to be no indepth explanation on how does linear SVM works in terms of predicting new data.

I understand that the main purpose of SVM is to find linear separating hyperplane $w^Tx+b$ and a linear SVM is actually a set of super long equation.
Let's consider a 2 class problem : A and B. Suppose $(w^*,b^*)$ are the minimizing hyperplane parameters for a fixed choice of $\lambda$.

Then how we classify a new, unlabeled test point $x_{test}$? Simple way that I thought would be reasonable is to have $(w^*)^Tx+b^*>0$ to class A and $(w^*)^Tx+b^*<0$ to class B. But how do we assign if it's 0 and what if there is an outlier in other class for example? IS this a good way of labeling test data?


texts:
I was reading an article about convolutional neural networks, and I found something that I don' understand, which is:

  The filter must have the same number of channels as the input image so that the element-wise multiplication can take place.

now, what I don't understand is : what is a chennel in a convolutional neural network? I have tried looking for the answer, but can't understand what is it yet.

Can someone explain it to me?

Thanks in advance.


texts: how to I replace numeric values with a string in an R dataframe?

texts:
I want to replace all numeric values in a column in my data frame with a string value. The following doesn't seem to work.

df <- within(df, myCol[is.numeric(myCol)] <- 'NOTMISSING')

Even though the df has some values as NA and others as numbers, all values are being replaced with NOTMISSING.

Also tried

df <- within(df, myCol[is_numeric(myCol)] <- 'NOTMISSING')

Any pointers highly appreciated.


texts: How to make an index on a large scale dynamic dataset in a search engine?

texts:
I want to create a textual search engine in which the dataset is not static. There will be additions of new documents in the dataset every day. I understood that in order to create a search engine, following steps should be taken:-

Preprocessing the documents
Converting them into embeddings or word vectors
Creating Index with the embeddings
Performing Search

I am using BERT for embedding creation and FAISS IndexIVFPQ for creating index and performing search.
My dataset is dynamic that means I need to create index again and again whenever any new document is added. But as the dataset increases , index building takes a lot of time.
So my query is how should i approach this large dataset issue ?
Do i need to make multiple indexes instead of a single index ?
If i divide the dataset into batches and build index separately on each of them, then I have to run the same query parallely on those indexes, then how would I know which index has returned the relevant result and which has not , because each index will return top most k documents ?
If I see the cosine score of the documents between the multiple indexes results and rank them in ascending order to display, would it compromise some accuracy ?
I read multiple articles and tried to search the solution, I got the following statement

If the dataset changes significantly over time, then rebuilding
the index by reclustering would be required. For large enough
datasets, one would probably have multiple shards of data that are
assumed to be reasonably independent. Individual shards can be
periodically reindexed to replace the old shard, but this demands that
the original vector data (e.g., 128-dim vectors * sizeof(float) = 512
bytes per vector) is kept around rather than the compressed form (4 -
64 bytes per vector).

What does the above lines mean ?


texts: What are the most concrete and easiest to understand applications of deep learning in the industry?

texts:
I am a researcher in deep learning, and have given courses on the theory. Recently I was approached to give the same course adding sections using some very concrete examples of deep learning for companies.

My audience will be industry managers never exposed to deep learning before. What are the most easily performed examples I could provide without having to develop millions of lines of code that can be of general interest to managers never exposed to deep learning before? These examples need to be quite concrete and easily applicable.

In particular, they look for examples that can have the label "big data" slapped on them.

I know about using ML for detecting digits, or all the other applications on kaggle, but they do not seem to care for those.


texts:
I am new to tensorflow and I am trying to predict tree species from a bunch of point data describing imagery in different ways. I have created a TensorFlow model but I think I am creating my y-data incorrectly. I am using a function to change string values to numeric values between 0 and 2. The resultant prediction that I am getting back from the model is all of value 2. Could this have something to do with my y-data not being one-hot encoded? How do I get my y-data, which is string data, into this model properly?

import tensorflow as tf
import pandas as pd
import numpy as np
import sys
from sklearn.model_selection import train_test_split

tree_data_file = r'\\David\Processing\p839\result_testing\first_test_feature_cols_v2.csv'

tree_data = pd.read_csv(tree_data_file)

#Make tensorflow less verbose
tf.logging.set_verbosity(tf.logging.ERROR)

def label_fix(lable):
    if lable == 'AW':
        return 0
    elif lable == 'PB':
        return 1
    elif lable == 'SW':
        return 2

### Normalize numeric data
columns_to_normalize = ['band3_sum', 'image3b3_s', 'image_red', 'image_blue',
                        'image_gree', 'image_nir', 'image_aver', 'cohort', 'sph_total',
                        'sph_under', 'sph_over', 'crown_area', 'ci_crown_a',
                        'ndvi']
tree_data[columns_to_normalize] = tree_data[columns_to_normalize].apply( lambda x: (x - x.min())/(x.max()-x.min()))

### Create feature columns for continuous data
band3_sum = tf.feature_column.numeric_column('band3_sum')
image3b3_s = tf.feature_column.numeric_column('image3b3_s')
image_red = tf.feature_column.numeric_column('image_red')
image_blue = tf.feature_column.numeric_column('image_blue')
image_gree = tf.feature_column.numeric_column('image_gree')
image_nir = tf.feature_column.numeric_column('image_nir')
image_aver = tf.feature_column.numeric_column('image_aver')
crown_area = tf.feature_column.numeric_column('crown_area')
ci_crown_a = tf.feature_column.numeric_column('ci_crown_a')
ndvi = tf.feature_column.numeric_column('ndvi')

tree_data['sp_call'] = tree_data['sp_call'].apply(label_fix)

feat_cols = [band3_sum,image3b3_s,image_red,image_blue,image_gree,image_nir,image_aver, crown_area,ci_crown_a, ndvi]

#TRAIN TEST SPLIT
labels = tree_data['sp_call']
x_data = tree_data.drop('sp_call', axis=1)
X_train, X_test, Y_train, Y_test = train_test_split(x_data, labels, test_size = 0.3)#, random_state=128)

input_func = tf.estimator.inputs.pandas_input_fn(x = X_train, y=Y_train, batch_size=100, num_epochs=1000, shuffle=True)
model = tf.estimator.DNNClassifier(feature_columns=feat_cols,hidden_units=[1024,512,256,32,3],n_classes=3, activation_fn=tf.nn.relu,optimizer='Adam',dropout=0.2)
model.train(input_fn=input_func, steps=20000)
results = model.evaluate(input_func)
print (results)
eval_input_func = tf.estimator.inputs.pandas_input_fn(x=X_test, y=Y_test, batch_size=128, num_epochs=1, shuffle=False)
results = model.evaluate(eval_input_func)
print (results)

pred_input_func = tf.estimator.inputs.pandas_input_fn(x=X_test, batch_size=100,num_epochs=1,shuffle=False)
predictions = model.predict(pred_input_func)
my_pred = list(predictions)

final_preds = []
for pred in my_pred:
    final_preds.append(pred['class_ids'])
print (final_preds)

UPDATE 1:
I have tried to implement the tf.one_hot as mentioned in the comments. I have changed

labels = tf.one_hot(tree_data['sp_call'],3)

This results in an error on:

X_train, X_test, Y_train, Y_test = train_test_split(x_data, labels, test_size = 0.95)#, random_state=128)

stating:

object of type 'Tensor' has no len()


texts:
I am currently analyzing investment strategies, and have implemented a backtest accordingly. This essentially means that I predict returns each month by using all prior historical data. Consequently, when I predict the last period, the sample will be much bigger than for the initial sample.

I am also using a validation sample which I use to determine the shrinkage parameter via a grid search. Now I noticed that as the sample increases, more coefficients are set to zero. Is there a reason for this? I would expect with more data the coefficients can be estimated more accurately and hence do not need to be set to zero?

Also when examining the added penalty term for Lasso, I can't come up with a good reason.


texts: Tool vs Python Script for Transforming Data in Mongo

texts: We have a bunch of Mongo collections (data collected from APIs, web scraping, etc) that we need to transform to a cleaner data structure (standardized schema) on a monthly basis. Are there any good tools to help us manage this process, or would you recommend writing a Python script instead?


texts:
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there is a consistent amount of white noise in the background.

I have manually decoded these streams, but I was wondering if I could use some sort of machine learning technique to learn the encoding schemes. This would save a great deal of time manually recognizing these schemes. The difficulty is that different signals are encoded differently.

Is it possible to build a model which can learn to decode more than one encoding scheme? How robust would such a model be, and what sort of techniques would I want to employ? Independent Component Analysis (ICA) seems like could be useful for isolating the frequency I care about, but how would I learn the encoding scheme?


texts: I have a dataset of 116 observations and 10 numeric variables. The dataset contains information about healthy patients and patients attained with breast cancer. I did a PCA plot showing the cluster of healthy/sick patients to see the relation they had with the variables but to my surprise the "N/A (breast cancer patient) cluster" is very far away from any variable. What could that possibly mean ? I'm fairly new to PCA or how to read the plots so any help would be appreciated !


texts: Visualize n-dimensional bayesian optimization results

texts:
I am working on a 6-dimensional bayesian optimization problem using (skopt's gp_minimize).
After the optimizer ran for j iterations I would like to somehow visualize the "progress/result" of the optimization. As I am new to Bayesian optimization I would like to ask for input on how and what to visualize. What are good parameters to visualize to show the improvement and maybe even the parameter dependency of the optimized parameters?


texts: Low number of inputs compared to outputs (per row) in neural network

texts:
I have data that one row has input of size 1x2 (two values) and output is matrix of size 15x3 (fifteen rows with three values) like that

            |-----------------|
            |y01,2 y01,2 y01,3|
            |y02,1 y02,2 y02,3|
            |.                |
            |.                |
            |.                |
            |.                |
            |.                |
-------     |.                |
|x1,x2|     |.                |
-------     |.                |
            |.                |
            |.                |
            |.                |
            |.                |
            |y15,1 y15,2 y15,3|
            |-----------------|

This is one row and I have around 3000 inputs like that.
Most values are real numbers except one output column that is binary (So output y$_{n,3}$ values are 0 or 1).

How should I preprocess this kind of data?
Is it of any sense to train two models, one for real numbers only and the second one for binary output?
What neural network architecture would work best for this task?


texts: Find the longest palindrome substring of a piece of DNA

texts:
I have to make a function that prints the longest palindrome substring of a piece of DNA. I already wrote a function that checks whether a piece of DNA is a palindrome itself. See the function below.

def make_complement_strand(DNA):
    complement=[]
    rules_for_complement={"A":"T","T":"A","C":"G","G":"C"}
    for letter in DNA:
        complement.append(rules_for_complement[letter])
    return(complement)

def is_this_a_palindrome(DNA):
        if DNA!=(make_complement_strand(DNA)[::-1]):
            print("false")
            return False
        else:
            print("true")
            return True

is_this_a_palindrome("GGGCCC")

But now: how to make a function printing the longest palindrome substring of a DNA string?

The meaning of palindrome in the context of genetics is slightly different from the definition used for words and sentences. Since a double helix is formed by two paired strands of nucleotides that run in opposite directions in the 5’- to-3’ sense, and the nucleotides always pair in the same way (Adenine (A) with Thymine (T) for DNA, with Uracil (U) for RNA; Cytosine (C) with Guanine (G)), a (single-stranded) nucleotide sequence is said to be a palindrome if it is equal to its reverse complement. For example, the DNA sequence ACCTAGGT is palindromic because its nucleotide-by-nucleotide complement is TGGATCCA, and reversing the order of the nucleotides in the complement gives the original sequence.


texts: Good performance metrics for multiclass classification problem besides accuracy?

texts:
I am trying to solve a multiclass classification problem. The dataset is balanced. I have been using accuracy as a performace metric till now. Are there any other good performance metrics for this task?

I already know about precision and recall but as far as I know they are used when the dataset is imbalanced.


texts: Input and output layer activation functions of neurons in Orange

texts: What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?


texts: Modeling the Price Movement- What analysis should be used

texts:
I am trying to model the price of a hotel as the check-in date arrives. I have a data set which looks like-

For e.g- if I am looking at the booking date of Dec 31st, I would want to analyze the movement of price for the hotels from 31st October onwards and model this movement with given variables-City, Star_rating, Accomodation_type, Chain_hotel, and Time before check-in.

I started with checking which of the variable(Feature) was more important in deciding the price of the hotel on a given day and applied multiway ANOVA. However, I am still not confident on my analysis and I am really confused that what could be the best way to model such a problem.

Thanks in advance for your help.


texts: Speed up Keras Model Prediction Load Times

texts:
I am trying to create a prediction API using keras which loads the model predicts and closes the model. But initializing time in python is about 3-5 secs so each request takes around 5 secs to return the prediction irrespective of number inputs rows(predictions)
Is there any way to keep the model loaded and then stream the input data to get prediction. Like a Pre loaded model either through a socket or a port.
Similar to open office document converter
\program\soffice.exe -accept="socket,host=127.0.0.1,port=8100;urp;" -headless -nofirststartwizard -nologo

Keras Prediction Code
#!/usr/bin/env python3.6
import sys
import pandas as pd
from keras.models import load_model
model = load_model('model.h5')
X = pd.read_csv(sys.argv[1]).values
prediction = model.predict(X)
pd.DataFrame(prediction).to_json(sys.argv[2])

Script is called as
python3.6 predict.py input_scaled.csv output_scaled.json

The prediction time are as follows
#row    time
1       4.76 secs
10      4.49 secs
50      5.37 secs
5000    5.46 secs
50000   12.7 secs


texts: How should I construct a binary classifier for small set of positive data and million of unlabeled data?

texts: Does anyone have suggestions for specific algorithm or implementation for labeled data of only one class and unlabeled data that can be from either classes? And I'm unsure what is the proportion of Class A to B that exists within the unlabeled data and also my labeled data is not randomly chosen.


texts: Deep Learning: Does starting the training on a smaller subset of the data make sense?

texts:
I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the entire data set (10X more data) and it improved further. When I give it just the entire data set the model seems to perform terribly, but also it was difficult to reach many epochs due to the extra time needed for training.

My question is: does it make sense to "warm up" the training a large deep neural network with a smaller subset of the data, or should one always just provide the full data?

To add a bit more detail: I had a heavy class imbalance. The model always predicts 0 when I give it the full data set, but when I overfit on a smaller training set, it learns at least to not always predict 0. I notice this guide in step 5, suggests something similar to what I did, but I am not sure if it is theoretically sound or just a nice way of checking the model "can work given enough epochs".


texts:
I'm currently studying from Andrew Ng's Stanford handouts here (I'm at part 8). Now from what I gathered from before, all the time our goal was to minimize ||w||^2 so that we can maximize the margin. However, now as he's writing about the regularization, he's saying:

  The parameter C controls the relative weighting between the twin goals of making the ||w||^2 large (which we saw earlier makes the margin small) and of ensuring that most examples have functional margin at least 1.

Why would making the margin small be currently a "goal"?


texts: How to interpret sudden jumps of improvement in training error vs. iteration?

texts:
In the Residual learning paper by He et al., there are a number of plots of training/test error vs. backprop iteration. I've only ever seen "smooth" curves on these plots, while in this paper's application, there are sudden jumps in improvement. In this figure (Fig. 4a in the paper linked above), they are at around 15e4 and 30e4 iterations:

What happened here? My intuition would say that the backpropagation hit a plateau with a gradient of close to zero, and then very suddenly found a path steeply downward - but is that a realistic shape for a cost function?


texts: What exactly is bootstrapping in reinforcement learning?

texts:
Apparently, in reinforcement learning, temporal-difference (TD) method is a bootstrapping method. On the other hand, Monte Carlo methods are not bootstrapping methods.

What exactly is bootstrapping in RL? What is a bootstrapping method in RL?


texts: Tensorflow training with batch size of (1, None, features), but model expects extra dimension

texts:
I've made an autoencoder like below, to accept variable-length inputs. It works for a single sample if I do model.fit(np.expand_dims(x, axis = 0) but this won't work when passing in an entire dataset. What's the simplest approach in this case?

import numpy as np
import tensorflow.python.keras.backend as K
from tensorflow.python.keras.layers import Input, LSTM, Lambda
from tensorflow.python.keras.models import Model

def repeat(x):
    step_matrix = K.ones_like(x[0][:, :, :1])
    latent_matrix = K.expand_dims(x[1], axis = 1)
    return K.batch_dot(step_matrix, latent_matrix)

timesteps = None
features = 2
latent_dim = 10

inputs = Input(shape = (timesteps, features))
encoded = LSTM(latent_dim, name = "encoded")(inputs)
decoded = Lambda(repeat)([inputs, encoded])
outputs = LSTM(features, return_sequences = True)(decoded)
autoenc = Model(inputs = inputs, outputs = outputs)
autoenc.compile(optimizer = "adam", loss = "mse")
encoder = Model(
    inputs = autoenc.input, outputs = autoenc.get_layer("encoded").output
)

x1 = np.ones((20, 2))
x2 = np.ones((30, 2))
x3 = np.ones((40, 2))
X_train = np.array((x1, x2, x3))

autoenc.fit(x = X_train, y = X_train, epochs = 10, batch_size = 1)


texts: Recommendation model that can recommend already bought item

texts:
Most recommendation algorithms recommend new products to users.

  If you bought this you might like that

But sometimes the item user is most likely to buy is an item that he bought sometime ago.

Is there any algorithm appropriate for this use?


texts: Strings/ features in Turicreate decision tree

texts: I am trying to create a prediction model by using a decision tree with Turicreate. While my problem does involve numbers, it also involves strings and ultimately I want it to return the string 'true/false'. Are Turicreate decision trees able to process strings as input and output?


texts: How do I combine predictions from classifiers for two different problem?

texts:
I am working on a classification problem for predicting whether the shipment is going to be late or not.
I would say the classifier is mediocre at predicting the positive class at the moment. But the ambition is to improve it.
However, after doing some analysis, I have found out that there is an important component (Customs) that has appeared as the cause of shipment delay in majority of FN.
Currently, I don't have a feature that would directly be associated to customs and be used in the model. Moreover, I think because of the product we are shipping, custom process may vary.
My original problem was at a shipment level, because of which I had to exclude the products in the shipment. But, now I want to include the products. It is a  many to many relation -> Shipment can have multiple products and Vice a versa.
Following is my thought:
To have a separate predictor in addition to the original one that would predict if the product in the shipment is going to be late/not based on planned days for customs.
This is where I am struggling, if this is a right approach how do I consolidate the predictions for both the models to come up with a single prediction as Late or No Late?
In addition to this I need to understand if there is another way to tackle this?


texts: Preprocessing of 3D CAD files for Keras Conv3D input

texts:
I'd like to apply some machine learning on 3D CAD data. File format should ideally be mesh-based like STL. Keras offers 3D convolutional layers (https://keras.io/layers/convolutional/), so it can handle 3D (5D) input data. The problem I have is the preprocessing of the input data. There seem to exist integrated methods in Keras for image preprocessing (https://keras.io/preprocessing/image/). However that does only seem to apply to 2D images.

I plan to voxelize the STL files somehow, and would appreciate input on the process of voxelization and preprocessing the data to be suitable for Keras.

PS: I know there are CNNs which directly take mesh data. However I'd like to approach the voxelation way to compare performance and results.


texts: XGBRegressor vs. xgboost.train huge speed difference?

texts:
If I train my model using the following code:

import xgboost as xg
params = {'max_depth':3,
'min_child_weight':10,
'learning_rate':0.3,
'subsample':0.5,
'colsample_bytree':0.6,
'obj':'reg:linear',
'n_estimators':1000,
'eta':0.3}

features = df[feature_columns]
target = df[target_columns]
dmatrix = xg.DMatrix(features.values,
                     target.values,
                     feature_names=features.columns.values)
clf = xg.train(params, dmatrix)

it finishes in about 1 minute.

If I train my model using the Sci-Kit learn method:

import xgboost as xg
max_depth = 3
min_child_weight = 10
subsample = 0.5
colsample_bytree = 0.6
objective = 'reg:linear'
num_estimators = 1000
learning_rate = 0.3

features = df[feature_columns]
target = df[target_columns]
clf = xg.XGBRegressor(max_depth=max_depth,
                min_child_weight=min_child_weight,
                subsample=subsample,
                colsample_bytree=colsample_bytree,
                objective=objective,
                n_estimators=num_estimators,
                learning_rate=learning_rate)
clf.fit(features, target)

it takes over 30 minutes.

I would think the underlying code is nearly exactly the same (i.e. XGBRegressor calls xg.train) - what's going on here?


texts: R: I need to understand the mechanism behind reading a subset of a file in R (e.g., via sqldf) or other data-science-centered programming language

texts: By reading only a subset of the records of a file into R (or Python), one can speed up performance tremendously. I am curious about the mechanism behind this kind of file input. To read only a small part of a file, my guess is that I have to somehow access the file, load it in memory and filter out that small part that I am interested in (Otherwise how else would I be able to query the subset of the records that I want?). In otherwards that should then mean I have anyway had to read the whole file into memory. So if it was a 20GB file, it means I have had to read 20GB so as to filter off the 1GB that I want. Where do the performance benefits come from then?


texts: In R's randomForest, is predict() non-deterministic?

texts:
Training a random forest model is inherently non-deterministic (absent control over the random number generator), but is predict() also non-deterministic? That is, if I construct randomForest (with an odd number for ntree per the caveat in the doc) and save an .rda, will loading that .rda give me identical results given identical inputs?

I tried answering this myself by looking through https://github.com/cran/randomForest , and it seems that nobody along the predict() path is calling for a random number, but I'm very new to R and rather rusty with C and may be missing something. Pointers into the code or the docs will be appreciated.


texts:
I have a model in keras with a custom loss. this loss is calculated using actual and predicted labels(or values) and is also based on some input value.

(an example would be to define loss based on reward or advantage as in a policy gradient method in reinforcement learning context )

example code:
note: the input value used in loss calculation is named 'in2' and variable is input_2

input_1=keras.layers.Input(...,name="in1")
output= keras.layers.Dense(...)(input_1)

input_2=keras.layers.Input(...,name="in2")
def loss_func(y_true,y_pred):
    # some custom loss i define based on input_2
    loss = keras.layers.Dense(..)(input_2)
    return loss

my_model=keras.Model(inputs=[input_1,input_2],outputs=output)
my_model.compile(...,loss=loss_func,)

Now the problem is that when I want to just predict an output, I have to give a dummy input for the unused input_2. The input_2 is only used for computing the loss and it is not used for computing the model output. but when i use this code, an error ocures for not providing 'in2':

# this throws error for not providing input value for 'in2'
my_model.predict({'in1':some_value})

so I have to provide some dummy value for in2 so the model outputs its value, although it does not use the in2 value provided. note that providing None value does not work either. I thought this functionality is implemented inside keras.

or maybe I am doing it the wrong way? if yes, what is the appropriate way?

idea:
An idea came to my mind was that maybe we can define two models. one model for training and another for predicting:

my_model_for_training = keras.Model(inputs=[input_1,input_2],outputs=output)
my_model_for_predicting = keras.Model(inputs=[input_1],outputs=output)

and when I want to predict, I just use the second model which does not require the input_2 (or in2) as in example. both models share the same layers so when train the first model, the weights are updated for both models as explained in docs (section: Shared layers). is there a better solution?


texts: How to classify a data as not matching any label

texts:
I'm creating a supervised convnet which recognizes a phrase in a regional language.

For example:

  I have a dataset which contains 100 sounds for labels A and another 100 sounds for labels B. Now normal operation of the network is to differentiate a sound between label A and label B.

But let's say that I give a new sound to the network to classify, which is actually neither A nor B. I want the neural net to tell me that it doesn't qualify any of the labels.

How can I achieve this?


texts:
I am trying to load a Keras model and make predictions with it and run into a strange error. An minimal example is the following:

from keras import models
import numpy as np

model = models.load_model('model_4hiddenLayers_16unitsPerLayer_relu_learningRate0p0001.h5')
x = np.ones(36, dtype=float)
prediction = model.predict(x )

The model expects an input shape of (36,), which should be the shape of x, which I verified:

print('x.shape={}'.format(x.shape) )

gives :

  x.shape=(36,)

However when running this code I get the following error message:

  ValueError: Error when checking : expected batch_normalization_1_input to have shape (36,) but got array with shape (1,)

What am I missing here? Thanks for the help.

Some additional info : I am using keras version 2.1.4 with TensorFlow as backend.


texts: Encoder-Decoder Sequence-to-Sequence Model for Translations in Both Directions

texts:
Is it possible to use a pre-trained sequence to sequence encoder-decoder model which translates an input text in source language to an output in target language to do an inverse translation? That is, take an input in target language, and output a sequence in source language?

Are there architectures which can do translations (or seq to seq generation) in both directions?


texts: looking for approaches to detecting outliers in individuals unequal sequential time series

texts:
I am looking for approaches related to outlier detection in time series.

Example:
A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, stress_level) at each occasion. Usually, the stress value will be the same for most of the individuals but somehow for one person increases continuously over time. So, the idea is to find such a person who are dissimilar to all others over time.

Some of the literature which I have already searched look to deal with time series on continuous data and some of the approaches involve similarity-based where an average threshold similarity is created(DTW and DTW-Adaptive) and with the nearest neighbor approaches a most dissimilar series whose distance greater than the threshold is identified. These series are usually long and are sensor observations.

The series which I am looking at is a short time series of discrete observations and may vary in length. I am looking for approaches that are usually suitable for them.

Could the multivariate data outlier detection algorithms be suitable over time or could be adapted over time?

Additionally, are there any outlier models which take time into account? (As I could not find anything on this in the literature)

An example:

Patient Time bmi blood_pressure stress
1       t1  v1  v2              v3
2       t1  v1  v2              v3
3       t1  v1  v2              v3
4       t1  v1  v2              v3
5       t1  v1  v2              v3
1       t2  v1  v2              v3
2       t2  v1  v2              v3
4       t2  v1  v2              v3
3       t2  v1  v2              v3
5       t2  v1  v2              v3

Some of the multivariate approaches already looked at are:
1. https://pyod.readthedocs.io/en/latest/ - Only holds for static data.
2. Shokoohi-Yekta, M., Hu, B., Jin, H., Wang, J., & Keogh, E. (2017). Generalizing DTW to the multi-dimensional case requires an adaptive approach. Data mining and knowledge discovery, 31(1), 1-31. (Time-series based approach but uses sensors)

I recently also thought, if I could include time itself as a feature to model outlier? Has anyone did this way or have any literature in these lines


texts: Can we use decreasing step size to replace mini-batch in SGD?

texts: As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample in each iteration? Can we compare the convergence rate of them?


texts: Calculate number of parameters for ConvLSTM2D layer

texts:
time_distributed_24 (TimeDis (None, 16, 64, 64, 512)   0

conv_lst_m2d_2 (ConvLSTM2D)  (None, 16, 64, 64, 128)   2949632

time_distributed_25 (TimeDis (None, 16, 64, 64, 128)   512

time_distributed_26 (TimeDis (None, 16, 128, 128, 128) 0

For example, why does this ConvLSTM2D layer have 2949632 parameters?
A standard LSTM layer has 4(n*m + n^2 + n) parameters where m = input dim, n = output dim.

A standard 3x3 conv layer with 128 kernels over a 64x64x512 tensor would have 3x3x512x128 parameters

What would be the n and m for the LSTM part?


texts: Where to upload large (0.5Gb) weights anonymously?

texts:
I need to upload a number of checkpoints for ConvNets (weights + optimizers, all dicts of pytorch tensors), each about 0.5Gb anonymously. I don't want to use Google Drive. I trained models on the university cluster (if it's relevant).

Where can I load these files anonymously? The files must be publicly available, but my identity must remain anonymous.


texts: Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?

texts:
I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000

Intuitively, it feels like using all the negative samples would prevent the classifier from learning invalid patterns (that might otherwise be learned using undersampling, for example).

Am I underestimating the effect of the imbalance on classifier performance?


texts: Multiprocessing in KNIME Python Nodes possible?

texts:
I am wondering if it is possible to enable multiprocessing in KNIME Python nodes?

This is not working:

import multiprocessing as mp

from multiprocessing import Pool

def parallelize_dataframe(df, func):
   df_split = np.array_split(df, num_partitions)
   pool = Pool(num_cores)
   df = pd.concat(pool.map(func, df_split))
   pool.close()
   pool.join()
   return df

if __name__ == '__main__':
   # the preprocess function does some stuff of string manipulation,
   # and could be any other function
   output_table = parallelize_dataframe(input_table, preprocess)


texts: What should I study to find optimal value of best feature combinations in machine learning?

texts:
I would like to do production optimization with machine learning and/or optimization problem.

My goal is not to find minimizing loss in loss function only to give the best y value. My ultimate goal is to find optimal values of the best feature combination. In other words, 'I would like to find the best x values (feature values) that yields the best y (minimizing loss). For example, 30 (temp), 100 (Pressure), 50 (weight) gave the best product y. The actual best set of x feature values.

Supervised machine learning techniques do not provide the best-predicted x values to me. but I would like to find the x. Therefore, my question is as below:

What is the field for finding optimal x values for optimal y values called? Is continuous and discrete optimizations are ones finding the x values for y?
I found many studies but I am not sure if it is really about finding the x values (the optimal values of the best combination of features). Can someone please tell me what field I must study for this?
What kind of techniques should I use to find what I want?


texts: Can OLS regression be used to predict from a complete sequence of data?

texts: Reading online and following this example from scipy I understand OLS can be used to find data between gaps in a sequence (interpolate?) but I already have a complete sequence and want to predict the future of it (extrapolate, i think). Can OLS be used for that?


texts: How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

texts:
I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an organisation.
Methodology
How to bin
I thought about using quantiles : knowing that there are, say, 10 levels in an organisation (e.g. President, Director, ..., Worker) I would like to estimate the average pay for the corresponding level.
I thought to use quantiles; am looking at the doc of pandas :

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html#pandas.qcut
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html#pandas.cut

Which choice would it be more appropriate ?
Describing observations
My distribution seems to be linear to a certain point, and then it seems to follow an exponential curve:

Could you advice best approach to approximate the quantiles ?
I am in doubt if calculating quantiles against the whole distribution, or if infer two distributions to best reflect observations.
I thought about extracting one distribution as a collection of data points lower than x standard deviations, and a collection of points greater than x standard deviations.
Respectively, I would have these distributions then:

(describing lower payscales, which seems to me linear)
and

(describing higher payscales, which seems to me exponential)
Advices for the analysis are much appreciated.
Considerations about the sample
Keep in mind each values are reported differently in their frequency (for lower payscales, there are more reports, which it makes sense because there is more turnover).
Should I keep into account to estimate an error ?


texts: Why Gaussian latent variable (noise) for GAN?

texts: When I was reading about GAN, the thing I don't understand is why people often choose the input to a GAN (z) to be samples from a Gaussian? - and then are there also potential problems associated with this?


texts: How to increase accuracy of classifiers?

texts:
I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and SVM. UCI letter recognition dataset with 20000 instances and 16 features is used, which I split in half for training and testing. I have experience with SVM so I quickly set its recognition error to 3.3%. After some experimentation what I got was:

UCI letter recognition:

RTrees - 5.3%
Boost -  13%
MLP -    7.9%
kNN(k=3) -   6.5%
Bayes -  11.5%
SVM -    3.3%

Parameters used:

RTrees - max_num_of_trees_in_the_forrest=200, max_depth=20,
min_sample_count=1
Boost -  boost_type=REAL, weak_count=200, weight_trim_rate=0.95,
max_depth=7
MLP -    method=BACKPROP, param=0.001, max_iter=300 (default values - too
slow to experiment)
kNN(k=3) -   k=3
Bayes -  none
SVM -    RBF kernel, C=10, gamma=0.01

After that I used same parameters and tested on Digits and MNIST datasets by extracting gradient features first (vector size 200 elements):

Digits:

RTrees - 5.1%
Boost -  23.4%
MLP -    4.3%
kNN(k=3) -   7.3%
Bayes -  17.7%
SVM -    4.2%

MNIST:

RTrees - 1.4%
Boost -  out of memory
MLP -    1.0%
kNN(k=3) -   1.2%
Bayes -  34.33%
SVM -    0.6%

I am new to all classifiers except SVM and kNN, for these two I can say the results seem fine. What about others? I expected more from random trees, on MNIST kNN gives better accuracy, any ideas how to get it higher? Boost and Bayes give very low accuracy. In the end I'd like to use these classifiers to make a multiple classifier system. Any advice?


texts: one-hot-encoding categorical data gives error

texts:
I am currently working on the Boston problem hosted on Kaggle.  The dataset is nothing like the Titanic dataset.  There are many categorical columns and I'm trying to one-hot-encode these columns.  I've decided to go with the column MSZoning to get the approach working and work out a strategy to apply it to other categorical columns.  This is a small snippet of the dataset:

Here are the different types of values present in MSZoning, so obviously integer encoding only would be a bad idea:

['RL' 'RM' 'C (all)' 'FV' 'RH']

Here is my attempt on Python to assign MSZoning with the new one-hot-encoded data.  I do know that one-hot-encoding turns each value into a column of its own and assigns binary values to each of them so I realize that this isn't exactly a good idea.  I wanted to try it anyways:

import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")

labelEncoder = LabelEncoder()

train['MSZoning'] = labelEncoder.fit_transform(train['MSZoning'])
train_OHE = OneHotEncoder(categorical_features=train['MSZoning'])
train['MSZoning'] = train_OHE.fit_transform(train['MSZoning']).toarray()

print(train['MSZoning'])

Which is giving me the following (obvious) error:

C:\Users\security\Anaconda3\lib\site-packages\sklearn\preprocessing\_encoders.py:392: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
  "use the ColumnTransformer instead.", DeprecationWarning)
Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Boston.py", line 11, in <module>
    train['MSZoning'] = train_OHE.fit_transform(train['MSZoning']).toarray()
  File "C:\Users\security\Anaconda3\lib\site-packages\sklearn\preprocessing\_encoders.py", line 511, in fit_transform
    self._handle_deprecations(X)
  File "C:\Users\security\Anaconda3\lib\site-packages\sklearn\preprocessing\_encoders.py", line 394, in _handle_deprecations
    n_features = X.shape[1]
IndexError: tuple index out of range

I did read through some Medium posts on this but they didn't exactly relate to what I was trying to do with my dataset as they were working with dummy data with a couple of categorical columns.  What I want to know is, how do I make use of one-hot-encoding after the (attempted) step?


texts: How to feed the input to a Memory Augmented Neural Network (MAAN) to do one shot learning?

texts:
In this paper by Deep-Mind on one shot learning they have published an architecture explaining how the system works with an external meory. I understand the mechanism perfectly. But what I don't understand is how they feed the data in to the algorithm.

In  this part of the paper they explain as follows ,

From this part I actually don't understand is do do the concatenate inputs with labels ? And what is label shuffling ?


texts: Is there a name for this type of line graph where the y axis is time, the x axis is some value, and it's shaded by some condition?

texts: This type of graph is used fairly often when comparing political parties, It shows some economic indicator over time, with an overlay of whichever party is in power. Is there a specific name for this type of graph? I'd like to look into how to reproduce it in various softwares, but don't know it's name.


texts:
I read this and have an ambiguity.
I try to understand well how to calculate the derivative of Loss w.r.t to bias.
In this question, we have this definition:
np.sum(dz2,axis=0,keepdims=True)

Then in Casper's comment, he said that the The derivative of L (loss) w.r.t. b is the sum of the rows
$$
\frac{\partial L}{\partial Z} \times \mathbf{1} =
\begin{bmatrix}
. &.  &. \\
. &.  &.
\end{bmatrix}
\begin{bmatrix}
1\\
1\\
1\\
\end{bmatrix}
$$
But actually, using axis=0, is it not the sum of the columns of ∂/∂ ?
I saw another examples and it seems that they do the sum per column. I don't get how to get this result. Could you give the details with a matrix example?


texts: selecting variable randomly at each node in a tree in Random Forest

texts:
In Random Forest method, for each tree we randomly select a set of variables (features) of fixed size. But once this set is frozen for that particular tree, does the tree behave like a regular decision tree algorithm?

I am assuming that random forest is nothing but generating a bunch of classical 'decision trees' and taking their votes towards the final classification. But in many places whatever description I have read seems to suggest that; for a given decision tree within the forest even at each node we randomly select variables. Is that the case?

Does it mean that at each node in the tree, we randomly select m variables from the variable set which is fixed for that tree? Or from the global variable set of the training dataset? And then from the selected set of variables we select 1 variable heuristically (e.g. whichever variable maximises information gain) -- is that a correct statement?


texts: I am trying to build a simple multi layer perceptron Neural Network in Java, but apparently my calculations are off. I am looking for a beginner-level tutorial which can help me to understand how to properly calculate forward and backward pass, preferably with examples.


texts: How to access an embedding table that is too large to fully load into memory?

texts:
I'm currently trying to find a way of loading/deserializing a .json file containing Flair word embeddings that is too large to fit in my RAM at once (>60GB .json with 32GB of RAM). My current code for loading the embedding is below.

def get_embedding_table(config):
    words_id2vec = json.load(open(config.words_id2vector_filename, 'r'))
    words_vectors = [0] * len(words_id2vec)
    for id, vec in words_id2vec.items():
        words_vectors[int(id)] = vec

    words_vectors.append(list(np.random.uniform(0, 1, config.embedding_dim)))
    words_embedding_table = tf.Variable(name='words_emb_table', initial_value=words_vectors, dtype=tf.float32)

The rest of the code that I am trying to reproduce with a different word embedding can be found here.

I wonder if it is somehow possible to access the embedding table without deserialization of the entire .json file, for example: by sequentially reading it, somehow splitting it, or reading it directly from my disk. I would greatly appreciate your input!


texts: How to improve accuracy of deep neural networks

texts:
I am using Tensorflow to predict whether the given sentence is positive and negative. I have take 5000 samples of positive sentences and 5000 samples of negative sentences. 90% of the data I used it for training the neural network and rest 10% for testing.

Below is the parameter initialisation.
   Batch size = 100
   Epochs = 30
   Number of hidden layers = 3
   Nodes in each hidden layer = 100
   epochs = 30

I could see in each epoch the cost function is getting reduced reasonably. However the accuracy of the model on test set is poor (only 56%)

 Epoch  1 completed out of  30 loss :  22611.10902404785
 Epoch  2 completed out of  30 loss :  12377.467597961426
 Epoch  3 completed out of  30 loss :  8659.753067016602
 Epoch  4 completed out of  30 loss :  6678.618850708008
 Epoch  5 completed out of  30 loss :  5391.995906829834
 Epoch  6 completed out of  30 loss :  4476.406986236572
 Epoch  7 completed out of  30 loss :  3776.497922897339
 -------------------------------------------------------
 Epoch  25 completed out of  30 loss :  478.93606185913086
 Epoch  26 completed out of  30 loss :  450.8017848730087
 Epoch  27 completed out of  30 loss :  435.0913710594177
 Epoch  28 completed out of  30 loss :  452.10553523898125
 Epoch  29 completed out of  30 loss :  539.5199084281921
 Epoch  30 completed out of  30 loss :  685.9198244810104
 Accuracy of Train : 0.88155556
 Accuracy of Test : 0.524

Is there any parameter that can be tuned to increase the accuracy of the model considering the same number of data set.


texts:
How to evaluate feature importance for categorical feature in regression problem please?

if I use RandomForestRegressor.feature_importances_ , then it will be done on a dummy variable (post onehot-encoding), then I average it for each feature?

Thank you!


texts: What do you do with one hot encoding items that are a non-match for all classes in a confusion matrix?

texts: I have trained a model for one-hot binary prediction for many classes, and am now applying it to the testing set of samples. However, a lot of the predictions for samples are 0 for every class. I'm not sure what to do with these results, as I need to make a confusion matrix (nxn for the number of classes) but I don't know where these predicted-no-class results should go. Do I just discard them? I would imagine that this would create a faulty image of the error rate of the model.


texts: Difference between stacked lstm vs attention mechanism in LSTM

texts:
What is the difference between stacked lstm vs attention mechanism in LSTM? It seem to me that both produce the same context vector at the end.
EDIT: From suggestion by @shepan6,

the difference in architecture of stacked ltsm and attention mechanism
not too sure i am right here, it seem that both used the notion of being able to select important features to construct the final context vector.
if the answer to above is yes, am i right to say the goal is the same for both, just that the mechanism is different

Thanks


texts:
I have read this paper about U-Net.

This kind of network is quite similar to an autoencoder, in addition it has concatenations between the encoder and the decoder parts.

I would like to know the meaning of this concatenations, and why the feature maps are cropped before this concatenation.


texts:
I am trying to make a CNN for 3D image recognition but everything is predicted to only one class out of three. And the prediction even quickly converges during the first epoch.   I have been working on this for an week and totally lost.
I have my own several guess why it always converges to one class.

My 3D tensor is huge as 40x35x30 and there are a lot of white spaces because I had to put my objects in cubic box. Would this be problematic?
I have only ~5000 samples for training and ~500 for test sets. Do I have too little samples?
My labels have 3 classes and ~75% of the whole data belong to the class number 1. During the first epoch,  my model quickly converges to predict 99% of data as this class. Would this unbalanced data cause the problem?
My model is designed as
conv1(kernel_size=10,stride=1,filter=32)-batch_norm1-maxpool1-conv2(8,1,64)-batch_norm2-maxpool2-conv3(8,1,64)-maxpool3-fc4(1024)-batchnorm4-dropout4(20%)-fc5(384)-batchnorm5-dropout5(20%)-fc6(3)
I standardized the input between -1 and 1 I use leaky-relu activation for conv layers. I use Adam optimizer with decay rate of 0.99.
First, I am not sure if it is okay to perform batch normalization at every layer. Do I miss any important concept for designing CNN model here?
or maybe my data is just bad.. I kind of suspect that my data are not significantly different from each other and there is no pattern at all.  In this case, is there any statistical method/model to check if my data have meaningful differences? (The 3D images are some chemical/physical data in 3D space that I converted as numpy matrix)

But I think it is more like vanishing problem because when I initialize the variable with Xavier's way, the convergence to the class #1 is slower. Please someone help me :(


texts: I have been advised by my supervisor that if my U-Net segmentation network has RGB images at the input then I could use the channels for different images - median filter for R, normalization for G, canny-edge detection for B (example). I have no idea how to do that. Tried to find something similiar here but unfortunately without a success. Would be greateful if somebody could explain me this in detail, I'm new in DL. Thanks


texts: Propensity Modeling for Retail Marketing: Model Adjustments Based on Consumer Life Changes.

texts: I built a predictive model using logistic regression for direct marking creatives. I built a “children’s” model which predicts buying behavior for those that purchase boys, girls and baby apparel.  But one may have a high propensity right now to buy baby clothes but in 12 months, when ones  kid is a toddler, their propensity drops for baby and increases for Girls/Boys.  I’ve been trying to brainstorm ways to take into account that scenario.  My question is, can anyone suggest options for adjustments to the model to based on that additional criteria?


texts: Data balance -before or after feature selection/engineering

texts:
Among the different stages of creating a prediction model, when should I balance the data?

Should it be at the beginning before I perform any manipulation on the data and creating new features, or after, when I already selected the most important features based on the real data?


texts: Keras create custom network with certain unconnected neurons

texts: I am trying to create an autoencoder where a few inputs can affect the other inputs, but not the other way around. An example of this is given by the following picture. I looked into defining my own Keras layers, but don't know how to proceed.


texts: Identifying members who are likely to move to a lower cost alternative product

texts:
Overview:
I am looking for some technical direction from the ML/data science community about how I could tackle my business problem.

Context:
In a nutshell, I have a group of customers who repeatedly purchase products from a category of products e.g. the frying pan category. In the frying pan category there are 15-20 different manufacturers of frying pans. Each of the different frying pans have different features i.e. large, small, non-stick, stainless steel, dishwasher proof etc. but essentially they're all frying pans. Some people buy the premium product while others buy the budget and some others buy the mid range product.

The most popular brand of product is the most expensive and is a loss making product for us. There is however another product (mid range product) that has nearly the exact same characteristics as the expensive pan but is half the cost and generates the most profit for my company.

Objective:
I would like to identify those customers who currently purchase the most expensive product and reach out to them based on their likelihood of purchasing the mid range product to convince them of the benefits of the mid range pan.

Approaches:
I thought one approach to this would be to find the members who buy the expensive product but are most similar in characteristics/features of the mid range purchasers and offer them a discount to incentivize them to switch products. To achieve this I would use a user - user similarity measurement rather than an item - item similarity measure.

The data:
I have access to a swathe of demographic information, purchase history data, product detail characteristics so building analytical base tables is not a problem.

If you have any suggestions as to how I could approach this problem I would be very grateful for your insights.


texts: How to deal with a skewed data-set having all the samples almost similar?


texts: Is CRF suitable for multi-words Named Entity Recognition?

texts:
I've a problem where I should create a custom NER by using sklearn CRF. In the official tutorial, they are using CoNLL2002 corpus is available in NLTK where the entities are represented with a single word but in my problem, an entity can be formed with multiple words ex: United States of America, Cinema at Miami, etc.
Can CRF handle this?


texts: How to interpret skimage orientation to straighten images?

texts:
I have a bunch of images that I am trying to straighten so the images are horizontal (major axis is horizontal) but I don't understand the orientation output from regionprops method in skimage. How to convert it into degrees ? What is the axis reference for the output angle ?
Here is the skimage doc:

orientation : float. Angle between the 0th axis (rows) and the major axis
of the ellipse that has the same second moments as the region, ranging
from -pi/2 to pi/2 counter-clockwise.

My question:
Given the orientation, how do I calculate the angle in degrees to rotate the image so that the major axis is horizontal with skimage?
Code sample
Basically, the main major and minor axis of this object belong to index 1 of the pandas dataframe. The orientation of the object is -1.184075 and should belong to the major axis.
from skimage.measure import label, regionprops_table

# connected pixels of same label get assigned a value
label_img = label(binary_image_here)
props = regionprops_table(label_img, properties=('centroid',
                                             'bbox',
                                             'orientation',
                                             'major_axis_length',
                                             'minor_axis_length'))

df = pd.DataFrame(props)


texts: What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

texts:
So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by dividing the weight vector by the $L_2$ norm. The equation (copied from here), for it, is:

$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_t}} \odot g_{t}$

where, if i understand correctly, $\sqrt{G_t}$, is the root of the sum of the square of the gradients, which is the $L_2$ norm. Now if I understand correctly, $g_t$ in this case would be considered the "loss function"

Then I got introduced to Ridge Regression, which basically adds the $L_2$ norm (scaled by a lambda) to the loss function. The intuition behind this is smoothing the loss function so that it doesn't overfit the data. The equation, (reworked from here) is:

$Loss_{L_2} = Loss + \lambda L_2$

So my question, assuming all my understanding so far is correct, is: What is the intuition behind adding as opposed to dividing by the $L_2$ norm?

I understand that you can regularize the loss function in AdaGrad and that they are two separate things, but I still am having trouble wrapping my head around how the math works the way it does. Like why does dividing by the $L_2$ norm make it converge faster? Why doesn't this just smooth the weight vector as well? What is going on here intuitively?


texts: Euclidean distance for more than 2 datapoints

texts: I need to compare $n$ 3-dimensional vectors with $k$ 3-dimensional vectors using euclidean distance. Is that possible?


texts: How Linear SVM Regression and Multiple Linear Regression different in terms of the regression result?

texts:
They starts from the same equation as below.

y = w*x + b

But they solve it differently. MLR specified the w and b by minimizing the square error whereas SVM specified w and b by minimizing the loss function defined by C and epsilon.

I am wondering if the result of regression is significantly different. I guess that if the given data set is clean and well-explained by input features, the resultant w and b between SVM and MLR will be close. Putting my original question differently, I don't find any reasons to use linear SVM regression over multiple linear regression.


texts:
Is there a way of making custom Python packages generally available within Jupyter lab?

While their documentation suggests to make install-able versions, and going through PIP, this is rather inconvenient for exploratory data science (especially, while I keep extending/developing these packages).

Usually I would point to the location of packages as a PYTHONPATH in .bash_profile (MacOS) – a step that would make them accessible in other contexts (e.g.: Jupyter notebooks) - but does not work with Jupyter lab (on the same machine).


texts: character level n-gram embedded LSTM is overfitting strong

texts:
I am working on a character level classification LSTM and I used uni-gram
(hello -> h, e, l, l, o). So my vocab size was 28 (alphabet + " " + "-"). Now I implemented (correctly, I checked many times) bi-gram
(hello -> he, el, ll, lo) but the LSTM is heavily overfitting. The vocab size is now 677 (len(alphabet)** 2 + 1).
My train- and validation accuracy are starting at ~10%. In the second epoch the train accuracy is ~60% while the validation accuracy wont really increase.
This is how I preprocess my words using pytorch (contains pseudo-code):
word = "hello"
bi_gram_word = ["he", "el", "ll", "lo"]
bi_gram_idx = [bi_gram_table[l] for l in bi_gram_word]
sample = torch.tensor(bi_gram_word)

# x is now a batch containing ```batch_size``` samples
x = batch
# pad every sample in the batch to the same size (=max sample size) using zeros
x = pad_zeros(batch, max(batch)

embedding_layer = nn.Embedding(vocab_size=677, embedding_dim=200)
x = ebedding_layer(x)

These are the learning curves:

I know that using bigger n-grams is more likley to overfit, but in this case it seems a little bit odd. I have enough data: 13 classes with each 40000 samples.
Does someone know how to counter this problem? Help appreciated :)


texts: Gradient Descent - how many values are calculated in loss function?

texts:
I'm a little bit confused how loss function is calculated in neural network training. There's is said that in theory when using Grid Search or Monte Carlo methods we can calculate all the possible loss function values. But obviously this requires too many resources and is not a good way for neural network training. Alternatively, when using Gradient Descent we have a possibility to evaluate a single value to know in which direction we should go in order to test for the next value. We then could climb down the ladder one step by another until we reach the optimal value.

But on the other hand, in the following PyTorch example there is said that the loss function is calculated on the basis of all predicted and real values. And after that the gradient is calculated.
So, what's the point of Gradient Descent in this way when all the loss values in previous step are calculated anyway?


texts:
Given a sentence "I like blue jeans", the output should be "blue".

I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag.

How do I build a model for this? I haven't learnt word2vec but could that work? Pre-trained embeddings? I have not worked on them before yet but I want to know if it would work beforehand.

I want to know what would be the best approach and any references to similar problems worked on before. Thanks.


texts: calculate days from a column to today in r

texts:
I have data set with a Date column, now i am trying to calculate duration in days from today to dates in date column with a new column named duration.
I tried

    duration <- today() - sales$Date

But is giving errors
What is the best way to it?


texts: Can we apply community detection algorithms for word vector space?

texts:
As I understand we can apply community detection algorithms such as Louvain to detect communities in a social network (i.e. involves people).

But I am quite interested in knowing if we can use the same community detection algorithms such as Louvain to identify communities in word vector space (e.g., word2vec), instead of clustering?


texts: How to define discrete action space with continuous values in OpenAI Gym?

texts:
I am trying to use a reinforcement learning solution in an OpenAI Gym environment that has 6 discrete actions with continuous values, e.g. increase parameter 1 with 2.2, decrease parameter 1 with 1.6, decrease parameter 3 with 1 etc.

I have seen in this code that such an action space was implemented as a continuous space where the first value is approximated to discrete values (e.g. 0 if it is < 1 or 2 if it is < 2 and > 1).

Does anybody know if the above solution is the correct way to implement such an action space? Or does Gym offer another way?


texts:
I have two separate files for Testing and Training.
In the training data, I am dropping rows that contain too many missing values .
But , In the test data , I cannot afford to drop the rows so I have chosen to impute the missing values using KNN approach .
My question is , to impute missing values in the test data using KNN , is it enough to consider only the test data ? As in , neighbors in the test data alone ?


texts: Machine Learning & Partial Differential Equations

texts: Are there any algorithms which were developed using partial differential equations for tackling some of the machine learning problems? Most works I see online are in the field of computer vision and a few bizarre ones in topic modelling. But just curious if someone has used or seen it being used for some decision making process or classification problems?


texts: How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content?

texts: I am trying to come up with a platform which can Synthesize Quality Content among many articles found about some particular "topic" on the internet, the algorithm should be able to say recommend top 10 articles found on the internet about the topic. What algorithms and methods should be used to evaluate writing quality and give some kind of a score on many other parameters like "source credibility". I have taken care of crawling and finding out the content, looking for algorithms and methods to evaluate this. Please suggest resources(course,research papers etc..) for the same.


texts: How to better represent three sets of categorical data?

texts:
Given three set of data with categorical integer x-axis with the same range (0-10):

from itertools import chain
from collections import Counter, defaultdict

from IPython.display import Image
import pandas as pd
import numpy as np

import seaborn as sns
import colorlover as cl
import matplotlib.pyplot as plt

data1 = Counter({8: 10576, 9: 10114, 7: 9504, 6: 7331, 10: 6845, 5: 5007, 4: 3037, 3: 1792, 2: 908, 1: 368, 0: 158})
data2 = Counter({5: 9030, 6: 8347, 4: 8149, 7: 6822, 3: 6153, 8: 5075, 2: 3973, 9: 3205, 1: 2247, 10: 1580, 0: 1059})
data3 = Counter({4: 9884, 5: 9058, 3: 9048, 2: 6942, 6: 6630, 1: 4596, 7: 3784, 0: 2524, 8: 1912, 9: 917, 10: 345})

df_data1 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data1.items()])))
df_data2 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data2.items()])))
df_data3 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data3.items()])))

I can plot three different barcharts for each set of data as such:

sns.set(style="white")
g = sns.factorplot(x=0, data=df_data1, kind="count", color='#4882D5', size=6, aspect=1.5, order=None)
g.set_xticklabels(step=1)
plt.show()
g = sns.factorplot(x=0, data=df_data2, kind="count", color='#9CD2D5', size=6, aspect=1.5, order=None)
g.set_xticklabels(step=1)
plt.show()
g = sns.factorplot(x=0, data=df_data3, kind="count", color='#EF988D', size=6, aspect=1.5, order=None)
g.set_xticklabels(step=1)
plt.show()

[out]:

What I want to highlight is the fact that:

for data1, we see that the center of mass is inclined to the right where the x-axis is higher at 8
for data2, it's normally shaped where the center is at 5
for data3, the center of mass is at 4 on the lower side.

Imagine:

data1 x-axis represents the no. of blueberry a person buys at the supermarket
data2 x-axis represents the no. of watermelon a person buys at the supermarket
data3 x-axis represents the no. of oranges a person buys at the supermarket

And

The y-axis for all 3 datasets represent the number of people who buys the respective no. of fruits as per the dataset.

It's a little hard to convey the message with the barcharts.

Is there a better way to represent the dataset and highlight what I want to say?

Is there a way to present the 3 graphs as one? How would it look like?


texts: Time-series prediction/forecast coupled with classification

texts:
I am trying to predict a danger zone.

I have a multidimensional time-series data and I want to forecast two timeseries for a defined number of time points. Then, I want to classify my predictions and see whether there will be a certain pattern that is medically dangerous.

Therefore, I am not sure "how to prepare my data optimally such that the prediction is the most accurate" and I guess there are different methods of preparation depending on the factors such as range of prediction vs overall accuracy.

Currently I am using Keras with Tensorflow.


texts: Best Feature extraction for at the end retrieving audio

texts:
I work on a machine learning algo, which basically learns sequences in an audio .wav and generates the most “logical” sequences. The algorithm learns features, so I generate MFCCs from the audio file. But at the end it’s impossible to do MFCCs to .wav properly.
I read this paper [1] which says, yeah MFCCs lose a lot of information and give other feature extraction techniques but I don’t really know how to implement them in python.
So if you know some good feature extraction techniques for retrieving/reconstruct audio, with which I can choose the number of coefficients and how many times a vector of these coefficients represents, it would be so much help for me !
I Have test, the Librosa library which can generate MFCC and reconstruct audio with them but the MFCCs generated have weird dimensions like for 12 coef, you get a matrix 12 lines and 1000 columns, or i want the transpose of this matrix.
Thanks.
[1] Gonzalez, Ruben. (2013). Better Than MFCC Audio Classification Features. 10.1007/978-1-4614-3501-3_24.


texts: How to grid search feature selection and neural network hyperparameters in the same grid?

texts:
I'm using the GridSearchCV () class from scikit to perform hyperparameter optimization in a sequential neural network. I've built a pipeline to also find the best number of features by putting a feature selector inside the pipeline. The problem is how to define the input_shape, since this depends on the k parameter from the feature selector. Is it possible to set the value of classifier__input_shape to be the same value (at all times) of feature_selector__feature__selector_k?
I've provided the correspondent piece of code below.
def create_model (learn_rate = 0.01, dropout_rate = 0.0, weight_constraint = 0, input_shape):
  model = Sequential ()
  model.add (Dense (units = 64, activation = 'relu',
                   input_shape = (input_shape, )))
  model.add (Dropout (dropout_rate))
  model.add (Dense (32, activation = 'relu'))
  model.add (Dense (1, activation = 'sigmoid'))
  model.compile (loss = 'binary_crossentropy',
                optimizer = Adam (lr = learn_rate),
                metrics = ['accuracy'])#, metrics.CategoricalAccuracy ()])
  return model

standard_scaler_features = remaining_features
my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

clf = KerasClassifier (build_fn = create_model, verbose = 2)
clf = Pipeline (steps = [('scaler', my_scaler),
                         ('feature_selector', feature_selector_transformer),
                         ('classifier', clf)],
                verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
              'feature_selector__feature_selector__k' : [7, 9, 15],
              'classifier__input_shape' : [7, 9, 15],
              'classifier__epochs' : [2, 3, 4]}
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = STATE)
grid = GridSearchCV (estimator = clf, param_grid = param_grid, scoring = 'f1',
                     verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit (X_train_df, y_train_df)

And the error:
ValueError: Input 0 of layer sequential_9 is incompatible with the layer: expected axis -1 of input shape to have value 9 but received input with shape [None, 7]


texts: DistinctCount for relatedtable with additional filter

texts:
In powerBi, a column is created with below formula, and it is working good.

activedays = COUNTAX(RELATEDTABLE(ActivesInactiveData),ActivesInactiveData[Surface Date])

However, I want to add a filter some thing like

ActivesInactiveData[Topic]<> null

Could you please advice on how to do so, most of the examples on web are using on relatedtable etc - where relation is already build, could you please advice/direct to me what I could find an example of adding filter to relatedtable or filter data from a unrelated table based on field like ID column etc.

Here there is a relation user ID in current table has more than 1 rows in ActivesInactiveData table.


texts: Time required to train a model based on features shape and CPU capacity?

texts: How to estimate the time required time to train a model, given feature shape, CPU/GPU sepcs and type of model