texts: Temporal data types: time-series vs temporal sequences

texts:
Regarding the types of temporal data, I saw on a temporal data mining book (Mitsa, Theophano. Temporal data mining. CRC Press, 2010), that there are 3 types of temporal data: time series, temporal sequences and a semantic temporal data. According to the author:

Time-series: ordered real-value measurements at regular time intervals
Temporal sequences: timestamped at regular or irregular time intervals

I further explored the difference between the two but is still not clear for me, because from what I could understand time-series is a special case of temporal sequences. Temporal sequences can accept nominal values, symbols, discrete, continuous, or categorical values right? And the data may be collected, as mentioned above, at regular or irregular time intervals. On the other hand, time-series data can only be continuous measurements at regular time intervals, which seems to be a special case of temporal sequences. Am I interpreting wrongly?


texts: Analyze additions and subtractions to/from a data set

texts:
I have a data set in the following form:
Product    |    Date
123        |    2019-01-01
456        |    2019-01-01
123        |    2019-01-02
123        |    2019-01-03
456        |    2019-01-03
123        |    2019-01-04
456        |    2019-01-04
789        |    2019-01-04

This is just a simplified version. The full set has ~300 products and four months of data. I want to understand how the product set changed over time. It's obviously easy to calculate the count per day and see that I lost one product on Jan 2nd and gained one on Jan4th, but then I don't know what product it was.
Is there a more systematic way of going about this? Ideally, the output would show me a list of days and what products dropped out / were added that day. I thought about min(date), max(date) by-product before, but products can drop and be added repeatedly and I wouldn't capture this back and forth this way.
Available environments are Python, SQL, and Excel.


texts: Epoch greedy algorithm for contextual bandits

texts:
I'm reading the following paper on the epoch greedy algorithm for the contextual bandits problem. I have two questions

http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf

I'm unsure how they've used the Bernstein inequality on page 6 to conclude $\mu_{n}(\mathcal{H},1) \leq c^{-1} \sqrt{k \mathrm{ln}(m)/n}$. Could someone please elaborate on this as it seems Bernsteins inequality seems to measure whp the deviation of a sum of random variables from it's mean. Where as the regret bound $\mu_{n}(\mathcal{H},1)$ is defined as the expected regret from the empirically best policy from the absolute best policy. Could someone fill in the detail?
Can we get any reasonable estimate for the constant $c$ if i was to try and implement this in practice?

I'd really appreciate any help, Thanks.


texts: How much training data does Word2Vec need?

texts:
I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy".

A brief plan was

Take the books mentioning the term "democracy" as plain text
In each book, replace democracy with democracy_%AuthorName%
Train a word2vec model on these books
Calculate the distance between democracy_AuthorA, democracy_AuthorB, and other relabeled mentions of "democracy"

So each author's "democracy" gets its own vector, which is used for comparison.

But it seems that word2vec requires much more than several books (each relabeled word occurs only in a subset of books) to train reliable vectors. The official page recommends datasets including billions of words.

I just wanted to ask how large should be the subset of one author's books to make such inference with word2vec or alternative tools, if available?


texts: Is there a deep learning method for 3D labels?

texts:
As the question says, I want to feed labels into a neural net that are three dimensional. Let's say that I have 3 possible labels and each one of my data points corresponds to a percentage of those labels. e.g, my first datapoint contains 20% of label A, 30% of label B, and 50% of label C.
Is there any architecture able to deal with this shape of label data?


texts: Feature Extraction Technique - Summarizing a Sequence of Data

texts:
I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the best way possible for inclusion as predictors in the model.

As a concrete example, say a model is being built to predict if a customer will leave the company in the next 90 days (anytime between t and t+90; thus a binary outcome). One of the predictors available is the level of the customers financial balance for periods t_0 to t-1. Maybe this represents monthly observations for the prior 12 months (i.e. 12 measurements).

I am looking for ways to construct features from this series. I use descriptives of each customers series such as the mean, high, low, std dev., fit a OLS regression to get the trend. Are their other methods of calculating features? Other measures of change or volatility?

ADD:

As mentioned in a response below, I also considered (but forgot to add here) using Dynamic Time Warping (DTW) and then hierarchical clustering on the resulting distance matrix - creating some number of clusters and then using the cluster membership as a feature. Scoring test data would likely have to follow a process where the DTW was done on new cases and the cluster centroids - matching the new data series to their closest centroids...


texts:
I have a load of documents, which have a load of key value pairs in them. The key might not be unique so there might be multiple keys of the same type with different values.

I want to compare the similarity of the keys between 2 documents. More specifically the string similarity of these values. I am thinking of using something like the Smith-Waterman Algorithm to compare the similarity.

So I've drawn a picture of how I'm thinking about representing the data -

The values in the cells are the result of the smith-waterman algorithm (or some other string similarity metric).

Image that this matrix represents a key type of "things" I then need to add the "things" similarity score into a vector of 0 or 1. Thats ok.

What I can't figure out is how I determine if the matrix is similar or not similar - ideally I want to convert the matrix to an number between 0 and 1 and then I'll just set a threshold to score it as either 0 or 1.

Any ideas how I can create a score of the matrix? Does anyone know any algorithms that do this type of thing (obviously things like how smith waterman works is kind of applicable).


texts: How to split train/test in recommender systems

texts:
I am working with the MovieLens10M dataset, predicting user ratings.  If I want to fairly evaluate my algorithm, how should I split my training v. test data?

By default, I believe the data is split into train v. test sets where 'test' contains movies previously unseen in the training set.  If my model requires each movie to have been seen at least once in the training set, how should I split my data?  Should I take all but N of each user's ratings for all the data and evaluate my performance on the held out NxUser_num  ratings?


texts:
it happens quite a lot for me that I declared batch_size=64 or any number and end up watching Keras training my net on whole data instead of the batch. then I edit it many times and finally finished with the very first code working.
for example this simple code:
data_train=pd.read_csv("train.csv")
y=data_train["label"]
dt=data_train.drop("label", axis=1)
inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(500, activation="relu", name="dense_1")(inputs)
x = layers.Dense(500, activation="relu", name="dense_2")(x)
x = layers.Dense(300, activation="relu", name="dense_3")(x)
x = layers.Dense(64, activation="relu", name="dense_4")(x)
outputs = layers.Dense(10, activation="softmax", name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)
opt = keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=opt, loss="sparse_categorical_crossentropy", metrics=["accuracy"])
X_train, X_test, y_train, y_test = train_test_split(dt, y, test_size=0.2)
X_train, Xvalid, y_train, yvalid = train_test_split(X_train, y_train, test_size=0.2)
history=model.fit(X_train,y_train, batch_size=64 ,epochs=30,validation_data=(Xvalid,yvalid))

and it train itself with :
26880/26880 [==============================] - 3s 108us/sample - loss: 0.4717 - acc: 0.8718 - val_loss: 0.4149 - val_acc: 0.8847
how I can fix it?
Thanks


texts:
2 images as input, x1 and x2  and try to use convolution as a similarity measure. The idea is that the learned weights substitute more traditional measure of similarity (cross correlation, NN, ...). Defining my forward function as follows:

def forward(self,x1,x2):
    out_conv1a = self.conv1(x1)
    out_conv2a = self.conv2(out_conv1a)
    out_conv3a = self.conv3(out_conv2a)

    out_conv1b = self.conv1(x2)
    out_conv2b = self.conv2(out_conv1b)
    out_conv3b = self.conv3(out_conv2b)

Now for the similarity measure:

out_cat = torch.cat([out_conv3a, out_conv3b],dim=1)
futher_conv = nn.Conv2d(out_cat)

Question is as follows:

Would Depthwise/Separable Convolutions as in the google paper yield any advantage over 2d convolution of the concatenated input. For that matter can convolution be a similarity measure, cross correlation and convolution are very similar.
It is my understanding that the groups=2 option in conv2d would provide 2 separate inputs to train weights with, in this case each of the previous networks weights. How are these combined afterwards?


texts: How to change the pooling to adaptpooling before FC layer in the Inception-ResnetV2 model in keras

texts:
Now I am using the keras model: Inception-ResnetV2 to do image classification using transfer learning. The main code about this model is as following:

 def model_InceptionResNetV2(img_dim, num_label):
    print('begin to get model')
    input_tensor = Input(shape=img_dim)
    #InceptionResNetV2
#     base_model = InceptionV3(include_top=False,
#                    weights='imagenet',
#                    input_shape=img_dim)
    base_model = InceptionResNetV2(include_top=False, input_shape=img_dim, weights='imagenet')
    bn = BatchNormalization()(input_tensor)
    x = base_model(bn)
    x = GlobalAveragePooling2D()(x)
    x = Dropout(0.5)(x)
    output = Dense(num_label, activation='softmax')(x)
    model = Model(input_tensor, output)
    print('finish getting model')
    return model

But the model is not good for my image classification, some guys suggest me changing the pooling to adaptpooling before FC layer in model_InceptionResNetV2 so that the mode is not sensitive to image size, I want to use 512*512 size not just 299*299. Could you guys suggest me how to do it. Thanks!


texts: What is the best way to scale a numerical dataset

texts:
I have a dataset with differents attributes which don't have the same range on their values which is a problem when we need to compute distance beetween objects. After some research i found that i can do the regularisation job with this formula : (value-min)/(max-min)  where min and max are respectively the minimum and maximum value in the domain of val attribute.
The question is that one, does it exist other ways ?

Thank you for your help.


texts:
I had a doubt about groupby operations. Lets suppose I have grouped my data based on one column and got 5 groups as output. I know that we can iterate over these groups and apply functions to them as a whole. But can I access the elements of each group (for example I have 5 groups each having 5 rows, can I access those rows one by one). I want to apply a function that compares two rows of a single group based on one or more parameters of the rows belonging to that group.
Year   Rank    Final_Score    Team
2000    1           50          A
2000    2           35          B
2000    3           65          C
2003    3           20          B
2003    1           53          C
2003    2           44          A

If I have grouped by years such that the two groups would be group for year 2000 and 2003 respectively. I want to compare row1 and row2 of group 2000 by comparing their attribute Final Score and for this I need to access each row in the group. Can I do that by grouby()?


texts: why is the model prediction output in keras lstm imdb, a vector?

texts:
If you run the example of the LSTM sentiment classification task example in keras

  https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py

and add

p = model.predict(x_test[0])
print(p.shape)

you get:
(80,1)

.. why? I thought the point of the model was to classify each sample as 0 or 1 as per each of the y_train elements (binary classification)


texts: One-Hot Vector representation vs Label Encoding for Categorical Variables

texts:
I have one categorical variable of string type in my dataset. I need to convert it to numerical value for further processing. I know standard way to represent categorical data is to use one-hot encoding. But that will convert each entry of the variable to a vector.

LabelEncoder of sklearn converts each entry to a scalar value. I realise this is a very naive and possibly stupid question but which representation is more commonly used and is there a reason for the bias?


texts: Averaging CNN perform worse than boosting

texts:
I'm trying to solve Quora Question Pairs with model stacking.
My first layers are:

CNN trained to predict the same target as whole model should
"Magic features" like question frequency in whole dataset

And the second is gradient boosting.
My first attempt was to train CNN on whole(with 10% to dev) train dataset and than train gradient boosting (GB) on the same split.
As expected it was terrible because GB learned that CNN almost always right.
So I split train data into 10 parts and trained 10 different CNN with 1 part as dev.
Than I trained GB with CNN predictions on dev parts(so it's almost unseen data). And used average of 10 CNNs for same feature on test dataset.
The result was worse than gradient boosting applied to "magic features" alone.
Can you help me to identify my mistakes?


texts: Prediction on timeseries data using tensorflow

texts:
I have an input and output of below format:

(X) = [[ 0  1  2]
       [ 1  2  3]]

  y   = [ 3  4 ]

It's timeseries data. The task is to predict the next number. Basically, the input was crafted by the below snippet:

 def split_sequence(arr,timesteps):
     arr_len = len(arr)
     X,y = [],[]
     for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component =  arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

     return np.array(X), np.array(y)

Now, I would like to train the model on the input and predict the next number. For instance, x = [81,82,83] and the predicted output would be y = 84. I learned how to do this in keras. However, I would like to try to do this in tensorflow as well.

Below is the code in tensorflow:

 # Data generator
 def generate_batch(X,y,batch_size):
    m = X.shape[0]
    indexes = range(m)
    n_batches = m // batch_size
    for batch_index in np.array_split(indexes,n_batches):
       yield X[batch_index],y[batch_index]

 # parameters
 n_inputs = 3
 n_epochs = 1000
 batch_size = 40
 learning_rate = 0.01
 n_steps = 3

 # generate the input and output using split_sequence method
 input, output = split_sequence(range(1000),n_steps)

 # Define the input variables
 X = tf.placeholder(tf.int32,shape=(None,n_inputs),name='X')
 y = tf.placeholder(tf.float32,shape=(None),name='y')
 theta = tf.Variable(tf.random_uniform([n_steps,1],-1.0,1.0),name='theta')

 # predictions and error
 y_predictions = tf.matmul(X,theta,name='predictions')
 error = y_predictions - y
 mse = tf.reduce_mean(tf.square(error),name='mse')

 # train the model
 optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
 training_op = optimizer.minimize(mse)

 init = tf.global_variables_initializer()

 with tf.Session() as session:
   session.run(init)
   for epoch in range(n_epochs):
      for X_batch,y_batch in generate_batch(input,output,batch_size):
         if epoch % 10 == 0:
             print('epoch',epoch,'MSE=',mse.eval())
         session.run(training_op,feed_dict={X:X_batch,y:y_batch})

To be honest, I am completely stuck with the below error:

You must feed a value for placeholder tensor 'X' with dtype float and shape [?,3].

My input is an integer, so that was the reason behind defining:

 X = tf.placeholder(tf.int32,shape=(None,n_inputs),name='X')

Can someone help me fix this? Also, if I wanted to add bias variable will I be able to achieve for the above input?


texts: How can I turn user audit data into behavior clusters?

texts:
I have logs of user activity on my system. This is a CMS system.

The logs consist of:

User ID
Action Performed (one of 110 possible actions - things like page edited, page read, login, etc.)
Site action was performed on (not relevant for all possible actions)
Object action was performed on (document name or document id - not relevant for all possible actions)
Date and time that the action was performed

I would like to use this data to cluster users into groups with similar behavior, so that I can then survey the groups to find out who they are and then target them with training interventions, newsletters etc.

I need help with converting the above list of things into clusters.

What techniques can I use to generate a matrix from the above data that I can feed into a K-Means cluster algorithm?

I tried counting action by user and using that to feed a cluster algorithm but the result was useless. I also tried treating the site and document ids as text, feature hashing that and feeding it into a K-Means cluster model, also got useless data out.


texts: Error: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2

texts:
I am getting the error:
Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2

Using the model:
X = np.reshape(x_train_tfidf.shape[0], 1, x_train_tfidf.shape[1])
print(X.shape)
model.add(LSTM(30, return_sequences=True,
               input_shape=X))

model.add(Flatten())
model.add(Dropout(0.25))
model.add(Dense(100,activation = 'relu'))
model.add(Dropout(0.25))
model.add(Dense(1,activation='sigmoid'))


texts: How to add a column for descending row numbers into dataset in R

texts:
I am new to R and would like to insert a new column that numbers the row to a large dataset.
I have no idea how to use 'mutate()' to insert this. Would appreciate any help.
Thanks.


texts: How to detect misclasification data after multiclass classification?

texts:
I have trained a neural network multi-class classification model with around 150 classes having around 85% accuracy. Once the model is trained and deployed, it's predicting on new data and I am saving the logs. Now I have to detect those data-points which is wrongly predicted by the model.
For example, The model predicted on 10 data-points, out of which there might be 3 data-points which are wrongly classified by the model. Is there any way to get those data-points? I have the following data.

The deployed model.
The data-points.
Corresponding predicted classes by the model.
Confidence scores on the prediction.

I know that using confidence score I might get some idea on wrong prediction, but I am thinking is there any other way to get that?


texts: How to preprocess with NLP a big dataset for text classification

texts:
TL;DR
I've never done nlp before and I feel like I'm not doing it in the good way. I'd like to know if I'm really doing things in a bad way since the beginning or there's still hope to fix those problems mentioned later.
Some basic info
I'm trying to do some binary text classification for a university task and I'm struggling at the classification because the preprocessing with NLP is not being the best.
First of all, it's important to note that I need to have efficiency in mind when designing things because I'm working with very large datasets (>1M texts) that are loaded in memory.
This datasets contains data related to new articles with title, summary, content, published_date, section, tags, authors...
Also, it's important to mention that as this task being part of a learning process I'm trying to create everything by myself instead of using external libraries (only for boring or complex tasks)
Procedure
The basic procedure for the NLP preprocessing is:

Feature extraction -> str variable with title, summary and content attributes joined in the same string
Lemmatization -> same str as input but with lemmatized words
Stopword filtering
Corpus generation -> dict object with lemmatized words as key and the index they're being inserted in the dictionary as value.

After generating the corpus with all those samples, we can finally safely vectorize them (which is basically the same process as above but without the building corpus step).
As you might guess, I'm not strictly following the basic bag of words (BOW) idea since I need to relieve memory consumption so it raises two problems when trying to work with AI algorithms like DecisionTreeClassifier from sci-kit.
Problems
Some of the problems I've observed till the moment are:

Vectors generated from those texts needs to have the same dimension Does padding them with zeroes make any sense?
Vectors for prediction needs also to have the same dimension as those from the training
At prediction phase, those words that hasn't been added to the corpus are ignored
Also, the vectorization doesn't make much sense since they are like [0, 1, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] and this is different to [1, 0, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] even though they both contain the same information


texts: How to treat Compass data in random forest regression

texts: I'm working on a project where two of the features are entryHeading and exitHeading. Both state the direction (N, NE, E, SE, S, SW, W) of a vehicle at multiple points. My question is how would i go about pre-processing this? My first thought would be to circularize it like I would a 24 hour period but I'm not sure I should go about it in the same way. The data will eventually be used to train a Random forest regression


texts: Segmentation model to predict face forward and profile parts of the face

texts:
I am developing a model for feature counting on a person's face that consumes three photos (one face forward and two profile pictures). My model can already detect features, but it counts some features twice.
Consider the image below. There are only four labels (1, 2, 3, 4 and 5). Model detects four labels (1, 2, 3, 4) on the profile photo and four labels on the face forward photo (1, 2, 3, 5).
Labels (1, 2, 3) are counted twice. A straightforward approach would be to segment hundreds of faces into "face forward region" and "profile region". This is tedious and accurate labeling is hard. Is there a better way to segment the face in regions or to detect duplication of the detected features?


texts: Predicting descrete value problem in regression or classification

texts:
In machine learning, regression algorithms attempt to estimate the mapping function (f) from the input variables (x) to numerical or continuous output variables (y).

I have one usecase where I predict shift_id. Shit_Id is ID values given to different city location.

As per my understanding this is regression problem because it predict numerical value. Is this right?

Also precision, recall f1 measure can be calculated for regression problem?


texts: Multiple binary dummy features Vs Multi-values single feature

texts:
I found a script on Kaggle's titanic competion in wich the creator convert Multi-values single feature (namely Pclass = {1,2,3}) to 3 binary features.

What are the pro/cons of such a conversion? Does it really improve something, or is it worse? Or doesn't change anything?

Despite taking longer to compute, I also worry that it could introduce a bias in the model.


texts: Keras: Softmax output into embedding layer

texts:
I'm trying to build an encoder-decoder network in Keras to generate a sentence of a particular style. As my problem is unsupervised i.e. I don't have the ground truths for the generated sentences, I use a classifier to help during training. I pass the decoder's output into the classifier to tell me what style the decoded sentence is.

The decoder outputs a softmax distribution which I was intending to feed straight into the classifier but I realised that it has an embedding layer, which in Keras, accepts only integer sequences not softmax / one-hot. Does anybody know a way to remedy this?

Thanks


texts: Is there any research on zonal OCR / field level ORC / template OCR?

texts:
I've recently found the term "Zonal OCR": (source 1, 2, 3, 4). It seems to be essentially OCR, but restricted to relevant parts of the document.
The interesting task about which I want to learn more is how to find those zones automatically. How is this called? Is this a field of research? Could you point me to a publication so that I can start learning about this topic?
("Field Level recognition" and "Template OCR" seems to be a synonym for zonal OCR)
What I found

Dynamic Programming Approach to Template-based OCR, but they don't reference anything OCR related


texts: How do we know that model learned those features from the image dataset in deep learning?

texts: I am implementing image classification using TensorFlow.How do we know that model learned those features from the image dataset in deep learning. can we visualize that?


texts:
There is a example in  sklearn.metrics.average_precision_score documentation.

import numpy as np
from sklearn.metrics import average_precision_score
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
average_precision_score(y_true, y_scores)
0.83

But when I plot precision_recall_curve

precision, recall, _ = precision_recall_curve(y_true, y_scores)
plt.plot( recall,precision)

I got the picture:

why the area under the precision_recall_curve is not 0.83?


texts: what is the best to do with highly correlated features?

texts:
In my data set 2 features C1 and C2 are highly correlated. I did the following steps. Could you please let me know if it is correct and make sense? do you have a better approach?
First I used the linear model to find the  fitted line:
C1=a*C2+b
from sklearn import linear_model
reg=linear_model.LinearRegression()
y_reg=data1['C1']
x_reg=data1['C2']
reg.fit(x_reg2,y_reg2)
a=reg.coef_
b=reg.intercept_
print(a,b)

After finding a and b I removed C1 and C2 from the dataset and added a new Variable: new=a*C1+b
My next question is how I can understand if this line is good?


texts: Why are there currently no content-based evaluation metrics for information retrieval?

texts:
Consider the problem of learning to rank for Google-like searching - i.e., learning to return a good ordering of URL's when given a query. Most (if not all) current evaluation metrics for this problem are URL-based.  Current evaluation metrics like Mean Reciprocal Rank (MRR) and Discounted Cumulative Gain (DCG) sum up weighted relevance scores for the pages in a ranked list.  If a ranked list contains n pages, for instance, these metrics give a sum of n terms.

From my own literature review, it seems no metrics incorporate words.  Although some theoretically incorporate "information nuggets", none incorporate "information nuggets=words" in practice.  I have not found recent or past literature that concretely utilizes words, phrases, or anything similar in scoring ranked lists. Since I haven't found past literature, it does not seem that word-based metrics were attempted once and then replaced with coarser URL-based metrics.  Documents contain words, as do the snippets returned in a Google-like ranked list, so it seems natural that word-based metrics should have been attempted.  Thoughts?

(Note: One obvious objection is that TREC-based assessment of "word relevance" is a vague and even expensive endeavor (asking assessors to mark for words for relevance).  But there's other ways to get relevance feedback that ignore this problem, such as using the dwell time on pages as a proxy.)


texts: Understanding logistic regression loss function equation

texts:
I am writing a scientific paper that - among other things - deals with logistic regression in the context of machine learning.
I read this article where the author states that, given a set of instance-label pairs $(\boldsymbol{x_i},y_i),\  i=1,...,l\ \ \boldsymbol(x_i) \in R^n,\ y_i \in \{-1,+1\}$, Logistic Regression solves the following optimization problem:

$$
\begin{equation}\label{eq:log}
  \min_{w} \frac{1}{2}\boldsymbol{w}^T\boldsymbol{w} + C\sum\limits_{i=1}^l log(1+e^{-y_i\boldsymbol{w}^T \boldsymbol{x_i}})
\end{equation}\ \ \ \ \ \ \ \ \ \  (Eq.1)
$$

Is it correct if I say that: "The first summand of Equation 1 is a regularization term while the second represents the negative log-likelihood loss function"?

Thank you for your kind assistance.


texts: MDP - RL, Multiple rewards for the same state possible?

texts:
This question is from  An introduction to RL  Pages 48 and 49. This question may also be related to below question, although I am not sure:
Cannot see what the "notation abuse" is, mentioned by author of book

On page 48, it is mentioned that p:S * R * S * A -> [0,1] is a deterministic function:

  The dynamics function $p : \mathcal{S} \times \mathcal{R} \times \mathcal{S} \times \mathcal{A} \rightarrow [0, 1]$ is an ordinary deterministic function of four arguments.

However, on page 49, in equation 3.4, there is summation over r:

  $$\sum_{s' \in \mathcal{S}}\sum_{r \in \mathcal{R}} p(s',r|s,a) = 1  ,\text{for all } s \in \mathcal{S}, a \in \mathcal{A}(s)$$

My question is, does this mean, it is possible that performing an action $a$ that takes us to state $s'$, could result in multiple rewards?


texts: Fast AI Lesson 4 - MNIST. Confused about multiplying weights by pixels?

texts:
I’m on lesson 4 of the Fast AI "Deep Learning for Coders" course, and have been back through the same lesson a few times now but I don’t think I’m quite getting a few things. I want to have an understanding of what’s going on before moving on.
This lesson is on MNIST - and Jeremy is recognising 3s vs 7s. So he has 12000 images (ignoring mini-batches) of about 800 pixels each, and his tensor has a shape of (12000, 800). He randomly generates 800 weights, 1 for each pixel, plus a bias (which I don’t quite understand the purpose of either).
So then for the first image, he multiplies the image’s value by the weight for that pixel. So if the pixel value is 0.9 and the weight is 0.1, 0.9*0.1 = 0.09. Or if the pixel value is 0.5 and the weight is 0.6, 0.5*0.6 = 0.3.
(and then a bias is added, and this is done for every image)
I don’t understand what the purpose or benefit is of multiplying these numbers together. If I wanted a score of how accurate is our weight to the pixel value, I’d do abs(value - weight).
And then adding these up would give me a score of how accurate my weights are.
This is where I’m confused. what am I missing?


texts: Find the time between two events by customer id

texts:
I need to find a customer has bought P1, and after how many days customer will buy P2.
I am unable to find the days between order of P1 and the next order of P2 by the same customer.

I have data as shown below.

Customer ID   Order_Date   Product
       C-87   11/20/2018        P2
       C-87    7/25/2018        P1
       C-87    7/19/2019        P1
       C-87     8/2/2018        P2
       C-87    12/9/2019        P1
        ...          ...       ...
       C-22    9/22/2018        P2
       C-22     9/4/2018        P2
       C-22    1/15/2018        P1
       C-22     9/5/2019        P2
       C-22    3/20/2018        P1


texts: How to deal with missing data for only some categories

texts:
Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly.
I can't find any logic how to deal with this relative data. So I am sorry that I don't show any effort.

In the following example, geographical zone is only present for Gaz entries.

Data sample:


texts:
I am building a entity detection and relation classification method using deep learning approach which requires vector representation of POS tags and entity label. I am familiar with word-embedding method but I don't know the answer of following questions:

How to convert pos tags into vector representation(say 20 dim)
How to combine both word embeddings and pos embedding together to build the classifier.

Similar question was asked in Cross validate community but I couldn't find the answer. Here is the link to question:
https://stats.stackexchange.com/questions/238016/deep-learning-word-embedding-with-parts-of-speech

Research paper link:
https://arxiv.org/abs/1601.00770


texts:
How can I deal with a time series that contains missing data which means something?

So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a loss of information.

So, you can think of it as something like "measure x only if condition y was met".
In my case a certain price is measured only if there are people interested in that on a given day.
This leads to a time series with several thousands data points, of which around 40% contains no value.

I want to use a 1D-CNN as part of an anomaly detector in order to find errors in the data. The CNN delivers predictions which are compared to the actual data. If the difference between the prediction and the real data point is above a certain threshold I want to mark that as an anomaly. (the missing_values however are no anomaly)

I thought of using embeddings to deal with missing values but am not sure if this would be the correct way of handling this.


texts: How can I get a forecasting model to improve its forecasts over time instead of fitting to training data?

texts:
Say I have a model that I'm using to forecast demand for some product.
I can train it on some training data in order to get good predictions.

But, the predictions it outputs will only really match past demand. This seems to be a problem related to the data that we give the model because if the past demand data is only how much we sold, but that was our supply as well, then this wouldn't be able to help improve future predictions.

For example, if I had in supply 10 units of product x, and I always sell 10 units of that product, then based on this training data, my model will always say that I should always order 10 units for every day. But, really, it might be better to order 25 units because if I'm always selling everything, then there might be more demand than supply. The problem is that I don't know what the true demand is.

So my question is, how would I get my model to account for that and actually improve predictions rather than trying to fit to historical data.


texts: Understanding word2vec vectors representation

texts:
I'm trying to obtain the word2vec representation of few words using gensim.

At present, this is the model that I have:

model = Word2Vec(desc, size=1, seed=7, window=1, iter=1000)

And the current output is

Ansible automate 0.37919244170188904
Docker container -0.05218552052974701
Ansible automation 0.39010459184646606
Automate Ansible 0.1009570062160492
Contain Docker 0.41532352566719055
Dock contain -0.38013115525245667
Dock container -0.17561538517475128

I'm wondering why Contain Docker is closer to Ansible automate and Ansible automation than it is from the other terms which contain the word Docker, Container etc.

How can I train the word2vec model to get this sorted out?


texts: How many run should we implement on the machine learning model?

texts: Theoretically, we can implement fix seed on the machine learning model to get the same results every run  (reproducible)but it may leads to bias. So, in order to prevent bias, I gonna run the model for several times without fix the seed/state. How many time should I run the machine learning model or normally done by researchers to get the result (10 runs,20.. or more)?


texts: Shiny: renderUI with Submit Button - How to make renderUI react to selectInput without Submit Button being pressed?

texts:
Short Version: How do I get renderUI to react to a drop-down list that is in a wellPanel that contains a submitButton without having to click the submitButton?

Long Version: I am building a Shiny app to perform many different analyses on a large dataset. The user will select their analysis from a drop-down list at the top of a wellPanel. Depending on the analysis selected, I'd like to provide a different set of additional inputs/filters in the wellPanel with a submitButton at the bottom.

I would like the user to have to change the dynamic inputs, then click the submitButton for their inputs to take effect. However, I want the renderUI function to react to the drop-down list at the top without the submitButton being clicked. From what I've read, I think I need to use observeEvent or something similar, but I can't figure it out.

Here is a simplified version of what I'm building. Changing the drop-down list selection and clicking submit will change the inputs. Submit will also change the text rendered on the right. However, I need to uncouple the dynamic UI from the submit button. Any help is much appreciated.

ui.r:

library(shiny)

ui <- fluidPage(
    fluidRow(
      column(4,wellPanel(
          selectInput("analysis", label = "Type of Analysis:",
                    c("Award Total" = "total",
                      "Award Average" = "average"),
                    width = validateCssUnit("70%")),
          uiOutput("filter_box"),
          submitButton()
               )),
      column(8, textOutput("sample_text"))
      )
  )

server.r:

shinyServer(function(input, output) {

  output$filter_box <- renderUI({
if(input$analysis == "total"){
  tagList(radioButtons(inputId = "input1", label = "Select One:",c("A", "B", "C"), selected = "A"))
} else {
  tagList(checkboxGroupInput(inputId = "input2", label = "Select all that apply:",c("1","2","3","4","5")),
          dateRangeInput(inputId = "input3", label = "Enter Date Range"))
}
  })

  output$sample_text <- renderText({
if(input$analysis == "total"){
  input$input1
} else if(input$analysis == "average") {
  c(input$input2, input$input3)
}
  })

})


texts:
At what correlation factor we consider two features highly correlated so that it is safe to drop one feature for supervised learning?
It is said that in supervised learning we should remove the high correlated fields or features so that the model would not be biased. In one of my project, I could see high correlation between many variables but not sure at what point should I consider them dropping off from my model.
Can someone suggest here on this question?


texts: Compare distance between embeddings in different dimensions

texts:
I am working on a problem with CNNs.
After the convolutional layers, comes a "flatten". One could interpret that as a representation of the input image in some high-dimensional continuous space. (In some way similar to word embeddings in NLP)
Suppose we have 2 architectures for the same input that result in representations in $\mathbb{R}^N$ and $\mathbb{R}^M$ with $N \neq M$.
Now suppose we have two images $a$ and $b$ which we pass through both neural nets and get their respective representations, lets call them $a_N$ for image $a$ in $\mathbb{R}^N$, $b_N$ for image $b$ in $\mathbb{R}^N$, and so on.
Is there any distance measure $d(x,y)$ so that $d(a_N,b_N)$ and $d(a_M,b_M)$ are comparable?
What I thought so far is calculating the N-dimensional and M-dimensional representation of many image samples from the same category and then use t-SNE so that I take them both to the same dimension.
One could also do the projection from the space of highest dimension to the other.
Does anyone know some proven technique to do that, or could give some advice?


texts:
I'm looking to use the k-Nearest Neighbors (kNN) algorithm.  What are the possible methods for determining the best K?  From what I have read, looking at many different values(say 10-100) should work, but that's a big range.

How can I reduce the range of possible K's I should test?


texts: Does encoding missing data with fixed values help in classification?

texts:
I have a lot of missing values for some variables in my data (70-80%). I have seen some people deal with missing values this way: encode the variable with missing values as 0 or 1. Where 0 is the value is missing and 1 as non missing.

I want to know if that technique is of any use, because I don't see any valuable information algorithm would able to pick from such variables. Also I am thinking of imputing them using mice but the problem is that in future use, we may not be able to get those variables with missing data, so the train and test set will have different number of columns


texts:
I have implemented a Keras Custom Layer in which I call a custom function inside the call method, the issue is when Keras is constructing the model, it is calling my custom layers call function with non - concrete data( maybe to check for shapes or error, don't know the exact reason ) with the shape ( None, 64, 64, 1 ) in which None specifies the batch which is unknown yet, and ( 64, 64, 1 ) is the shape of each image I am feeding to the model. Now, since these images are not concrete as the fit method is not called yet, the custom function I have defined is throwing the error relating to the NULL array passed. My question is:-
1.) Is there any way to tell Keras not to call this custom function if not at runtime?
or
2.) To not let Keras call the custom layer call method before the fit method is called?
Questing Addition:- If while constructing the model, Non-Concrete Data is passed, how come other layers( eg Dense ) don't throw up errors since even they are performing an operation( eg. matmul ) on data which doesn't actually exist?


texts:
I would like to use a model with a bidirectional LSTM layer to predict a sequence of outputs from a sequence of inputs (of length $l$).

To compute the output $o_i$ I would like to use the inputs $x_1, \dots, x_{i-1}$ (the result of the LSTM units up to $i-1$ from the forward pass), and inputs $x_l, \dots, x_{i+1}$ (the result of units $l, \dots, i+1$ from the backwards pass).

In other words I want to exclude input $x_i$ for the LSTM layer's (and network's) output $o_i$.

Does this operation have a name? Is there a simple way to do this in keras or other popular frameworks? Is there some alternative way to achieve the same?


texts: What's the intuition behind nonlinear predictors in multivariate regression?

texts:
I'm learning statistical learning with the well known ISLR (Introduction to Statistical Learning with Applications in R) and doing the exercises, right now in the linear chapter regression. Despite being linear regression, non linear transformations are also mentioned because of the high bias issues.
It's quite straightforward to understand the meaning of a linear regression model with multiple predictors, e.g.:
Y = β0 + β1X1 + β2X2 + β3X3

So as X1 increases by one unit, predicted response Y will increase by β1 units (assuming X2 remains constant), same for the rest of predictors.
Also for regression with interaction terms:
Y = β0 + β1X1 + β2X2 + β3X1 X2

Which in this case indicates there's a synergy between X1 and X2 predictors, removing the additive assumption.
I'm also getting familiar with F-statistic, t-value or p-value for assessing the statistical significance of the predictors and decide whether or not there's an actual relationship with the predicted variable.
In the applied exercises section for this chapter, one of the exercises used with a dataset with several values, contains the following question:

Try a few different transformations of the variables, such as log(X), sqrt(X), X^2. Comment on your findings.

How should one try nonlinear transformation of variables following an intuition? It's easy to get a sense of this for simple regression with a single predictor (just by plotting the dataset), but what's the technique here without just randomly try stuff, considering that there are several predictors to try them all?
Moreover, given the following hypothetical model:
Y = β0 + β1X1 + β2X2 + β3X3 + β4X1^2 + β5log(X2)

That is, a predictor having both a linear and non linear relationship with the response? What can be a real-world example to understand this? As said, this would be very easy with a single predictor, but I don't find an intuition with multivariate regression.


texts: Massive variation in results with tensorflow and keras

texts:
I'm new to Tensorflow and Keras and I some background knowledge of how CNN's work.
I'm using a basic sequential model based on the code by https://pythonprogramming.net/convolutional-neural-network-deep-learning-python-tensorflow-keras/

I have a problem where my results variation is very big. The first time I ran the model today I got around 90% accuracy. But the runs after that were around 25% which is as good as guessing since I have four classes.

Here's my code:

tf.reset_default_graph()

batch_size = 32

logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

X = X/255.0

model = Sequential()

model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))

model.add(Dense(4))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
          optimizer='adam',
          metrics=['accuracy'])

with tf.Session() as sess:
model.fit(train_X, train_y, batch_size=batch_size, epochs=3,
validation_split=0.1, callbacks=[tensorboard_callback], validation_data=(test_X, test_y))

Am I doing something completely wrong with the model? I do have quite a small dataset, just 1640 images. But why do some runs perform so good then?


texts: Data Selection according to Feature Values

texts:
I am given a dataset consisting of 10 million molecules. Each row contains:

The average value predicted by an ensemble of regression models trained to predict a
certain property about chemicals (value of interest)
The ensemble variance of the predicted value
A heuristic value that characterizes the molecule with respect to some property (value 1)
Another heuristic value that characterizes the molecule with respect to some property
(value 2)

and the requirements:

High values for the value of interest are desired
High values for heuristic value 1 are desired
Low values for heuristic value 2 are desired

Design a protocol that filters and selects 1000 chemicals from the full dataset, and incorporates the mean value of interest, variance of the value of interest, heuristic value 1, and heuristic value 2. Comment on your rationale for the protocol. How would my protocol be adjusted if there was a constraint of diversity in the selected molecules (ie: selecting for chemicals that are very different from one another in the final set of 1000).
EDIT:
I am thinking about PCA if I select only 1 prior component. But is it possible to control that the weights of the features that are desired if higher are positive, while those that are desired if lower negative?


texts: How do I improve the accuracy of this classifier?

texts:
I have a dataset of 20000 x 3072 images for a homework assignment. This is just the training set, and the images are 32x32 and can depict one of four labels/classes, namely cars, trucks, boats and planes. There is a separate data set of 4000 x 3072 images of which only the input is provided and the actual class hidden. We are allowed only sklearn and python.

My approach so far, since I'm quite new to the field, apart from using the classifier we were told to try decision trees, nearest neighbors and a linear SGDClassifier, has been to try grid searches varying a few parameters(not many) for these classifiers and also random searches. Now I moved on to ensemble methods, and seeing that I tried random forest classifier, my accuracy is up to 66%. However, for these ensemble methods I provide no hyper parameters since I don't understand how I would try different values, or where to start really and I guess that is my question.

How do I go about thinking of things to try and improve the classifier apart from just trying different classifiers? Ideally I would like some guidance but no direct answers really, maybe a starting point and some tips since it is a homework assignment and I would like to learn.

Thank you!


texts: Confusion with using different classes in neural networks (training vs testing)

texts: I am new to deep learning and I am confused about having a neural network trained on certain classes and tested on different ones. Suppose I want to have a convolutional neural network that learns authorship attribution (Identifying the author of a certain text). For example, this model is trained on 10,000 authors/classes over hundreds of thousands of texts. How does the model generalize what it learnt to predict the author of a new author/class in a test set? Will it not be trained to identify the authors it was initially trained on and not the new one? For example, a CNN that learns whether an image is a cat or a dog is trained as such and when testing it predicts the same classes.


texts: How to get p-value and confident interval in LogisticRegression with sklearn?

texts:
I am building a multinomial logistic regression with sklearn (LogisticRegression). But after it finishes, how can I get a p-value and confident interval of my model? It only appears that sklearn only provides coefficient and intercept.

Thank you a lot.


texts:
I started dabbling in neural networks quite recently and encountered a situation which is quite strange (at least with my limited knowledge).

The problem I'm using a NN is a regression problem which tried to predict the sales of a product for a particular kind of promotion in an FMCG.

Although the data is not strictly time series, it still has some time related attributes like a prediction may depend on a similar kind of promotion last year (which I've modelled using feature engineering).

Now to the problem I'm having:

I took the data from 2015-2017,augumented it by adding small amount of noise, shuffled it, and ran it through a neural network(I think the architecture is not important, but let me know if it is, I'll try and post it).

Optimizer: Adam with a decay
Loss Function : Volume weighted mape
Validation set : Randomly selected 20% of the data

The network trained well and gave me errors as small as 8%. And both training and validation set errors decreased together and did not indicate an overfit.

But the kicker is that when I applied the algorithm to new unseen data(2018, first half), the errors rose to 55%. .

I tried researching the issue on Google but didn't find anything useful. What is happening here? Am I doing something wrong?


texts: Training and testing AdaBoost for low probability classification

texts:
I have a dataset that I want to classify as fraud/not fraud and I have many weak learners. My concern is that there is much more fraud than not fraud, so my weak learners perform better than average, but none perform better than 50% accuracy in the complete set.

My question is whether I should set up testing and training sets that are half fraud and half not fraud or if I should just use a representative sample.


texts: Why doesn't this CNN model need fetures for reducing overfitting?

texts:
I found this CNN model by Nvidia end-to-end-deeplearning and with training this model, I'm wondering why this model doesn't need to have dropout layers to reduce overfitting. Neither, this doesn't have activation function.

I know we can tune the number of epochs and it reduces overfitting.
I'm curious why this model works better without those layers?


texts:
I would like to ask what is the goal of fine-tuning a VGGNet on my dataset.
What does fine-tuning mean? Does it mean to change the weights or keep the weight values?


texts: How do I get one overall prediction, where each data point has many pictures?

texts:
My task is not a simple image -> category. I have between 5 and 10 images of an object, and I must classify it. The problem is that the category isn't "visible" in each picture, so I would need many images -> category.
For instance, let's say I'm trying to classify men vs. women. I have a number of pictures taken from random angles. I need to determine, based on the aggregated score of these pictures, which category the bunch belongs to.
All the pictures contain a little bit of unique information.


texts:
in the tutorial of Boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I wonder what does the init_score mean? In the help page, it says, "init_score: initial score is the base prediction lightgbm will boost from ;". Another thing is what does boost mean in lightGBM?

Pardon me if I asked something not precisely. I am a newbie to lightGBM & machine learning seeking for help eagerly.
Thanks.


texts: Connect output node to next hidden node in RNN

texts:
I'm trying to build a neural network with an unconventional architecture and a having trouble figuring out how.  Usually we have connections like so, where $X=$ input, $H=$ hidden layer, $Y=$ output layer:

$X_t \rightarrow H_t \rightarrow Y_t$

and

$H_t \rightarrow H_{t+1}$

Normal Tensorflow or Keras nodes build above, where we end up with weight matrices connecting each component, i.e. $W_{xh}, W_{hh}, W_{hy}$.

I would also like to introduce a connection:

$Y_t \rightarrow H_{t+1}$, defined by a new weight matrix $W_{yh}$.

I have looked at the Tensorflow and Keras implementations and there is a self.kernel ($W_{xh}$) and self.recurrent_kernel ($W_{hh}$), where the second is defined in terms of the previous hidden node's output prev_output = states[0], defined here. Does anyone have a sense of how to access the output of the output layer in the subsequent step?  Is it states[1]?  How are states defined, and where is that documented?

Thanks for any insight you can give!

Edit: Before anyone asks yes, I recognize that such a connection is not strictly necessary. I am trying to understand information flow through network nodes.


texts: If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show?

texts: If in t-SNE digaram of binary classification both classes follow the similar curve  what does  t-SNE diagram show for instance: Figure1 or  Figure2


texts: Get how similarity between the training data and the income data?

texts:
I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections

I want something like a semi-supervised model. I want to predict some class labels, however, I cant training my model with some anomalies/new behaviors that are going to happen eventually.

I'm working with motors vibration, I'll train my model with the vibration when they are "ON" and the vibration when they are "OFF".

With the time the vibration of the "ON" state will start to become slightly different because of some defects, however if I just train with "ON" and "OFF" for sure it'll continue to identify that my motor is "ON" because the vibration with defects are more close to the model "ON".

Therefore I would like to identify how far is my new data from the data that I have already trained my model.

For example, 1 - My predict_prob returns ON because has $98$% of chance to be ON and $2$% to be OFF. So far so good, however my DATA is slightly different from the ON mode in the training data, so I would like to measure this difference, like it has $98$% probability to be ON between ON and OFF and it is $95$% similar to the original model, so with the time this number ll be reduced. For example:

1 day: $98$% to be ON $2$% to be OFF and $95$% similar to the training data

2 day: $98$% to be ON $2$% to be OFF and $93$% similar to the training data

3 day: $99$% to be ON $1$% to be OFF and $90$% similar to the training data

4 day: $95$% to be ON $5$% to be OFF and $85$% similar to the training data

Can someone help me with this part "and $85$% similar to the training data"?


texts: How to combine nlp and numeric data for a linear regression problem

texts:
I'm very new to data science (this is my hello world project), and I have a data set made up of a combination of review text and numerical data such as number of tables. There is also a column for reviews which is a float (avg of all user reviews for that restaurant). So a row of data could be like:

{
    rating: 3.765,
    review: `Food was great, staff was friendly`,
    tables: 30,
    staff: 15,
    parking: 20
    ...
}

So following tutorials, I have been able to do the following:

Created a linear regression model to predict rating with the inputs being all the numerical data columns.
Created a regression model to predict rating based on review text using sklearn.TfidfVectorizer.

But now I'd like to combine models or combine the data from both into one to create a linear regression model. So how can I utilize the vectorized text data in my linear regression model?


texts: Tool For Multi-Label Image Classification

texts:
I am currently working on a project that requires multi-label image classification. The best way to achieve this seems to be through Binary Relevance.

I was intending to use a convolutional neural net, and I've looked into scikit-multilearn on top of Keras. However, I don't see a way as to how I can load image data as a dataset into scikit-multilearn, only ARFF formatted data. Is this possible, or should I manually create the binary classifiers in Keras myself?

And if so, can Binary Relevance in Keras be implemented in parallel, or only sequentially? How does scikit-multilearn implement it?


texts: Which input to use when generating a new sequence

texts:
I want to use sequence-to-sequence architecture to generate sequences.

My data has such structure

[0, 0, 1, 0, ..., 0, 1] --> [12.34, 0.78, 1.54, 6.90, ..., 5.32]

I follow this tutorial to achieve it.

After forwarding through Encoder network encoder_hidden is used as a decoder_hidden. But what should I use as a first decoder_input to Decoder network?

The original tutorial uses a Start Of the Sequence token, but I can't use it because it is encoded as 0. Probably 0 as a number will give some additional information for decoder.


texts: Downloading a large dataset on the web directly into AWS S3

texts:
Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?

Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I just want to supply the download URL to S3 and wait for them to download it to their filesystem. It seems like an easy thing to do, but I just can't find the documentation on it.


texts: What does a blue empty cell mean [ ] in Anaconda if the cell above is still [*]?

texts:
I've been running a machine learning algorithm for two days now, I have a large dataset and a slow machine.
I noticed today that the cell below the "fit model" cell turned from green to blue, however, the cell above (the one with fit) is still running [*]


texts: Clustering algorithm prior to model building?

texts: I would like to understand, how a clustering algorithm can be used (if possible) to identify naturally occurring groups within a data set, prior to building predictive models/model, and to hence improve accuracy of models/model


texts: Machine Learning with sometimes missing data

texts:
I'm trying to do an indoor locationing system based on my RSSI signal on my routers, I'm sniffing my network so I know what's the RSSI of my phone related to my routers antennas (I have 5 antennas all over my house).

The iPhone is not always broadcasting probe requests so I only get know in-real time the RSSI signal related to the router the device is connected to and no the other routers. To make it simple to understand: I sometimes have the RSSI of a device related to the 5 routers and sometimes only to 3 and in the worst case escenario only to 1 so my data sometimes looks like this:

room: 1, device_id: 1, rssi1: -80, rssi2: unknown, rssi3: -55, rssi4: unknown, rssi5: unknown

or:

room: 1, device_id: 1, rssi1: -80, rssi2: -95, rssi3: -55, rssi4: -102, rssi5: -96

or:

room: 1, device_id: 1, rssi1: -80, rssi2: unknown, rssi3: unknown, rssi4: unknown, rssi5: unknown

It is room based, so I'm not triangulating. I am trying to forecast in which room I am inside the house just by looking on the behaviour of the signals.

Machine Learning would be perfect, but if the data was always there and not some values missing sometimes. What kind of data algorithm should I use for this scenarios when sometimes I have all the data and sometimes just a few.


texts: Can LSTM Predict The Next Few Days Of Stock Price?

texts:
I have searched many websites and forums describing stock price forecast using LSTM.

They shared two things in common: one is that all the sources make predictions with same set of data and none of these websites show a prediction for next 10 days.

So I would like to know whether it's possible to predict stock prices for the next few days (n days) using LSTM.
And why do most sources use the same dataset?


texts: Which approach to select category based on keywords

texts:
I want to assign a certain category to a group of keywords.
So i.e. people can upload images or videos, when they do this they can set keywords for this.
These keywords are free to type so words can be spelled in different ways.
The amount of keywords are 95% between 0 and 20 words.

I want to create categories from these. So that I can assign a combination of keywords to a category.

The categories and the amount of categories are undefined.

From what I've researched is that this is probably a Topic modeling or clustering issue. Although with topic modeling most examples I see are based on long texts instead of a couple of keywords.

What would be a good approach to handle this?

I thought about first some simple fuzzywuzzy to find different spellings of the same words.

Create a big word list from this. Then each keyword will be matched against the list and rewritten if it matches one and added if there isn't a good match.

Then I would need to create groups and here I don't know which algorithms I should use.

I was thinking maybe do k-means clustering and then see on which k I get the best results and then assign a category to it manually by looking at which keywords are in it.

So it would be nice to just let the algo figure the amount and categories out, but it can be relaxed that I will set them before.

Does anyone have a better suggestion or are there just complete algorithms for this already available?


texts: Oversampling before Cross-Validation, is it a problem?

texts:
I have a multi-class classification problem to solve which is highly imbalanced. Obviously I'm doing oversampling, but I'm doing cross-validation with the over-sampled dataset, as a result of which I should be having repetition of data in the train as well as validation set. I'm using lightgbm algorithm, but surprisingly there is not much difference between cross-validation score and the score on the unseen dataset.

However I just want to know whether its fine to do cross-validation after oversampling the dataset, if not why am I getting such close score on the validation set and the unseen test set?

Also if its not correct to do oversampling before the cross-validation, then it becomes to lengthy to split the data into validation and training and then again sample the training set, and again during final prediction if you're looking to use all the data then you've to append the validation and the training data and then again oversample. Is there any shortcut method to solve the problem?


texts:
The meaning of multi-class classification rules

Example: I have two classification rules (Refund is a predictor and Cheat is a binary response):
(Refund, No) → (Cheat, No) Support = 0.4, Confidence = 0.57
(Refund, No) → (Cheat, Yes) Support = 0.3, Confidence = 0.43

=> multi-class classification rules:

(Refund, No) → (Cheat, No) v (Cheat, Yes)

When predicted classification for test data, (Cheat, No) will be selected priority so why we need to have (Cheat, Yes) in multi-class classification rules here?


texts: Section/Topic segmentation in HTML and plaintext documents

texts:
GROBID (https://github.com/kermitt2/grobid) is terrific for fine-grained (section, chapter) segmentation of PDF files, but I need to do the equivalent for HTML and plaintext files.

I've tried TextTiling but it seems to work only for coarse-grained topic change detection (like distinguishing between newspaper articles regarding very different subjects).

What are methods and/or SW toolkits that I could use to segment a document into sections regarding particular fine-grained topics?


texts: Gradient descent parameter estimation Package for R

texts:
I am looking for a package that does gradient descent parameter estimation in R, maybe with some bootstrapping to get confidence intervals. I wonder if people call it something different here as I get almost nothing on my searches, and the one article I found was from someone who rolled their own.

It is not that hard to implement, but I would prefer to use something standard.


texts: Infoviz for multinomials: miniature plots within plots in R's ggplot

texts:
I'm plotting fluctuations in a trinomial distribution across two dimensions.

colorspace_plot <- function(zz) {
    zz$z2 <- zz$x * zz$y
 zz$z3 <- zz$x * (1-zz$y)
    zz <- cbind(zz, t(apply(zz[,c("z1","z2","z3")], 1, function(x) x/sum(x)) ))
    setnames(zz, c("x", "y", "z1", "z2", "z3", "p1", "p2", "p3"))
    zz$pcolor <- with(zz, rgb(p1, p2, p3, 1))
 pcolors <- zz$pcolor
    names(pcolors) <- pcolors
    return( ggplot(zz, aes(x,y)) + geom_bin2d(aes(fill=pcolor), binwidth=1/11) +  scale_fill_manual(values = pcolors ) + guides(fill="none") )
}

clear <- data.frame(x=rep(seq(from=0, to=1,length.out=11),times=11), y=rep(seq(from=0, to=1,length.out=11),each=11), z1=rep(1/3,11*11))
colorspace_plot(clear)

I don't know what its called, but this kind of plot has been used for this kind of thing before (Figure 4 here):

However, my actual data doesn't have the same clean separation of values as the ideal example. It's dominated by one of the three categories, so it comes out varying between clean and muddy variations of just one color. It doesn't communicate.

muddy <- data.frame(x=rep(seq(from=0.6, to=1,length.out=11),times=11), y=rep(seq(from=0.6, to=1,length.out=11),each=11), z1=rep(1/3,11*11))
colorspace_plot(muddy)

While this plot still works, it doesn't "work".

My question: is there another way to represent trinomial data within-bin? For example: can I insert into each bin little pie charts or histograms, or something?  A solution that stays within R and ggplot (and even with geom_bin2d()—I think I could figure this out with facet_grid()) would be ideal.

One promising idea that came up was to represent the distribution in each bin by filling each bin with a 10x10 grid of appropriately colored dots: a pointillist approach to my muddy plots.  But I have no idea how to pull that off, or if it would even look good. Things I'll ask you to disregard:

rescaling the green or the mapping so that the other percentages come off larger than they really are.
breaking this one panel into three panels, one for each category.


texts:
I have this error when running training on my model.
I found this issue on different sites, but could not find a solution to my problem.
Here is my model :
import keras
import tensorflow as tf
import tensorflow.keras.layers as L
import tensorflow.keras.models as M
import tensorflow.keras.callbacks as C
import tensorflow.keras.utils as U

def make_model_lstm_pooling(inshape=50000):
    z = L.Input(shape=(inshape, 10))
    x = L.AveragePooling1D(pool_size=1, strides=100)(z)

    x = L.Bidirectional(
        L.LSTM(10,
            dropout=0.1,
            return_sequences=False,
            kernel_initializer='ones',
            bias_initializer='zeros')
    )(x)

    x = L.Dense(10, activation='linear')(x)
    x = L.Dense(1, activation='linear')(x)

    model = tf.keras.Model(z, x)
    model.compile(optimizer='adam')
    return model

I run the training then :
callback_lr = C.ReduceLROnPlateau(
                monitor='val_loss',
                patience=3,
                verbose=0,
                mode='min')

checkpoint = C.ModelCheckpoint(
                filepath='best_pool.h5',
                save_best_only=True,
                monitor='val_loss',
                mode='min')

model = make_model_lstm_pooling()
model.summary()
history = model.fit(
            X_train, Y_train,
            validation_data=(X_dev, Y_dev),
            epochs=100,
            callbacks=[checkpoint, callback_lr]
                   )

The whole error is this one :

ValueError: No gradients provided for any variable:
['bidirectional_16/forward_lstm_50/lstm_cell_83/kernel:0',
'bidirectional_16/forward_lstm_50/lstm_cell_83/recurrent_kernel:0',
'bidirectional_16/forward_lstm_50/lstm_cell_83/bias:0',
'bidirectional_16/backward_lstm_50/lstm_cell_84/kernel:0',
'bidirectional_16/backward_lstm_50/lstm_cell_84/recurrent_kernel:0',
'bidirectional_16/backward_lstm_50/lstm_cell_84/bias:0',
'dense_90/kernel:0', 'dense_90/bias:0',  'dense_91/kernel:0',
'dense_91/bias:0'].

I get the problem when running fit.
I saw that the problem can appear when using bad types : I have float type in the input and int in the labels. I have NO nan in the input.
I see in the error there is a problem related to the kernel initializer, the default is glorot_uniform, it appears to me it is not zeros, if I am not mistaken.
I tried to change kernel_initializer but did not improve.
Something else : I made a test on several samples, and in my test I have less samples then features. Have anyone any idea if the problem is related to this ?
Any help will be appreciated.


texts: Data-generating probability distribution, probability distribution of a dataset, in ML

texts:
In Goodfellow I, Bengio Y, Courville A. Deep learning. MIT press; 2016 Nov 10.
http://thuvien.thanglong.edu.vn:8081/dspace/bitstream/DHTL_123456789/4227/1/10.4-1.pdf

p. 102 (for example), it is said that with Unsupervised Learning, one usually wants to ''learn the entire probability distribution that generated a dataset'', $p(\vec{x})$.

My question is that I would like to have a better interpretation/understanding of the concept, i.e., can we say that $p(\vec{x})$, for a given an example (represented as a vector) $\vec{x}$, in the dataset, means the ''probability to find this example à priori'' in the wild... or something like that? E.g. if it is an image of a cat of breed X, e.g., $\vec{x_i}$ = ''vectorized image of this cat of breed X'' does $p(\vec{x_i})$ means the probability (we estimate given our limited dataset of course) to obtain this image of this cat of breed X, if we draw a sample from this dataset - and even if we want to generalize - the prob to obtain this cat if we sample the TEST SET. ?

There is a similar question here, but the answer is far from answering anything about the question:
What does it mean for the training data to be generated by a probability distribution over datasets


texts: Which data sets would help to predict (exponential) market trends?

texts:
Which kind of datasets do websites such as MeetGlimpse, trends.co, explodingtopics.com use to detect exponential market trends? I love them (not affiliated) and would like to better understand how they work.
I would assume they somehow buy access to clickstream data or something similar? Any hints?
Also, which algorithms/approaches do they use?


texts: Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)

texts: Pandas has a method called get_dummies() that creates a dummy encoding of a categorical variable. Scikit-learn also has a OneHotEncoder that needs to be used along with a LabelEncoder. What are the pros/cons of using each of them? Also both yield dummy encoding (k dummy variables for k levels of a categorical variable) and not one-hot encoding (k-1 dummy variables), how can one get rid of the extra category? How much of a problem does this dummy encoding create in regression models (collinearity issues - a.k.a. dummy variable trap)?


texts: Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?

texts:
I have to build a recommender system & it will be evaluated using map@10 criteria.
I have rolled up the data/rows at user-item level & is using Gradient Boosting in scikit learn to build the model.

Challenge

But there is a big class imbalance where only 6%( out of 40 K use-item pairs) have actually purchased the item in the entire base. Given this challenge i want to know the accuracy metric I should use to tune the parameters of GBM model.

Question

I thought about using recall, precision, f1 score etc. to tune the hyperparameters of GBM. But which one will maximize map@10 given the class imbalance?

Thanks


texts: In Keras how to get the `class_indices` or prediction labels for an existing model

texts:
I know that keras provides a .class_indicies dictionary containing the mapping from class names to class as part of .flow_from_directory() from its ImageDataGenerator class  (https://keras.io/preprocessing/image/).

HOWEVER, is there a way to access the corresponding class labels from an existing saved model (a model saved in as an .h5 file)?  This seems important when putting my model into production and serving predictions since the classes are not known upfront and therefore the images are not separated in pre-labeled directories.

Similarly, are there any good examples of how a keras model should be deployed into production (especially for mobile)?  Thanks!


texts: Dealing with input images of different shapes in PyTorch

texts:
I've started to work with a leaf classification dataset on Kaggle. All input images have different rectangular shapes. I want to transform the input into squares of a fixed size (say, 224x224) with a symmetric zero-padding either on top and bottom or on the left and right sides of the rectangle. Prior to that, I think that I need to rescale the image (some images in the dataset have shapes >1000). I've encountered the next two problems:

torchvision.transforms.Resize(size) rescales the image so that its smaller side is matched to size (size is a scalar). I want the opposite, the bigger side to be matched to size.
torchvision.transforms.Pad(padding) seemingly works only with some fixed padding, but this transform will not always output a square.

How would you resolve this problem? I'm aware of RandomSizedCrop, but I feel like some datasets (like this one) aren't good for this method (some of the input images are too oblong). Also, I heard that RandomSizedCrop shouldn't be
used for the test data loaders.

More generally, how do you deal with input images of different rectangular shapes in PyTorch? Any help is appreciated.


texts: Python: How to test a RandomForest regression model for Overfitting?

texts:
I'm a beginner in this area so maybe I'm doing something wrong here.
I'm using RandomForest for a regression model and wanted to see if my model is overfitting. Here is what I did:
EDIT:
I use GridSearchCV for hyperparameter tuning:
rf = RandomForestRegressor(random_state=0)
rf_params = {'n_estimators': [100, 500, 1000], 'max_depth': [3, 6, 9, None],
             'min_samples_leaf': [2, 5, 10], 'max_features':['auto', 'sqrt']}

gs_rf = GridSearchCV(rf, rf_params, scoring = 'neg_mean_absolute_error', cv = 10)

gs_rf.fit(x_train, y_train.values.ravel())
b = gs_rf.best_params_

Then I create a RandomForestRegressor with those parameters:
RF = RandomForestRegressor(n_estimators=b['n_estimators'], max_depth=b['max_depth'], min_samples_leaf=b['min_samples_leaf'], max_features=b['max_features'], random_state=0)

Then I fit the model using the train dataset:
model = RF.fit(x_train, y_train.values.ravel())

Then I predict with the test dataset:
y_pred = model.predict(x_test)

Then I did the exact same with x_train instead of x_test:
y_pred = model.predict(x_train)

Here are the results that I achieve:
Test Data:
MAE: 15.11
MAPE: 26.98%

Train Data:
MAE: 6.17
MAPE: 10.97%

As you can see there is a pretty significant difference.
Do I have a big problem with overfitting or am I doing something wrong when using x_train to predict?
Any help is much appreciated!


texts: Keras weird loss and metrics during train

texts:
I am doing some testing with tensorflow, and I bumbed into a very weird behaviour.
Here is my code
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images1, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images1[:32] / 255.0
train_labels = train_labels[:32]
test_images = test_images / 255.0

batch_size = 32
epochs     = 1

train_data = tf.data.Dataset.from_tensor_slices((train_images, train_labels)).batch(batch_size)

adam = tf.keras.optimizers.Adam(lr=0.002, beta_1=0.9, beta_2=0.999)

input_layer   = tf.keras.layers.Input(shape=(28,28,))
flatter       = tf.keras.layers.Flatten()(input_layer)
dense1        = tf.keras.layers.Dense(128,
                                     kernel_regularizer=tf.keras.regularizers.l2(0.01),
                                     activation='relu')(flatter)
dense2        = tf.keras.layers.Dense(64,
                                     kernel_regularizer=tf.keras.regularizers.l2(0.01),
                                     activation='relu')(dense1)
output_layer = tf.keras.layers.Dense(10,
                                     activation='softmax',name='output')(dense2)
model_naive = tf.keras.models.Model(inputs=input_layer,outputs=output_layer)

model_naive.compile(optimizer=adam,
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])
model_naive.summary()
history = model_naive.fit(x=train_data, validation_data=train_data, epochs=epochs)
model_naive.evaluate(train_data)

I simply import fashon_mnist (only one batch) and pass it as both train and validation data, to make comparison. No dropout in the network... so I would expect to find same loss and metric but this is the output
1/1 [==============================] - 1s 815ms/step - loss: 5.4532 - accuracy: 0.0312 - val_loss: 5.0106 - val_accuracy: 0.3125

To be sure I even did a model.evaluate() and this is what I find
1/1 [==============================] - 0s 12ms/step - loss: 5.0106 - accuracy: 0.3125

exactly the same found during training.
So, provided the evaluation is correct...what are these numbers "loss: 5.4532 - accuracy: 0.0312" ? I am using only one batch, so I would expect no averages over batches are involved.
Please help me to understand, this is driving me crazy. Thank you!
EDIT:
With only 1 batch keras weirdly seems to print loss and score before applying gradients. The same does not happen with more than one batch where probably some strange average is performed.
Still not solved the issue, any comment is still very welcomed!


texts: Continue with LSTM or try other approaches?

texts:
I am trying to predict error (deviation from the actual behavior) in a signal, here is an example
Blue -> Reference/Actual signal (cannot be fed to the network, used to calculate error only)
Orange -> Signal that deviates

and a typical error signal with noise removed looks like this.

With LSTM and one lucky run and a lucky architecture I was able to get this result. I do not mind the noise but for some reason no matter the amount of data I'm using or the deeper the network I make, I don't seem to make progress and also not able to reproduce the same results.
Any suggestions?
Can a SVM be used to predict something so non-linear?


texts:
I have a dataset with 85k positive labels and 53k negative labels. For this use-case, I am trying to maximize my efforts to the negative class (accurately identify true negatives, and minimize false negatives).
Currently, I am able to train a xgboost classifier to 71% accuracy and my confusion matrix looks like this when benched against a test set
[ 3890 | 8887 ]
[  844 | 20044]

Again for this task, I'd really like to improve my recall and minimize false negatives. However, even in the models' current state, if I try to submit 43k new records for prediction (results are unknown), my model predicts that all 43k records are non-compliant.
Given this information, my two questions are as follows:

How irregular is this occurrence. I thought that skewing to the majority class was possible, but not the minority.
Are there any 'best practices' that can be applied to reduce the aggressiveness of my model? not only does it predict that all claims are negative (0), but it does so with very strong confidence (minimum 'probability of occurrence' is above 90%). Again, less than 40% of the records in my trainset are of the minority class, so I don't understand why it's skewing this hard.

Thanks


texts: Predictions receiving in Unknown (8)[e,e,e,e,e,e,e,e] format from TensorflowJS Mobilenet

texts:
I have trained a mobilenet on diamond images to count diamonds in broswer. I then converted the SavedModel format to TFJS format. I have the following code in my JS file.
async function initialize() {
try{
    model = await tf.loadGraphModel('trained-model/model.json');
}catch(e) {
   console.log("The model is not loaded")
}

classifierElement.style.display = 'block';
loaderElement.style.display = 'none';

document.getElementById('predict').addEventListener('click', () => predictOnClick());
}

async function predictOnClick () {

  const imageElement = document.getElementById('img');
  let tensorImg = tf.browser.fromPixels(imageElement,3).cast('int32').resizeNearestNeighbor([640, 640]);//.expandDims();
  const t4d=tensorImg.expandDims(0);
  t4d.print()
  data = await t4d.data()

  prediction = model.executeAsync(t4d).data().then(prediction=> {
  console.log("FULL STRUCTURE",prediction)})

I am getting the following in console output

The tensorflow model Non-Javascript version returns the following dict
['detection_multiclass_scores', 'detection_classes', 'detection_boxes', 'raw_detection_boxes', 'detection_scores', 'raw_detection_scores', 'detection_anchor_indices', 'num_detections'])

So how do I obtain the scores and detection boxes data from the TFJS model rather than (8)[e,e,e,e,e,e,e,e] format?
Thanks


texts: Help with replacing values in a data frame

texts:
I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine portion. The target is the quality of the wine, rated from 1-10. I would like to change the values in the column 'quality' such that if the value is from 1-4, it is replaced with 'low', if the value is from 5-6, the value is replaced with 'medium', and if the value is 8-10, it is replaced with 'high'. My code is as such:
#Read in data
white = pd.read_csv('Wine_data.csv')

#Rename columns to exlude the u
columns = ['fixed acidity', 'volatile acidity', 'citric acid','residual sugar', 'chlorides', 'free sulfur dioxide','total sulfur dioxide', 'density', 'pH', 'sulphates', 'alcohol', 'quality']
white.columns = columns

quality = np.array(white['quality'])

for i in range(len(quality)):
    if quality[i] in [1,2,3,4]:
        quality[i] = 'low'
    elif quality[i] in [5,6,7]:
        quality[i] = 'med'
    else:
        quality[i] = 'high'

However, I am getting this error:
Error in lines 13-19
Traceback (most recent call last):
  File "/cocalc/lib/python2.7/site-packages/smc_sagews/sage_server.py", line 1013, in execute
    exec compile(block+'\n', '', 'single') in namespace, locals
  File "", line 5, in <module>
ValueError: invalid literal for long() with base 10: 'med'

Here is a link to my code.
Help this nOOb please.


texts:
I am working in a small sales company. Started with making relatively easy, small, static reports in Excel, but then it was required to make a lot of highly-customizable and adaptable reports, that are really troublesome to do in Excel. I do a lot of VBA, but that still doesn't meet my needs. Our ERP has no BI solution. Excel can't keep up with amount of data used (millions of lines).

I am only accustomed to JS and HTML and have some experience working with
relational databases (MySQL, MariaDB in particular).

Basically, what I need help with is deciding what tools would be appropriate for my  needs considering I am not in a rush and would like to develop some skills along the way.

I would like to create a dashboard that would retrieve data from a local DB and process it. I would also need the dashboard to be dynamic - be able to change some inputs and run calculations accordingly. I am interested in packages that are suitable for mathematical analysis. Reports would include both tables and graphs.

I've read about R and Python packages, but still am not sure which one would be the most appropriate.

In the future I guess I would like to distribute it to other colleagues (run it from a server; have different accounts with different access level etc.).

Will highly appreciate any tips!


texts:
Suppose I have a dictionary:

{apple:large apple, apple:red apple, apple:aple, orange:mandarin, orange:orang, orange:blood orange}

and so on...

And then I want to replace a large document of entries with the keys. However, occasionally a new value will come up, i.e. {apple:green apple}

Is there a method where I can replace all values with the corresponding key, but then also replace 'close' values like the one given if they appear?

Example document:

var1
_____
aple
apple
orange
Apple
Red apple
gren Apple
blood Orange
orang

var1_replaced
______________
apple
apple
orange
apple
apple
apple
orange
orange


texts:
I'm working on keyword/phrase extraction from a single document. I started by doing term frequency analysis, but this returns words like "new" which aren't very helpful. So I want to penalize the common words and phrases, for which we normally use idf (inverse document frequency). But since it's for a single document, I'm not sure how to do idf analysis.

Is it possible to use tf-idf method with pre-calculated idf values for (all?) words?
And are such values available somewhere?


texts: Hands on Machine Learning with Scikit Learn and TensorFlow Confusion Matrix with VERY BAD score

texts:
I followed the steps EXACTLY in the Hands on Machine Learning with Scikit Learn and TensorFlow ch. 3. But the confusion matrix for the multinomial classifier is very very bad. Even though the book expects the matrix score to be very good.

The matrix looks like this:

But the matrix should ACTUALLY look like this:

Here is the link to the notebook: https://colab.research.google.com/drive/1x1vhgajdDYovFop59MoT5asQ9qyyP1ka

My question is, what mistake did I make in my steps? And should I adapt my program to future updates in scikit-learn, numpy, and pandas? (The book is stated as outdated in some parts)


texts: Recommendation system depend on user rating and favorite list

texts:
In my project I have a database of Japanese Sake(rice wine).Each Sake has following attributes which has direct link to the taste of the Sake:

classification (enumeration 1-5 integer)
alcohol percentage (1-100 integer)
polish percentage (1-100 integer)
acidity (double)

Users of the system can :

Rate Sake (1-5 double value)
Add Sakes to his/her favorite list.

Depend on these user data, I am thinking of developing a  recommendation system for sake.
I am looking at Collaborating Filtering methods,but guess it doesn't fit this scenario?

What kind of algorithm/techniques can be used for this kind of user recommendations?

[Edit] I am looking for a recommendation algorithm not only considering user rating,but also the taste attributes.


texts: Different extraction pipeline for train and test

texts:
I'm trying to create a production-ready ML model.
The problem is as follows:

Training data: Database A + Plus python aggregations.

Testing data: Database B + Plus python aggregations.

Both databases have different attributes and context-based aggregations. Overall, however the data fetched after transformations is in the same format.

Is it ideal to build two different extraction pipelines in this case? Or, is there a better solution for this problem.


texts: Are there any good NLP APIs for comparing strings in terms of semantic similarity?

texts:
I want to create a chatbot which informs the user about traffic at the streets but not in real-time for the moment. I have created a small database with MySQL which has some data stored regarding traffic and I fetch them with a PHP script whenever this is appropriate depending on the interaction of the user with the chatbot.

I wonder how to deal with the case when the user asks variations of the same question which therefore can be answered with the same answer.
For example:

Why is there traffic at High Street?
What is the cause of traffic at High Street?
Why did I encounter traffic at High Street?
I am stuck in traffic at High Street. Why is this?

Obviously, I can start by removing stopwords (e.g. did), by naming entities (e.g. road -> High Street), by defining synonyms and by applying a text similarity measure (e.g. Levenshtein distance etc).

However, I feel like reinventing the wheel if I do this. Therefore, my question is:

Are there any APIs which can compare strings in terms of semantic similarity (without even requiring training)?

I know that there software platforms such as Dialogflow which are suitable for these tasks but still you must explicitly state all the variations of the same question so that you will get the same answer. Therefore, I look for a API where you will explicitly state only one of these variations of the same question (e.g. Why is there traffic at High Street?) and then the API will figure out by itself which other variations are identical to it in terms of meaning or not.


texts: Looking for algebras designed to transform time series

texts:
I am looking for information on (formal) algebraic systems that can be used to transform time-series - in either a practical or academic context.

I hope that there exists (at least one) small, expressive, set of operators - ranging over (finite) time-series.  I want to compare and contrast different systems  with respect to algebraic completeness, and brevity of representation, of common time-series transformations in various domains.

I realise this question is broad - but hope it is not too vague for datascience.stackexchange.  I welcome any pointers to relevant literature for specific scenarios, or the general subject.

Edit... (Attempt to better explain what I meant by an algebraic system...)

I was thinking about "abstract algebras" as discussed in Wikipedia:

http://en.wikipedia.org/wiki/Algebra#Abstract_algebra
http://en.wikipedia.org/wiki/Abstract_algebra#Basic_concepts

Boolean Algebras are (very simple) algebras that range over Boolean values.  A simple example of such an algebra would consist the values True and False and the operations AND, OR and NOT. One might argue this algebra is 'complete' as, from these two constants (free-variables) and three basic operations, arbitrary boolean functions can be constructed/described.

I am interested to discover algebras where the values are (time-domain) time-series.  I'd like it to be possible to construct "arbitrary" functions, that map time-series to time-series, from a few operations which, individually, map time-series to time-series.  I am open to liberal interpretations of "arbitrary". I would be especially interested in examples of these algebras where the operations consist 'higher-order functions' - where such operations have been developed for a specific domain.


texts: Linear regression : ValueError: operands could not be broadcast together with shapes (3,) (1338,)

texts:
I try to use linear regression for insurance data . But had error on the when try to call a function with features parameter.
Here is my code:

def h(x):
    global w
    return np.sum(np.transpose(w)*x)
    raise NotImplementedError()

when try with a simple data it works fine,

w, x = [1,2,3], [2,3,4]
h(x)

the output is : 20

but when try to use the dataset, it errors:

features = dataset.drop(["charges"], axis=1).values
h(features )

it returns error:

ValueError: operands could not be broadcast together with shapes (3,) (1338,)

so the features looks like this:

array([[0.1173913 , 0.        , 0.35698144, 0.        , 1.        ],
       [0.1       , 1.        , 0.48331988, 1.        , 0.        ],
       [0.27391304, 1.        , 0.46674738, 3.        , 0.        ],
       ...,
       [0.1       , 0.        , 0.5496099 , 0.        , 0.        ],
       [0.15217391, 0.        , 0.3117837 , 0.        , 0.        ],
       [0.84782609, 0.        , 0.38216303, 0.        , 1.        ]])

The data i used is insurance.csv from kaggle.com


texts:
I have a problem with labelling this bar graph. The 'blue' bar that indicates ML has 5 ml models and I'm trying to rename each of the 5 blue bars given into 5 ml model names obtained from the 'index' variable. Can you please help me code this part ?

[code]

import matplotlib
import matplotlib.pyplot as plt
import numpy as np

year = [2014, 2015, 2016, 2017, 2018]
index=[test_2014ing1['RMSE'].index[0],test_2015ing1['RMSE'].index[0],test_2016ing1['RMSE'].index[0],test_2017ing1['RMSE'].index[0],test_2018ing1['RMSE'].index[0]]
ML = [test_2014ing1['RMSE'][0],test_2015ing1['RMSE'][0],test_2016ing1['RMSE'][0],test_2017ing1['RMSE'][0],test_2018ing1['RMSE'][0]]
CAPM = [test_2014ing1['RMSE'][1],test_2015ing1['RMSE'][1],test_2016ing1['RMSE'][1],test_2017ing1['RMSE'][1],test_2018ing1['RMSE'][1]]
Market = [test_2014ing1['RMSE'][2],test_2015ing1['RMSE'][2],test_2016ing1['RMSE'][2],test_2017ing1['RMSE'][2],test_2018ing1['RMSE'][2]]
TimeSeries = [test_2014ing1['RMSE'][3],test_2015ing1['RMSE'][3],test_2016ing1['RMSE'][3],test_2017ing1['RMSE'][3],test_2018ing1['RMSE'][3]]

x = np.arange(len(year))  # the label locations
width = 0.35  # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(x - width/2, ML, width/4, label='ML')
rects2 = ax.bar(x - width/4, CAPM, width/4, label='CAPM')
rects3 = ax.bar(x, Market, width/4, label='Market')
rects4 = ax.bar(x + width/4, TimeSeries, width/4, label='TimeSeries')
# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_ylabel('RMSE')
ax.set_title('Test RMSE across Years: International Banks')
ax.set_xticks(x)
ax.set_xticklabels(year)
ax.legend()
fig.tight_layout()
plt.show()

for i in range(0,2): # your number of bars
    plt.text(x=year[i], #takes your x values as horizontal positioning argument
    y = ML[i], #takes your y values as vertical positioning argument
    s = index[i], # the labels you want to add to the data
    size = 4) # font size of datalabels


texts:
I am working on a historic time series dataset and using RNN, LSTM, GRU models, and I didn't find an answer if in test time, the h (or h, c) weights should be zeors for each batch?
If the weights should be zero, what they should be? the last updated weights from the training?
Thanks


texts: How do I control for some patients providing multiple samples in my training data?

texts: I have a dataset with 50 patients. The patients are tracked over many years with a set of a few thousand features measured at somewhat random intervals. I am trying to predict a certain outcome (this is a regression problem) which is measured either once, twice, or three times per patient over the entire time period the patient is tracked. So far, I have been assuming that each outcome feature is a data point in my Y vector, and using the closest time point where the features are measured for the patient. I want my final model to be able to predict the outcome variable from only one measure of features, which is why I am OK ignoring the temporal aspect of the data. However, I am not sure how to control for the fact that some patients contribute 2 or even 3 samples to my training data. Wouldn't this affect the performance of my model since there is some in-patient variation? How do I account for this?


texts: SGDClassifier - Why do I need to use argmax instead of argmin to find the lowest threshold satisfying given precision?

texts:
I am an experienced programmer, but new to Python and data science. I am following Aurelien Gerone's book and I don't understand one thing.
I create SGDClassifier and calculate its precision_recall_curve(). Then I am trying to find the lowest threshold to satisfy precision equal to 90%:
precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)
threshold_90_precision = thresholds[np.argmax(precisions >= 0.90)]

Why on earth I am searching for argmax if I need to find the minimum threshold value? If I try to use argmin I get the wrong value, with precision equal to 0.1.
As I understand this:

precisions >= 0.90 creates an array with precision scores only above or equal to 0.90,
argmax returns an index, at which I find the highest value in the given array (so this should be as far from 90% as possible, but it's not!),
then I choose a threshold with returned index.

What am I missing?


texts: How to train a spacy model for text classification?

texts:
Can i know the way or steps to train a spacy model for text classification. (binary classification in my case)

Please help me with the process and way to approach.


texts: How to represent a dataset as a linked list-like graph?

texts: I would like to visualize this data set using Python and probably a visualization tool like Matplotlib. The data set contains three columns: a user id with a question, a user id with an answer, and time. I would like to visualize this data set as a linked list style graph. That is a tree-like structure which would show relationships based on the connections between questions and answers. I've tried different sets of keywords in an attempt to find out how to visualize a data set in this way, but to no avail. [I'm new to data science or analysis]. So I would appreciate some help with this.