texts: Best way to add border to rectangular images to train on StyleGAN2

texts:
TLDR: What is a best way to add borders to rectangular images with widely variable aspect ratios (from 0.5 to 2.0) to later use them as training data in StyleGAN2 with data resolution of 1024x1024 and preserve the benefit of mirror augment

So I have a custom dataset consisting of ~70k images of variable resolutions and aspect ratios of the images ranging from 0.5 to 2.0. I want to use this dataset to train a StyleGAN2 model but it requires a square images (and even if it didn't, my images are still of different aspect ratios). One of the most common solutions is to just crop the images down to squares but I cant use this approach since >90% of the image area contains an important information and cropping it out will render the whole process useless.

As an example consider a following image of a street map (not my actual data, but illustrates the problem):

It has an aspect W/H ratio of about 0.5, and suppose all of the pixels contain a useful information so we don't want to crop the image.

Now the possible options to turn this image of a street map to a square are:

Adding a constant border:

Now the image is a square but contains a hard boundary between the content and the border which is probably not ideal at all.

Adding a replicated border (I am using OpenCV's BORDER_REPLICATE):

Probably a better solution than a constant border but the lines from pixel replication will probably make the NN training more complicated since now a large portion of images contains an alien lines which are basically noise compared to the image data itself

Adding a reflected border (BORDER_REFLECT)

Now the border contains basically identical information to the image, but now it seems like doing a mirror augmentation in StyleGAN2 will produce less value since now images contain a reflection on their own.

I am also considering shifting the image to the side so that the border is only added to the right, but it will probably cause the same issue with mirror augmenting, but I am not sure

Warping an image at the border (BORDER_WRAP):

May also be a good option, but again it contains a hard boundary (like in option 1 with constant border) between the image and the data. Another thing (which applies to the reflected borders too) is that the extra images placed as borders will in most cases be cropped (like if image occupies 70% of a square and we fill remaining 30% with reflections, they will not be full and will only partially reflect/wrap the original image).

Resizing the image:

One of the last options I was thinking of. The issue is that I will have to then manually find the aspect ratios for the output images from the StyleGAN2 which I am sure will be a pain, not to mention how such data manipulation will affect the training process (don't think it will help to be honest)

As a result

I am asking for any help in this question of what is a best way to turn rectangular image to a square ones. I understand the there will brobably won't be the best approach but I am looking for some insight on what is a better way to do this. I would be happy to try them all on my own, but training the StyleGAN2 is a lengthy process so I don't really want to waste a couple of attempts for bad results, so I would be really thankful if someone can provide some information on this.


texts:
I have used an LSTM with 4 layers deep each layer having 10 LSTM units to predict the AAPL stock 500 steps away by looking 50 steps back and it was predicting well (only a lag was there).
However when I try to predict the difference (future stock value- current stock value), I am getting an almost flat curve.

In this plot, the two (full) verticle lines are the validation split and train-test splits. The blue curve is the true value and orange curve is the predicted value.
Now, as you can see, the prediction in the validation and test is always constant.
Then I thought may be this is because the model is not having enough flexibility and increased number of LSTMs in each layer, but that did not help.
Further I tried to map the differences to bins and the results was the same (see the image below)

However in the otherhand, an MLP with few financial features (4 layers deep with each layer having 10 neurons) was doing a better job.
In here I have used 1 minute data and there was around 3.5million datapoints in the full time-series.
Any idea why is this happening?


texts: Working with more than one dummy variable in neural net

texts:
I am building a neural net and am at the point of working with categorical variables.   Below you can see that I transformed the categorical variables into numeric variables.

#Encode "UserName"
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Encode "Token"
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])
#Encode "ChildEXE"
labelencoder_X_3 = LabelEncoder()
X[:, 3] = labelencoder_X_3.fit_transform(X[:, 3])
#Encode "ParentEXE"
labelencoder_X_4 = LabelEncoder()
X[:, 4] = labelencoder_X_4.fit_transform(X[:, 4])
#Encode "ChildFilePath"
labelencoder_X_5 = LabelEncoder()
X[:, 5] = labelencoder_X_5.fit_transform(X[:, 5])
#Encode "ParentFilePath"
labelencoder_X_6 = LabelEncoder()
X[:, 6] = labelencoder_X_6.fit_transform(X[:, 6])

Now I am supposed to transform any that have more than 2.  For instance "UserName" has "Machine Name", "User", and "Local Service".  "Token" has about 10 options, "ChildEXE" has thousands etc.

X
array([[2, 0, 20788, ..., 46, 31, 24],
[2, 0, 19088, ..., 46, 31, 24],
[2, 0, 2840, ..., 27, 42, 15],
 ...,
[2, 0, 20148, ..., 17, 40, 32],
[2, 0, 20148, ..., 47, 23, 0],
[2, 0, 3176, ..., 48, 42, 32]], dtype=object)

Do I use the code below several times for each variable that has more that 2?

onehotencoder = OneHotEncoder(categorical_features = [0])
X = onehotencoder.fit_transform(X).toarray()


texts: I'd like to perform sentiment analysis on stock comment using scikit and nltk. I already have about 100 comments on different stocks like "this stock will rock" which I marked as positive (1) or "this is doomed stock" which I marked as negative(0). So I'd like to train classifier which can tell whether new comments I add are negative or positive. So my question is how to perform it. I've already searched the Net but all I found was movie review sentiment analysis which is quite remote from the topic.


texts: GMM in speech recoginition using HMM-GMM

texts:
I am trying to solve/understand ASR using HMM-GMM.

At the abstract level i do understand what's happening but I did not
understand how GMM fits into it.

My data has 5K hours of speech from single user. I took the above picture from this article.

I do know what is GMM but i am unable to wrap my head around it.
Can somebody explain with a simple example.


texts: Visualizing the difference of a set of strings

texts:
I have a distance metric on a collection of strings on the order of tens of thousands. What would be an intuitive way to summarize how 'different' these strings are or when they overlap?
My goal is, to visually ensure high entropy and be able to recognize clustering regions and the strings associated with it.
I envision a kind of clustering plot where there is some radius around each string that captures its neighbors.... but this requires having a meaningful coordinate system.


texts: NER vs Text classification for very short sentences

texts:
Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ).

The location of each annotation is not impotent (i.e i only need to know if the annotation is included in the sentence)

Which method will be more beneficial ? using NER models with labels attached to the tokens from each sentence, or text classification where the sample is the whole sentence .

The labels are action that physician  is doing (i.e "clean wound" , "remove skin" etc)


texts: How to read binary video format in HDFS using pyspark?

texts:
We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark?

Please help me on this issue.

Thank you


texts: What are advantages or disadvantages of training deep learning model from scratch?

texts: I want to know advantages and disadvantages.Also, What are advantages and disadvantages of transfer leraning?


texts: Scraping mixed elements and passing to SQL

texts:
I'm running webscrapes via python which are retrieving data from csv's hosted on the web.  I'd like to pass the data into a MSSQL database.  An issue I have is the mixed elements/data types in the csv.  Here is an example of the data

Item    Val1    Val2
A       100      200
B       101      201
C       Null    -2/2(%)
D       Null    2019-Nov-18

I would like to import all of this data into the db, but the critical data is in the "Val2" column.  It contains numbers (formatted as numbers and general), dates, and strings with numbers as above.

I would appreciate your thoughts on two things:

After I get this data into python, whats the best way/package to
use to send it to SQL
Do I need to set up the SQL table in a certain way to handle the mixed elements?  Should I filter those different values out with python and put them somewhere else?

For 1, I've looked at things like pd.to_sql and iterating over rows and sending via pyodbc. Data won't be particularly large to begin with. I'll be setting up a lot of scrapes so just want some ideas on the "best" way to do this transfer so I don't have to re-design them all in the future

Thanks I know this is open ended but would love to hear any thoughts on the "right" way to do this


texts: How do you define the steps to explore the data?

texts:
I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is:

Frame the problem
Collect the data
Clean the data
Work on the data
Report the results

I'm struggling to connect the dots when comes to work on the data. I'm aware that step 4 is where the fun happens, but I don't know where to begin. What are the steps taken when you work on the data? Example: do I need to find the central tendency or the standard deviation? Is machine learning needed?

Ps: I know these are broad questions, so please answer it within your own domain expertise.


texts: How can I get a measure of the semantic similarity of words?

texts:
What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal:

# Using the 840B word Common Crawl GloVe vectors with gensim:

# 'hot' is closer to 'cold' than 'warm'
In [7]: model.similarity('hot', 'cold')
Out[7]: 0.59720456121072973

In [8]: model.similarity('hot', 'warm')
Out[8]: 0.56784095376659627

# Cold is much closer to 'hot' than 'popular'
In [9]: model.similarity('hot', 'popular')
Out[9]: 0.33708479049537632

NLTK's Wordnet methods appear to just give up:

In [25]: print wn.synset('hot.a.01').path_similarity(wn.synset('warm.a.01'))
None

What are other options?


texts: How to set the Reconstruction error threshold for anomaly detection using autoencoders?

texts:
Hi I am doing anomaly detection using auto encoders.I have trained the model using 'Non Anomalous' values.Now when I give anomalous points as test data.

What should be the Reconstruction error threshold I should give to classify it as whether it is anomalous or not?

I have currently set it to 0.1 , but it doesnt find any anomalies(actually test data have lot of anomalies)
What are the factors that I should consider before setting the value?


texts: What is explicit fitness sharing in NEAT?

texts: I cannot understand what is explicit fitness sharing. Could someone explain it?


texts: Tool to Generate 2D Data via Mouse Clicking

texts:
Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or tutorial.

In some cases I just create a function, add some noise, and plot it, but there are many times when I wish I could just click my mouse on a graph to generate points. For instance, when I want to generate a fairly complex relationship between x and y, it's a hassle to think of the right formulation to generate the points programmatically.

Does there exist a tool that will allow me to generate data points using my mouse, with an option to export to CSV (or other simple format)?

For example, I am currently learning how to use mutual information and entropy as a metric of dependence between variables. I'd like to see what happens when I have data that is clearly dependent but does not have a linear relationship, so I drew this image:

Now I just need a way to export the coordinates of the points to CSV. I realize this is a simple program and I could write my own, but surely someone has already done this and created a tool to do so? It could be a website, an .exe, Python source, or any other application.


texts: Result of uniform weight initialization in all neurons

texts:
Background
cs231n has the question regarding how to initialize weights.

Question
Please confirm or correct my understandings. I think the weight value will be the same in all the neurons with ReLU activation.
When W = 0 or less in all neurons, the gradient update on W is 0. So W will stay 0. When W = 1 (or any constant), the gradient update on W is the same value in all neurons. So W will be changing but the same in all neurons.


texts: Data augmentation from directory and oriented features with Keras

texts:
Let this, be a dataset labeling the head pose for each photo:

image_path, head_pose
---------------------
michael/01-right.png, right
donie/01-right.png, right
lea/01-left.png, left
jack/01-up.png, up
(...)

The dataset is too big to be load at once but a data frame containing the above information is manageable (ie, not loading the images). Using Keras, I was thinking of creating my own DataGenerator in order to load the images batch by batch.

Then using ImageDataGenerator I can augment each batch. But I'd like to know when a horizontal flip is done because I want to change the associated prediction. Like if the person was facing her left, the expected class should be right and left if the image is flipped.  Maybe extending the ImageDataGenerator ?

What's the best way to handle that with Keras ?


texts:
I tried to install opencv using cmake to work with gpu. After installation, I try to confirm installation. So, I open anaconda prompt and python >> then import cv2. However, I got the error of module not found.
I couldn't find a solution for that on the Internet. Does anyone know how I can solve it?
Thank you in advance for your help.


texts:
I have a dataset of which has 30M rows each like [current_jobtitles, nextjobtitles].
[['junior software programmer', 'senior software programmer'],
 ['senior software programmer', 'lead software programmer'],
 ['sales associate', 'regional sales associate']]

I want to build a deep learning model to predict the nextjobtitle when a currenttitle is given. Are there any ways that I could acheieve this using some deep learning model? if yes, what kind of model ?
Can we use any of the text generation models for this scenario ? I have seen some examples, but I cannot relate to my solution for the problem. Any suggestions please ..!
update:
I have sequences of jobtitles in each row.
[['junior programmer', 'senior programmer', 'lead programmer'],
 ['sales associate', 'sales regional associate', 'sales manager'],
 ['chairman', 'ceo'],
 ['associate', 'intern', 'junior', 'student']]

for the orginal data (in update) i have modifed data as above (before update) by just taking the nextjob title.
Can I train model with the original sequence data to predict the next jobtitle?
** - all the rows will be of different lengths.


texts:
The dataset that I am experimenting with is in the form of a table with columns userid and itemid. If there is a row for a given user and a given item, that means the user accessed the item (like in an online store). I am trying to cluster similar items based on this data. If a pair of items is accessed together often, then the items are similar.

Because this is a case of a high dimensionality (# of users and items will be in 10,000's) I think I am justified in trying to use SVD as a pre-clustering step and then do some classical clustering. When I tried doing this I got poor clustering results when compared with simple hierarchical clustering. Items that weren't very similar were being bucketed together in one dimension, while there were available dimensions that weren't used. The results weren't completely random, but they were definitely worse than the output from the hierarchical clustering. I attempted the SVD step with Mahaut and Octave and the results were similar. For the hierarchical clustering I used the Jaccard measure.

At this point I am starting to doubt the notion of SVD as a way to reduce dimensionality. Do you think that SVD cannot be used effectively in this case (and why?) or do you think that I made some mistake along the way?


texts: How to interpret Low Pearson correlation coefficient between stable signals and high Pearson correlation coefficient between unstable signals?

texts:
I calculated the Pearson correlation coefficient between two signals, that described the state of the unit. During normal operation of the unit, both signals were fairly stable and fluctuated very little.
At some point in time, the unit began to form a defect, in connection with which the oscillations of the signals increased, and also a trend of growth in their absolute values began to be observed.

Signals describing the normal operation of the unit.
Signals describing emergency operation of the unit.
Signals that describe both normal and emergency operation of the unit (combination 1) and 2)).

In each of these three cases, I calculated the correlation coefficients between the signals.
In the first case, it was minimal at 0.5, in the second case, at 0.9, and in the third case, at a level close to 1.
I have the following questions:

Why is there such a low correlation between the signals during normal operation of the unit, when the signals behaved "by eye" in the same way? In fact, both signals at this time had minimal variation and behaved in the same way.

High correlation during the occurrence of a unit failure. Both signals showed an upward trend, both again behaved the same "by eye". But why did the correlation become high in this case? During normal operation, both signals also behaved in the same way, stably kept at the same level.

Maximum correlation when I combined the normal and emergency signals of the unit. Why, by combining data samples with low (normal operation of the unit) and high (emergency operation of the unit) correlation, I get the samples, the correlation between which turned out to be the highest of all three cases? What could have caused this effect?


texts: Using Mean Squared Error in Gradient Descent

texts: I've recently been writing linear regression algorithms from scratch to gain an understanding of how the maths behind it works (something that was a bit of a black box beforehand), and so I got around to differentiating the cost function. Without realising it I used the Squared Error for the cost function - the MSE but without dividing by the dataset length. Is there any benefit (faster approach of the minimum or other) to using the Mean Squared Error over just summing the squares of the error?


texts:
How does reordering the features impact model training and its performance?

Per my understanding, it should not impact the model performance as weights get tuned according to feature value and not per the feature order.

Interested to know from others as well.


texts:
I'm learning about the variational inference - mean field approximation on from this online course DeepBayes2019 page 30
The probabilistic model is written as follows:
$p(X, Z \mid \pi, \mu, \lambda) = Dir(\pi \mid \alpha)\prod_{i=1}^{N} \prod_{k=1}^{K} [\pi_k \mathcal{N}(x_i \mid \mu_k, \lambda_k^{-1})]^{z_{ik}}$
The E-step is defined as $p(Z, \pi \mid X, \mu, \lambda) \approx q(Z, \pi) = q(Z) q(\pi)$ with
$$
q(Z) = \prod_{i=1}^{N}q(z_i) \quad q(z_i) = \prod_{k=1}^{K} \rho_{ik}^{z_{ik}} / \sum_{k=1}^{K} \rho_{ik} \\
q(\pi) = Dir(\pi \mid \alpha'), \quad \alpha_k' = \alpha_k + \sum_{i=1}^{N} \mathbb{E}_{q(Z)}z_{ik}
$$
And the M-step is:
$$
\mu_k = \frac{\sum_{i=1}^{N} \mathbb{E}_{q(Z)}z_{ik}x_i}{\sum_{i=1}^{N} \mathbb{E}_{q(Z)}z_{ik}}, \quad \lambda_k = \frac{\sum_{i=1}^{N}\mathbb{E}_{q(Z)}z_{ik}}{\sum_{i=1}^{N}\mathbb{E}_{q(Z)}z_{ik}(x_i - \mu_k)^2}
$$

My problem is that I don't understand how they defined the $\rho$ in the proof they wrote:
$$
log q(Z)= \sum_{i=1}^{N} \sum_{k=1}^{K} z_{ik}(\mathbb{E}_{q(\pi)} log \pi_k + log\mathcal{N}(x_i \mid \mu_k, \lambda_k^{-1}))+ cst \\
= \sum_{i=1}^{N} \sum_{k=1}^{K} z_{ik} log \rho_{ik} + cst
$$

I am not sure that I understand the role of the $z_{ik}$ matrix


texts: Encoding ML classification features that are relative to the dependant categories

texts:
I have a classification problem with three classes A, B & C.
I have features x1, x2 and x3 that represent my data items. But I also have a fourth feature that represents a similarity between my data item and each one of A, B & C.
My question is, how to encode this similarity feature given that it is relative to each of A, B & C.
My guess would be to have three similarity features sA, sB and sC. One for each category in the style of one-hot-encoding. So my resulting features are:
x1, x2, x3, sA, sB, sC
Questions:

Is this approach problematic in any way?
Is there an alternative?
Is there a name for this kind of encoding?


texts: What is the better to pre-process data, MS-Excel or Python/R

texts: While handling data, many anomalies occur that might change the results. For example, outliers, missing values, etc. So in this context, what is the better tool to pre process data? A visual tool like MS-Excel or programming tool like R or Python


texts: Exclude regression constant from non-linear regression

texts:
For example, suppose I've data set and I want to perform a non-linear regression on the data set as follows:
$Y=B_1e^{B_2 X} + B_3$
and I found that if I exclude $B_3$ the $R^2$ increased also standard error of the regression decreased, can I exclude $B_3$?


texts: How to combine spectrogram image with its human-labelled data to be processed with CNN in Python?

texts:
I am doing a final project at campus: pitch estimation from a song using CNN.
Input to CNN is spectrogram of a song, generated by plt.specgram(), with size 334 x 217. The song dataset is taken from MIR-QBSH, with this specification: 8 sec duration, mono, 8KHz sampling, 8-bit quantization, frame size = 256, overlap = 0, and the first frame starts from the first sample of the audio file.
This is one example of the spectrogram:

As far as I understand now, I need data label (in my case: pitch labels) combined with the spectrogram for CNN to be able to process the computation. My data label contains 250 pitch labels for 1 song. These pitch labels are in the unit of semitone (MIDI number).
This is the example of pitch labels for spectrogram above. I have done math.floor() method to these pitch labels from the original file to simplify the computation.
Pitch values:  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 48, 50, 50, 52, 52, 53, 54, 54, 53, 53, 53, 53, 54, 54, 54, 54, 54, 53, 0, 0, 54, 54, 54, 54, 54, 54, 53, 0, 0, 0, 0, 46, 46, 46, 47, 48, 48, 48, 48, 48, 49, 49, 49, 50, 50, 50, 50, 0, 0, 0, 50, 50, 50, 50, 50, 50, 50, 49, 0, 0, 51, 47, 47, 47, 47, 47, 47, 47, 47, 48, 49, 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 57, 58, 58, 58, 58, 58, 57, 57, 57, 57, 57, 57, 57, 58, 58, 57, 57, 57, 57, 56, 55, 55, 56, 56, 56, 56, 56, 55, 56, 56, 56, 56, 55, 55, 55, 56, 56, 54, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 52, 52, 52, 53, 53, 54, 54, 0, 0, 54, 54, 54, 54, 54, 54, 54, 54, 0, 0, 0, 54, 54, 54, 54, 54, 53, 52, 51, 47, 47, 47, 47, 47, 47, 47, 47, 47, 48, 49, 49, 49, 49, 50, 50, 50, 50, 49, 0, 0, 0, 50, 49, 49, 49, 49, 49, 50, 50, 49, 0, 0, 47, 47, 47, 48, 48, 48, 48, 0, 0, 0, 0, 0, 0, 0]

My question is, what should I do to combine the spectrogram and its pitch label before it is processed by CNN in Python? Is using


texts:
While running rattle in my system I am getting this error

rattle()

Error: attempt to apply non-function
In addition: Warning message:
In method(obj, ...) : Unknown internal child: selection

I am using R version 3.1.0 (2014-04-10)


texts: How has data generation evolved over the years (and centuries)?

texts:
I would like to know how data generation has evolved through the years.

My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does someone have a reference for this? I couldn't find any. It's OK if you point me to a book (available on eBay) that has this information.

Thanks!

Edit: when I talk about data generation, I talk about combined information generated by both machines and humans. I know this is not very well defined, but, for example, before the inventions of computers, you can consider books and movies.


texts: Agglomerative Hierarchical Clustering on Images

texts:
My goal is to implement the agglomerative hierarchical clustering algorithm on an RGB image to cluster every pixel until some stopping criteria is reached. In order to do so, I assumed that each pixel belongs to a cluster initially. Then, to merge pixels, it is necessary to calculate the distance between every other pixel, which makes agglomerative hierarchical cluster inefficient in terms of computational time, especially if the image is large.
I somehow need to get around this problem, but don't exactly know what approaches would help.
One thing came to my mind is to apply SLIC to obtain superpixels and select each superpixel as a cluster at the beginning. If this sounds reasonable, what similarity metrics should I consider to merge superpixels ?


texts: What is the classification accuracy of a random classifier?

texts: I have a build a classification model using machine learning technique (SVM). I want to compare the classification accuracy of my model with a random classifier. My data set contains only two classes(1 or 0). The ratio of 1 and 0 instances are 35% and 65%. That means, 35% instances belong to 1 and 65% belong to 0 class. In that case, what will be the classification accuracy of random classifier (Random Guess)?


texts: Mobile App Recommendation: How to get the rate of a specific user submit for a specific application

texts:
I have a mobile app recommendation project, so I need data set which has user-app matrix-rate.
Actually, I want to know what rate does a specific user submit for a specific application.
in other words, "user-app matrix-rate" is a 2-D matrix, rows are users, columns are applications and cell(i,j) is rate of user i submit for application j.

In addition, It's good for me to know "user-app matrix-install" where
"user-app matrix-install" is a 2-D matrix, rows are users, columns are applications and cell(i,j) is 1 if user i installed application j, otherwise cell(i,j) is 0

thanks for your help


texts: Nested cross-validation for regression over small dataset

texts:
I'm trying to do nested cross-validation for regression model parameter selection and prediction evaluation. I'm using temporal data (series of count).

The problem is that I don't have a lot of data for training and test so I'm to follow this process:

Split the data into two folds (80% - 20%)
Use standard GridSearchCV for parameter selection over the first
fold
Evaluate the model with the best parameter by TimeSeriesSplit
cross-validation. Here, I will use the whole dataset but I will define a new TimeSplit function to work like this: only the second fold will be divided into N subfolds and for each iteration i, the model learns on the first fold dataset (80%) + i subfolds of the second fold and test on the next subfold.

Did I perform the nested cross-validation properly so that my evaluation results are unbiased?

One other question, is it required to use TimeSeriesSplit cross-validation when using regression model over time series data?

Any help is appreciated


texts: How to use gradient checkpointing on packed sequence RNN

texts:
I have a batch of sequences that have a variable length. To save computation I used pack_padded_sequence as following:
input = torch.nn.utils.rnn.pad_sequence(input, batch_first=True)
input = torch.nn.utils.rnn.pack_padded_sequence(input,
                                                batch_first=True,
                                                lengths=lengths)

Because sequences are long, I use gradient checkpointing to save memory
output, hiddens = cp.checkpoint(self.gru, *(input, hiddens, self.dummy_tensor))
As a result I have such error:
File ".../src/sequence_models/gru.py", line 86, in forward
    output, hiddens = cp.checkpoint(self.gru, *(input, hiddens, self.dummy_tensor))
  File ".../torch/utils/checkpoint.py", line 177, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
TypeError: CheckpointFunctionBackward.forward: expected Tensor or tuple of Tensor (got PackedSequence) for return value 0

How can I handle it?


texts: Input Shape Problem with Ragged Tensor input in RNN

texts:
I have created ragged tensors from python lists as below:

list_a=tf.ragged.constant(list_a)

list_b=tf.ragged.constant(list_b)

Checking the shapes gives me:

list_a.shape

TensorShape([10555, None, None])

list_b.shape

TensorShape([10555])

dataset = tf.data.Dataset.from_tensor_slices((list_a, list_b))
dataset

T.ensorSliceDataset shapes: ((None, None), ()), types: (tf.float64,      tf.float32)>

train_data=dataset.take(9000).batch(5)
train_data

BatchDataset shapes: ((None, None, None), (None,)), types: (tf.float64, tf.float32)>

My problem is that I cannot figure out how to specify the input_shape in a SimpleRNN layer with this type of dataset.


texts: Do batch norm makes sense for regression problems?

texts:
When my network is performing regression (like DQN) it makes sense to use batchnorm in network when output of my network should vary from [0, 100000]?

one way to tackle it is to normalize output but output is used for ranking choices in RL system so scaling can lead to small numerical errors that can explode after upscaling...


texts:
I am still having many doubts about the working of categorical embedding.
In particular I have 2 points not clear:
1.  Are 1-Hot variables converted to a lower dim vector?
2.  What target are neural embedding trained on? Lots of explanations I have seen don't show any target, and I wonder how the back-prop is done.
3.  It is not clear at all how the keras functional API works to build an embedding.
Anyone is so helpful to explain step by step how to build one showing matrices dimension?
Thanks in advance.


texts: High dimensional space is dense or sparse?

texts: I read some blog articles recently. One mentions that you could not imagine high dimensional space as 2d or 3d as distance between any 2 points in high dimensional space tends to be similar, which means 'dense'. However in the t-SNE paper, it says high dimensional space tends to be sparse such that you have to employ special dimensionality reduction techniques to visualize in 2d or 3d space in meaningful way. So how to reconciliate these 2 different views?


texts: Dummy vectors and performance measurement for vector search Face Recognition

texts:
I have about thousands of person face (from celebrity dataset LFW), which each person represented by 512 x 1 vector. I stored it on vector DB to build face searching system using embedded feature (MTCNN for face detection and arcface for embedding model). Someone suggest me to add many vectors as "dummy faces" to the database with unknown class (the number of the vectors is larger than the personal class).
It's still unclear for me why I need to add many unknown faces as "UNKNOWN" class and put it together with thousands of vector from each person. As far as I know, its pretty easy to check the performance by get the similarity score with only from known vectors (the vectors from each person) without the unknown one, for example let said if i put k = 3 or k = 5, i will take the minimum distance as the result and get the class of the vector (ID or label).


texts: I'm new to Deep Learning. I only know about scikit learn when I have to pre process data for a neural network. are there any other good libraries for that ?


texts: Semantic Segmentation using Scipy or Scikit-image

texts:
Many days ago I saw a user using scipy function to get region of interest (RoI) in an image using different colors. I believe it was some sort of filter. Now I'm trying to do the same but I'm not able to find that content anymore.

I would grateful if you could help me with that problem.

Image Example: Example of Semantic Segmentation


texts: What is the role hidden vectors play in RBM?

texts:
I'm learning about the restricted Boltzmann machine, but I can't decide whether the hidden vectors of RBM are useless or not.

The following are my two understandings for the role hidden vectors play. Have I misunderstood something? If not, then how could the role those vectors play  be so different?

This is the first understanding:

I learned from this video RBM that hidden units are just structural support and we don't care about what those hidden vectors really are. We introduce hidden units into RBM just for it to gain more expressive power for probability distribution.

Let's say I have only one image, and I transform this image to a binary vector and feed this vector into a RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.

(my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)

And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.
My second understanding goes like this:

RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

Given a RBM, each hidden units of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by this hidden unit(by the weights and bias). So, if we input an image into this RBM, what the RBM will do is to project this input vector onto those hyperplanes defined by all the hidden units. Thus for an input vector, the corresponding hidden vector is very important, it is some abstract representation. And we can further feed this representation into other models for doing classification.

So, these are my understandings, if you can answer the question by explaining how RBM is used for MNIST that would be extremely helpful for me.


texts: Non semi positive definite kernel matrix

texts:
What happens if we run a support vector machine model using a kernel that does not satisfy requirements such as non-positive semi definite?
This is my flow of thought: In kernel methods $w.x$ is replaced by $\sum_i \alpha_i k(x, x_i)$. Now if k is not positive semi definite, $\exists y$ such that $k(y,y)<0$. This means there exists y such that $k(y, y) < 0$. So I feel that the classifier will become noisy, because it will keep misclassifying y. Is this correct? If so, how do I make my arguments more rigorous?


texts: Using Cross Validation technique for a CNN model

texts:
I am working on a CNN model. As always, I used batches with epochs to train my model. When it completed training and validation, finally I used a test set to measure the model performance and generate confusion matrix. Now I want to use Cross Validation to train my model. I can implement it but there are some questions in my mind:
1- Why most CNN models don't use Cross Validation?
2- If I use Cross Validation, how can I generate the confusion matrix? Can I split dataset to train/test and then do cross validation on train set as train/validation (i.e., doing cross validation as train/validation except for the usual train/test) and at last use test set the same way? Or how?


texts: How to combine and separate test and train data for data cleaning?

texts:
I am working on an ML model in which I have been provided the data in 2 files test.csv and train.csv. I want to perform data cleaning on both files together be concatenating them and then separating them.
I know how to concatenate 2 dataframes, but after data cleaning how will I separate the two files? Please help me complete the code.
CODE
test = pd.read_csv('test.csv')
train = pd.read_csv('train.csv')

df = pd.concat([test, train])

//Data Cleaning steps

//Separating them back to train and test set for providing input to model


texts: SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

texts:
I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes in. I know I can use partial_fit() for this. The following seems to be a common example on StackOverflow:

clf = linear_model.SGDClassifier()
x1 = some_new_data
y1 = the_labels
clf.partial_fit(x1,y1)
x2 = some_newer_data
y2 = the_labels
clf.partial_fit(x2,y2)

The problem is, partial_fit() performs just one step of gradient descent:

  Internally, this method uses max_iter = 1. Therefore, it is not guaranteed that a minimum of the cost function is reached after calling it once. Matters such as objective convergence and early stopping should be handled by the user.

Would one step be enough? How do I know if I don't need to make more steps than just one per each new labeled datapoint / minibatch that comes in?


texts: How does Keras' fit function calculate when validation set/validation split are NOT defined (I understand that the default values are None/0.0 respectively, so being not defined is practically default)? One always need a reference set to evaluate model performance...


texts: How to predict next visit date based on this data

texts:
I have a dataset shown below. Here, status is if visit has been done or not and schedule is if next_action_scheduled.

df =

visit_date |status |scheduled_visit

2018-09-13|1|0000-00-00

2018-11-13|1|0000-00-00

2019-01-13|1|0000-00-00

2019-03-13|0|0000-00-00

2019-05-13|0|0000-00-00

So consider this case like user is visiting in every 2 month.

I would like to find this pattern for all users based on the historical visits with the customer. I would like to find the next predicted visit for that customer and if we missed that visit, then I ask "Do you want to schedule a visit" Or "You missed a visit with this customer."

I'm new in Ml. I tried to use RNN for time series forecasting but I'm getting a really high error rate. Prediciton is nowhere close. What model/algorithm I should use and learn about to make a model for this? I want to create a solution like if I feed in the user_id it will check all these values and send notification within app.

Thank you


texts: Orange 3.15 GUI window size issue (Windows 8.1)

texts:
I'd just like to report an annoying GUI bug I am experimenting under Windows 8.1 using Orange 3.15.
The work space available doesn't size up with the window making the use of the software quite frustrating.
Here is a video of the bug.
Does anyone have a fix for that by any chance?


texts: Modelling a startup's funding journey with Brownian Motion

texts:
I am trying to implement a "light" version of a paper (Hunter, Saini & Saman 2017), in which the  authors build a model capable of predicting the probability that a startup will exit (either by getting acquired or IPO).

Both when it comes to the math and programming, I am in over my head, but I am adamant in understanding what is going on and how to implement the paper myself.

I understand the general idea behind the their model, but there are two things that are unclear to me:

How is the prediction problem set up? In supervised learning terms, what is the label?
How should the likelihood function be implemented programatically (preferably in Python)?

Data

I have collected data on several thousand companies, where each row represents a company.

My features include, but are not limited to, sector, investor type, number of founders and founder education. Additionally, I have data about the funding rounds a company has raised. Specifically, I have the announcement dates and the deal size in USD.

Model

I would like to train a model that is able to return predicted probabilities of a given startup's chance of exiting given a feature vector. The authors suggest a Brownian Motion Model to solve this task. They specify it in the following way:

They specify the Data Likelihood like this:

Using this specification, it is possible to figure out 1) what are the labels going into the model, and 2) how can this be implemented with Python?

In relation to 1), for each company, my understanding is that I will need a) the timing between each rounds and b) the set of funding rounds raised.

The authors then proceed to estimate the parameters using a Bayesian approach, but for my purposes that would be slightly overkill, so I believe gradient descent should be able to do the job.


texts: How do we decide on the classification algorithm to use with huge training size?

texts:
I am solving a questions binary classification problem and the training size for this is huge(291 billion). The data has bloated because of using tfidfvectorizerfor the questions column. Here, in the problem, I have to classify the questions.

I have used Logistic regression and have also kept MultinomialNB, Randomforest and svm for training. However, instead of doing such trial and hit method, Is there a logical explanation why one classification algorithm must perform better than others in this context.

Previously, I have tried randomforest and logistic regression for spam filtering and observed that the training error was less for logistic regression as compared to randomforest. I understand that it could be an overfitting solution.
But Is there a way I can say for certain that 'this' is the classification algorithm you must use.

Note: I'm am yet to remove stopwords and do some dimensionality reduction.


texts: Preprocessing float data CSVs for evaluation using Resnet

texts:
I'm currently working on evaluating float32 data in CSVs using Wide-Resnet.
Data Explanation
The CSV data also have nan values within them (which is represented with "-"). The data doesn't have any specific min or max range to them, unlike byte images, where the range is 0-255.
Here's 2 snippets of a part of the data.

Preprocessing Explanation
I evaluated the data by reading them to a dataframe using pandas library, and converted the "-" to nan value. I then converted the dataframe to a PIL float image and preprocessed it using torchvision.
The code below shows the preprocess used.
Preprocessing Code

def __getitem__(self, idx):
    # get path of csv file
    x = self.x[idx]

    # convert csv to dataframe
    x = pd.read_csv(x, delimiter=',', skiprows=5, na_values="-", dtype=np.float32)

    warnings.filterwarnings("ignore")
    x1 = x.to_numpy()

    # convert to image and apply resize and other preprocess
    x2 = self.transform_x(x1)

    # pre-normalization
    scaler = MinMaxScaler()
    x3 = scaler.fit_transform(x2)

    # replacing nan with lowest value
    x3[np.isnan(x3)] = 0
    warnings.filterwarnings("default")

    # Create a fake 3 channel tensor by copying and stacking the data
    x4 = T.ToTensor()(np.array(x3))
    x5 = torch.cat([x4, x4, x4], dim=0)

    x6 = T.Normalize(self.ImageNetNormParam["mean"],
                     self.ImageNetNormParam["std"],)(x5)

    # Tensor type seems to change to double along the way, so it
    # is once again reconverted to float
    x7 = x6.to(dtype=torch.float)

    # it turns to a tensor and returned
    return x7, y

    self.ImageNetNormParam = {"mean": [0.485, 0.456, 0.406],
                              "std": [0.229, 0.224, 0.225]
                              }
    self.resize = (416, 288)

    self.transform_x = T.Compose([T.ToPILImage(),
                                  T.Resize(resize, Image.ANTIALIAS),
                                  ])

Problem Statement
My questions are as below:
1
In my preprocessing, I copied and stacked my 1 channel tensor to create a fake 3 channel tensor so that Resnet could read it.
Is it OK to do this? Is there a more correct way to evaluate 1chn data using Resnet? Are there pretrained architectures where it accepts 1 channel data?
2
In my preprocessing, I replaced the nan value in the data with 0. However, this will overlap with actual 0 values existing within my data.
Is there a way to mask nan data such as that Resnet would read this data as not-effective values, or are there other ways to approach this?
3
My normalization used Imagenets's stdev and men preset to normalize. However, my type of data isn't the same such as the ones in Imagenet. Is that OK, or should I calculate my own stdev and mean based on my data?
4
I used scikit-learn's min max scaler to scale my data to 0-1. However, I would like so that my data wouldn't be scale based on each individual data's min and max, but with the min and max that is based of all the data. This is the same as scaling byte images, where every image has a determined min max which is 0-255.
However, problem arises when I evaluate multiple batches of these data. For example, supposedly for the 1st batch of float csv data, I get a min max of -5to3. I then used this as a standard and scaled the 1st batch and 2nd batch with the standard min max. But for the 3rd batch, the min was -50 and under the standard min. This would lead to an improper scaling of the 3rd batch.
Can Resnet still read the improperly scaled batch? Or is there a way to scale float CSVs (which doesn't have any min max range) properly?

Thank you very much for taking your time in answering my questions.


texts: Keras Conv1D for simple data target prediction

texts:
I am trying to use conv1D layer from Keras for predicting Species in iris dataset (which has 4 numeric features and one categorical target). Following is my code:

import numpy as np
import pandas as pd

irisdf = pd.read_csv('iris.csv')

Xall = irisdf.drop('Species', axis=1)
print(Xall.shape)
Xall = np.expand_dims(Xall.values, axis=2)
print(Xall.shape)

Yall = irisdf['Species']
nb_classes =  3

import keras
from keras.models import Sequential
from keras.layers import Dense, InputLayer, Dropout, Flatten, BatchNormalization, Conv1D
input_shape = (Xall.shape[1:],)
print(input_shape)
model = Sequential([
    InputLayer(input_shape=input_shape),
    Conv1D(32, 2),
    Dense(nb_classes, activation='softmax')
])
model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])
model.summary()
model.fit(Xall, Yall, epochs=25, verbose=True)

However, it is giving following error:

Traceback (most recent call last):
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/eager/execute.py", line 141, in make_shape
    shape = tensor_shape.as_shape(v)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 946, in as_shape
    return TensorShape(shape)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 541, in __init__
    self._dims = [as_dimension(d) for d in dims_iter]
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 541, in <listcomp>
    self._dims = [as_dimension(d) for d in dims_iter]
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 482, in as_dimension
    return Dimension(value)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_shape.py", line 37, in __init__
    self._value = int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rnkeras_conv1d_iris.py", line 40, in <module>
    InputLayer(input_shape=input_shape),
  File "/home/abcde/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/abcde/.local/lib/python3.5/site-packages/keras/engine/input_layer.py", line 86, in __init__
    name=self.name)
  File "/home/abcde/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 515, in placeholder
    x = tf.placeholder(dtype, shape=shape, name=name)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 1735, in placeholder
    return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4923, in placeholder
    shape = _execute.make_shape(shape, "shape")
  File "/home/abcde/.local/lib/python3.5/site-packages/tensorflow/python/eager/execute.py", line 143, in make_shape
    raise TypeError("Error converting %s to a TensorShape: %s." % (arg_name, e))
TypeError: Error converting shape to a TensorShape: int() argument must be a string, a bytes-like object or a number, not 'tuple'.

Where is the problem and how can it be solved?

(PS: If you find this question to be interesting/important, please upvote it;)


texts: How can I improve the accuracy of my neural network on a very unbalanced dataset?

texts:
I have a dataset which contains data for about accidents. The dataset consists of about 15.000 entries and I can't get more. The Distribution is as follows:

88.6% of the data are class 1 accidents
10.6% of the data are class 2 accidents
0.8% of the data are class 3 accidents

As you can see, the biggest part of the training data belongs to one class. I have only very few examples for class 3 accidents (about 100 rows out of 15.000), but it would be most important to classify class 3 accidents correctly.

I trained a pretty standard deep neural network on the data and got an accuracy of ~93% on the validation set. I used a custom Tensorflow estimator with an AdamOptimizer and tuned the parameters as good as possible.
The Problem is, the network still classifies most of the accidents as class 1 accidents. So if I have for example 25 class 3 accidents in the validation set, the network misclassifies 10 of them as class 1. I want to improve that.

Are there any methods to improve the performance in this case? The obvious choice would be to get more data of class 3 accidents, but sadly that is not possible.
Does it make sense to show the existing class 3 data multiple times? So for example, train 5 Epochs with all data and then 3 additional Epochs with just the class 3 accidents?

Or could I do something during the data preprocessing? Right now I'm MinMax-Scaling the input data to get to the [0, 1] interval. Is there maybe any other way to emphasize outliers more? (If you assume outliers mostly belong to class 3)

I hope someone knows some methods to increase the accuracy in this case.

EDIT:
The Dataset has mostly categorical columns like:

Street Class (e.g. highway or country road)
Light (e.g. "good")
Weather (e.g. "rainy" or "sunny")
...

Additional it has these columns:

Accident Date (just month and day)
Age
Time of the day
Number of injured persons
Number of Vehicles

So an entry might look like this:

{
 street_class: 'highway',
 light: 'daylight',
 date: '23. Jan',
 age: 59,
 injured_persons: 2,
 vehicles: 2,
 time: 1724,
 label: 1
 ...
}


texts: Derivative of logarithm of loss function. Logistic regression

texts:
I am reading machine learning literature. I found the log-loss function of logistic regression algorithm:
$$
l(w) = \sum_{n=0}^{N-1}\ln(1+e^{-y_nw^Tx_n})
$$
Where $ y \in {-1;1}, w \in R^P, x_n \in R^P$
Usually I don't have any problem with taking derivatives. Think that derivatives w.r.t. to a vector is something new to me.
I can't figure out on how to take derivative w.r.t w. My try was:
$$
\frac{dl(w)}{dw}=\sum_{n=0}^{N-1}\frac{e^{-y_nw^Tx_n}y_nx_n}{1+e^{-y_nw^Tx_N}}
$$
Is it possible that:
$$
\begin{bmatrix}
... \\
\frac{dl(w)}{w_i} = \sum_{n=0}^{N-1}\frac{-e^{-y_nw^Tx_n}y_nx_n}{1+e^{-y_nw^Tx_n}}x_i  \\
...\\
\end{bmatrix}
$$
I suspect that here is something wrong. Can you please show me some example, some hints on how to do that.
Thanx


texts: Clusterize item set with items as vectors of features

texts:
I have to clusterize this dataset in which I have houses and water consumption in this form:
$$
House1 = (x_{1},x_{2}... x_{n});\\
House2 = (y_{1},y_{2}... y_{n});\\
House3 = (z_{1},z_{2}... z_{n});\\
$$
where $x_{i}$ is the daily consumption in liters while  $n$ is a fixed parameter (length of dataset).
I need to cluster these houses in k clusters based on their water consumption.
My question is: how can I handle data expressed in this form to feed in the clustering algorithm?
Maybe I will have to agglomerate each vector in some real value?


texts: Datanode not starting on Slave Nodes for Apache Hadoop+Spark Setup

texts:
I know similar questions have been asked and I already looked through a lot of them to figure out what the issue might be, but to no avail.

I am a College student currently working on a Big Data Project. I have to setup 5 Computers (1 Master + 4 slaves) to run Apache Spark on YARN.
I had an initial Hadoop setup that I setup Spark on but I messed on some steps as the guides were from different sources.

So, I decided to start fresh with a guide for both Hadoop and Spark from the same source.
I found and followed this Hadoop Setup Guide.

Note: Since the PCs have a good amount of Memory, I did not Configure Memory Allocation and left it as is.

After setting Hadoop up, I tried to start the daemons with start-dfs.sh and start-yarn.sh.

On the master Node, jps showed this:

21922 Jps
21603 NameNode
21655 DataNode
21787 SecondaryNameNode

but, on all the slave nodes jps came up with nothing:

19819 Jps

Is there something that I could have messed up here or is there an error in the Guide?

I had setup Hadoop previously and use this guide.

I tried some solutions that I found online but was unable to fix the issue.
For reference, the Spark Installation Guide though I haven't started with this yet.

I would really appreciate any and all help.
Thanks!


texts: Using LSTMs for continous learning and predicting

texts:
I'm trying to develop a model to predict a commodity price movement direction based on previous observations. The model should learn common technical analysis patterns, e.g. head and shoulders. So, I think I should use a stateful LSTM so that it maintain a long term state to keep track of technical analysis patterns.

On the other side, as the data set is updated daily, i.e. new observations are added, I need the model to keep learning and making predictions every day. So, in order to update the model parameters on each new observation, I think I should use batch_size=1.

If I use the last N observations to predict the next M steps, the input tensor to the model would have the shape of (1, N, num_features):

$$
    X_1, X_2, ..., X_N \rightarrow Y_1, Y_2, ..., Y_M
$$

Questions:

Why should we reset_states after training on train set and prior to predication on test set? I think we shouldn't as a TA pattern may be half in the train set and half in the test set. If we reset the states, the model can't recognize this pattern.
Can I reset_states and feed in data of a different commodity? as I only need the model to learn TA patterns not commodity specific characteristics. If so, how can I restore the states to make prediction on the first commodity?


texts: Real time noise removal using Savitzky-Golay Method

texts:
I would like to ask if Savitzky-Golay can be implemented on real-time data.

I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone refer me to appropriate implementation or hint online implementation. Thanks.


texts: How to select correlation between object type data for modelling in Data Analysis?

texts:
I am analyzing the data with Python. I am at the stage of finding correlation of columns with the target column (price in this example) after data cleaning, to use the columns which have correlations with the target column in the model to perform the prediction.
I have found the correlation between integer type columns through analysis of results of Pandas method .corr() and then calculated Pearson Coefficient together with P-values in order to confirm the correlations. Then I have used .describe method of the Pandas library for object types to have the results of the first part of an image (from cmd). In order to find the correlation between object type columns and a target column, I have implemented ANOVA analysis for all of the columns of object type and analyzed the F test values and P-values of each column.
As it can be seen from the picture, just 2 columns (drive-wheels, num-of-cylinders) have high F values which are 68 and 55 (low values have been deleted).
As I know, ANOVA can be implemented also to find which combinations of categories within the column can have a significant correlation, but I don't know how to use this and don't have a good understanding of why we do it and how it is used in building model for prediction. Could someone explain?
I also did 21 ANOVA analyses within num-of-cylinders column to reveal the correlation between categories which can correlate with the target column. Is it right to do? And what does it give us? I have printed the results that have a good correlation, but I don't understand what they give us.


texts: Time Series pattern recognition and classification problem

texts:
I have some labeled sensor data.

Now, I would like to know how to extract features from time series using DFT, DWT, and HAAR transforms.

I know that the transformations above transform a signal to frequency or time frequency domain, but how can I get average frequency for this non stationary signal?

Thanks.


texts: How can one use unstructured data for forecasting?

texts:
How can one use unstructured data for forecasting purposes? (by unstructured, I mean unstructured in the database sense).

I have a forecasting system that uses historical data and a set of additional explanatory features to forecast demand. In terms of databases, this is as structured as data can get.

I've heard mentioned in several places that big data and unstructured data can be used to improve demand forecasts.

But when I dig into the details, it seems that they always end up reducing the unstructured data to some structured format first, then feed it to whatever forecasting method they are using. They aren't using "true unstructured data".

How can one use true unstructured data in a forecasting algorithm? Any publications, references on this?


texts: How do i iterate functions until convergence in R?

texts:
I am looking to iterate until convergence but I am not sure how i should code it.
A simple similar example is something like
$X_0=5 ,  Y_0=3$
for $n=1,2..$
$P_n=X_{n-1}-Y_{n-1}$
$Y_n=3P_n$
$X_n=P_n+Y_n$
and end when it converges.
I have seen that for the repeat loop it loops indefinitely with an exit condition but I am unsure how i can word the iterations  within R and how to word the exit condition for convergence. Any references or help would be appreciated.


texts:
The problem I face is of a language barrier. I have a set of samples consisting of the following data:

Sample number    Organism Count
1                5
2                8
3                5
4                7
5                13
...              ...

I need to find the number of samples required to make an accurate estimate of the organism within a square meter of ground. This is for a school assignment. I don't need an answer to that. What I need to do is find the standard deviations between the set like follows:

(sample 1) = Standard deviation
(sample 1, sample 2) = standard deviation
(sample 1, sample 2, sample 3) = standard deviation

In a rolling way like that. The only information I can find about rolling standard deviation is with a specific window of samples. What I need is to grow from the first one. What is this called? I'm writing a program in python to calculate and display it in a graph. I want to know the name so I can prevent reinventing the wheel in a likely less performant way (if there is a name to this specific question). It is likely that the libraries I'm working with already provide a method to calculate what I'm asking.

So the problem is:
I need to find the standard deviation of a 'growing set' and I want to know if that has a specific name, so I can look for it in the documentation of the python library I'm using.

Feel free to change the name of the question to make it better fit, since I didn't know the name I couldn't state it in the title.


texts:
I am trying to plot time threshold maps, proposed by Fryzlewicz. The relevant paper is: Time-Threshold Maps: using information from wavelet
reconstructions with all threshold values simultaneously.

Questions: Is the top plot in Figure 1 a heat map? How can I plot this in R?


texts: Dueling DQN - can't understand its mechanism

texts:
I am trying to understand the purpose of Dueling DQN.
According to this blogpost:

our reinforcement learning agent may not need to care about both value and advantage at any given time - this seems to be what I can't understand.

Let's assume we are in state $S_t$ and we select an action which has the highest score. This is the promised total reward that we will get in the future if we take the action.

Notice, we don't yet know the V or A of the future state $S_{t+1}$ (where we will end up after taking this best action), so decoupling V and A in any state, including S_{t+1} seems unnecessary. Additionally, once we do get to work with them we still seem to recombine them during $S_{t+1}$ into a single Q-value, just as was noted in the blog post.

So, to complete my thought: V and A seem to be a "hidden intermediate step", that still gets combined into Q, so we never know it's even there. Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q?

A slightly unrenated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Can someone provide a different example to the sunset?

Edit after answer was accepted:

Found a friendly explanation about this architecture here.

Also, if someone struggles to understand what V, Q and A is, read this answer, and my comment under it.


texts: Efficient way to replace incorrect words in Series of strings

texts:
I'm working with text data, that is handwritten, so it has lots of ortographic errors. I'm currently working with pyspellchecker to clean the data and I'm using the correct() method to find the most likely word when a word doesn't exist. My approach was to create a dictionary with all poorly written words as keys and the most likely word as value:

dic={}
for i in df.text:
    misspelled = spell.unknown(i.split())
    for word in misspelled:
        dic[word]=spell.correction(word)

Even though this is working, it is doing so very slowly. Thus, I wanted to know if there's a faster option to implement this. Do you have any ideas?


texts: Confusion matrix of UNET image sgemenation model

texts: I have used Unet model for image segmentation. I have used RGB images and corresponding image masks and at output i got corresponding region of interest. Now i want to find confusion matrix of this model and from this i want to evaluate other parameters. Can some help? I could not get what are predicted and actual values which are required to build confusion matrix. Thanks


texts: LSTM - How many times should I look back to predict next six hours -Multivariate Time-Series

texts:
I am still finding confusing on look back topic when using LSTM for time-series analysis.
If I have hourly data and I want to predict next 6 hours with multiple predictors, should I look back up to 6 hours when I prepare my training set?

OR

Should I look back one hour (shift 1) and then predict next hour and take that predicted value and feed it back to predict the value after until next 6 hours?

This concept is still a little fuzzy for me in LSTM. Any thoughts would be appreciated.


texts:
I'm making a Text Chunking program using Bi-LSTM from the model I of the paper "Neural Models for Sequence Chunking". The inputs are sequences of words and the outputs are "B-NP", "I-NP", and "O".

The problem I have is the output. The expected output is 2 dimensions, not 3 dimensions.

The X Train shape is (631, 80) while the Y Train shape is (631, 80, 2641).

I get the problem, but I don't know how to solve it. I've used np.array and reshape on Y Training, but no luck.

I don't have experience using Tensorflow and Keras and this is my first time, so I was hoping you could help me.

# Token Indexing
word2idx = {"PAD": 0, "UNK": 1}
word2idx.update({w: i+2 for i, w in enumerate(vocab_words) if w in vocab_words})
tag2idx = {t: i for i, t in enumerate(vocab_tags)}
idx2tag = {v: k for k, v in iteritems(tag2idx)}

# Padding
X = [[word2idx[w[0]] for w in s] for s in quran_sentences]
X = pad_sequences(maxlen=max_length, sequences=X, padding="post",value=word2idx["PAD"])

y = [[tag2idx[w[1]] for w in s] for s in quran_sentences]
y = pad_sequences(maxlen=max_length, sequences=y, padding="post", value=tag2idx["O"])
y = [to_categorical(i, num_classes=n_tags) for i in y]

# Split Data
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, shuffle=False)

# Parameters
hidden_state_encoder_size = 100

batch_size = 64

training_epoch = 200

embedding_size = 80

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+2, output_dim=embedding_size, input_length=max_length, name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size, return_state=True, name="LSTM"), name="Bi-LSTM")
encoder_outputs, forward_h, forward_c, backward_h, backward_c = encoder(embed)

# Concatenated
state_h = Concatenate(name="Concat_1")([forward_h, backward_h])
state_c = Concatenate(name="Concat_2")([forward_c, backward_c])

# Average
average = Average()([state_h, state_c])

# Outputs
outputs = Dense(num_decoder_tokens, activation="softmax", name="Output")(average)

# Build Model
model = Model(inputs, outputs, name="Seq2Seq Chunking")

# Compile & Training
# Compile
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')

# run training
model.fit(X_tr, np.array(y_tr),
          batch_size=batch_size,
          epochs=training_epoch,
          validation_split=0.1)

Error:

ValueError: Error when checking target: expected Output to have 2 dimensions, but got array with shape (631, 80, 2641)


texts: Predicting missing data. Looking for good data predicting technique

texts: I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data science technique to fill the missing Trade in GDP for the below example.


texts: Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

texts:
I am creating decision trees modeling data that looks like this.

pelvic_radius   degree_spondylolisthesis    class
82.45603817    41.6854736      Abnormal
114.365845     -0.421010392    Normal

When finished, I run my test data through my tree and compare the outputs from the run to the ones I am given. This will allow me to check my accuracy, precision, and recall values.

TP = 0; % True Positives
TN = 0; % True Negatives
FP = 0; % False Positives
FN = 0; % False Negatives

And then once those values are calculated, I can calculate the following.

precision = TP/(TP+FP);
accuracy = (TP+TN)/(TP+TN+FP+FN);
recall = TP/(TP+FN);

However, this can be done in two ways. One considering the 'Normal' class as Positive and one Considering the 'Abnormal' class as positive.

Here is the sudo code to further explain what I mean.

for k=1:length(resultsOfTestSet)
    if(strcmp(resultsOfTestSet{k},'Normal'))
        if (strcmp(testSet{k}, 'Normal'))
            % TRUE POSITIVE
            TP = TP + 1;
        else
            % FALSE POSITIVE
            FP = FP + 1;
        end
    elseif(strcmp(resultsOfTestSet{k},'Abnormal'))
        if(strcmp(testSet{k},'Abnormal'))
            % TRUE NEGATIVE
            TN = TN + 1;
        else
            % FALSE NEGATIVE
            FN = FN + 1;
        end
    end
end

The above case assumes Normal as the 'Positive' resultant class. However by just flipping the compare statements, I can get alternate values.

for k=1:length(resultsOfTestSet)
    if(strcmp(resultsOfTestSet{k},'Abnormal'))
        if (strcmp(testSet{k}, 'Abnormal'))
            % TRUE POSITIVE
            TP = TP + 1;
        else
            % FALSE POSITIVE
            FP = FP + 1;
        end
    elseif(strcmp(resultsOfTestSet{k},'Normal'))
        if(strcmp(testSet{k},'Normal'))
            % TRUE NEGATIVE
            TN = TN + 1;
        else
            % FALSE NEGATIVE
            FN = FN + 1;
        end
    end
end

So after running it for both cases, I get the following values.

For Abnormal being my Positive case

precision = 96.5517
accur = 95
recall = 87.5000

For Normal = Positive case

precision =  94.3662
accur = 95
recall = 98.5294

So how do I calculate the combined result, prec, accu, and recall? Or, am I just missing the point and you just calculate it for one class at a time, like the one you are focusing on.

The reason I am asking is because now lets say I have a set with multiple class outcomes in my decision tree. This is where I realized I have to pick a class to determine as my positive, or just look at classes individually.

Here is a similar set with 3 class possibilities. Again, how do I calculate for the whole data set? Or is an individual class thing? Or do you calculate the individuals and then come together with a total for the whole decision tree.

pelvic_radius   degree_spondylolisthesis    class
82.45603817    41.6854736      Abnormal
114.365845     -0.421010392    Normal
95             25              Perfect


texts: DBPedia as Table not having all the properties

texts:
I browsed a sample for available data at http://dbpedia.org/page/Sachin_Tendulkar. I wanted these properties as columns, so I downloaded the CSV files from http://wiki.dbpedia.org/DBpediaAsTables.

Now, when I browse the data for the same entity "Sachin_Tendulkar", I find that many of the properties are not available. e.g. the property "dbpprop:bestBowling" is not present.

How can I get all the properties that I can browse through the direct resource page.


texts: loss increases but accuracy and macroF1 are still stable and don't change dramatically

texts:
I have a classification task with 2 classes. The dataset is imbalanced. When I train the model, at some point, the loss of test dataset starts to increase but the values of accuracy and macro-f1 don't change dramatically and are still stable. It's so weird because I thought that overfitting should happen here and the accuracy of the test gets lower. I thought that it might be the problem of imbalanced classification and the classifier is trying to predict all the samples as one class, but it's not.
The value of stable accuracy and macro-f1 are around 0.69 and 0.66 respectively. If it predicts all the samples as one class, the accuracy and the macro-f1 values will be 0.66 and 0.39. I don't know what's happening in the model.


texts: if gbm overfit, will variable importance calculation be accurate

texts: If we are building GBM and we have too many trees that leads to overfitting, will the variable importance be affected?


texts: How to find the relationship between two categorical variables?

texts:
I'm building a model and trying to get the relationship between two multi-level categorical variables.

For example, I want to know the relationship between race and likelihood of graduate from collage, we have 5 races and YES or NO for graduation. How to find the which race have higher likelihood or which have lower?

Thanks


texts:
I'm working on a unusual issue (for me) and I need some advice.

My goal is to have a recommendation algorithm that propose tags for a document, based on all the previously tagged documents. For example, if I upload each month an invoice from "stackoverflow", and I apply each time the same tags ("stackoverflow" and "invoice" for instance), the algorithm should recognize the document as similar to previous stackoverflow invoices and suggest the same tags to the user.

I was thinking about extracting some features of the document ( text, sender's email...), compare the similarity with previous documents and suggest the tags of the most similar documents.

The tags are user-defined, and thus not a finite set, so the algorithm should learn continuously. This is my issue, because I will have to store all the documents in a database and evaluating the similarity between a new document and all the previous ones will eventually be very expensive as the number of documents in my database grow.

On the other side, incremental learning algorithms can learn continuously without a massive database, but don't support an non-finite set of output classes.

So, as I see it, I have 2 options :

massive database with all documents, and evaluating similarity
between a new document and each document in the database : not very
scalable, possibly time-expensive
re-train from scratch a classification algorithm every time a new document is uploaded : I will still have a massive database, but training can be done in the background. This is the best of the 2 options, but I'm wondering if there could be a better way.

Any advice about more appropriate algorithm would be greatly appreciated. If it matters, I work with Python.


texts:
Given an image of a worker, I need to verify if he/she is wearing the company's uniform.
I tried to Google, but either the search results are about uniform sampling or some object detection tutorials.
Might you help me to point out some starting points, ideally on some similar projects published somewhere?
Many thanks,


texts: Imputation of missing values and dealing with categorical values

texts:
I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean and median are not the solution here.

I tried to research other methods for that but none of them works since I have many categorical variables. I also tried to use one hot encoding to convert categorical variables to integers but I am not sure if that is a solution in my case since from only 1 categorical column I would get 600 new columns. If I do the same with other categorical columns, I would get many millions of new columns. One of the categorical columns is URL string and it is different for every row, so I have 10 million different URLs for example.

The other categorical column is a description and it is also different for every row. I could probably remove the URL column, but I can't remove description, title, location and others for example. I tried PCA, but it also doesn't work with categorical data. I have missing data for both categorical and integers/floats values. Would get_dummies method be a good approach to deal with this? For missing values imputation I tried KNN and maximum likelihood but I am getting errors due to categorical variables. Missing data is completely randomly missing.

Do you have any suggestions how to approach this problem and also which packages should I use?


texts: Which data mining or machine learning algorithm would be appropriate for learning ordered frequent patterns?

texts:
I have a dataset as (var1, var2, out), where the ordered pair <var1, var2> gives out. Most of the frequent pattern mining algorithms like the Apriori and FP growth algorithms does not preserve the order of var1 and var2.

Which are some of the available pattern mining algorithms (may also be a NN trick), to find association rules between ordered pair <var1, var2> and output variable out?

Thanks.


texts: reinforcement learning: PPO vs. DDPG vs. TRPO - difference and intuition

texts: I know there is a lot of blog talk about the PPO, DDPG and TRPO, but I am wondering would it be possible to explain the differences of these methods in layman's term? What's the intuition behind them without using the complex mathematics? Thanks!


texts: How can I load file from my local drive in keras using get_file?

texts: I am using google colab and I need to load data from my local drive. How I can do that using get_file? what should be the origin?


texts: How do I predict continuous value from time series data?

texts:
I have a glove that have 2 IMUs (Inertial Measurement Unit) attached to it. It can give the rotation value as Quaternion (x,y,z,w) and acceleration of the hand (x,y,z).
I put it on my left hand and I want to predict the position of the hand relative to some fixed point on my body (e.g. my head).

When I collect the data, I just use the Vive controller to track my hand.

Right now I just feed the value into a simple model like SVM to predict (x,y,z) of the hand relative to my head.
The problem is that the output is jumpy as the sensors are not stable enough. And because it doesn't take into account data in the previous timestep.

I want to utilize time series data as my data is time series in nature.

Which algorithms I should use if I'm treating my data as time series?
You can suggest me things I should learn about too.


texts: Evaluation of semantic segmentation network with mAP

texts:
I am interested in evaluating a semantic segmentation network. I've seen lots of challenges such as PASCAL VOC use the mean average precision metric(mAP). I understand how this would work with an instance detection approach, however I am unsure how it works for networks which just provide a  class label to each pixel. Should I treat each image as a single instance of each class?

Thanks


texts: Which of the following linear regression model is better?

texts:
Consider 2 regression models on the same data set.
Model 1 : $R^2 = 90$% , $R^2(adjusted) = 80 $%, $R^2(pred) = 70$%
Model 2 :  $R^2 = 60$% , $R^2(adjusted) = 59 $%, $R^2(pred) = 58$%
In the first model , all  values are very high but the difference between them is also very high.
In model 2, the values are low but difference between them is less.
Which of the above 2 models is better and why?


texts: How can I use transfer learning to predict height given age in Japan, using a model developed with USA data?

texts:
Suppose I have a (training) set of $n$ observation $\{(Y_i^{(U)},X_i^{(U)})\}_{i=1}^n$ of age $X_i^{(U)}$ and height $Y_i^{(U)}$ from people in the USA.
Now suppose I also have a (test) set of $m$ observations $\{X_i^{(J)}\}_{i=1}^m$ of age $X_i^{(J)}$ only from people in Japan, where people are shorter on average.
I want to predict the heights of people in Japan in the test set using transfer learning from the USA dataset. Suppose for simplicity the USA data is well-fit by the standard simple linear regression model
$$
Y_i^{(U)} = \beta_0 + \beta_1 X_i^{(U)} + \varepsilon_i, \quad \quad i=1,\dots,n,
$$
with the usual simple linear regression assumptions for the error.
How can I use transfer learning of the USA data to make predictions of height given the age data in the Japan dataset $\{X_i^{(J)}\}_{i=1}^m$?
Are there any theoretical results on the accuracy of the predictions given by transfer learning?


texts: Can someone provide me the code of the MiLoF(Memory Efficient Local Outlier Factor) algorithm?

texts:
I have to code the MiLoF algorithm for detecting outliers in an unsupervised manner using non-stationary data. I am attaching the paper which explains the algorithm here. However, there are many things involved and things are becoming very complicated once I start coding it. It will be great if anyone can help me with coding this. Coding is to be done in Python.

https://www.researchgate.net/profile/James_Bezdek/publication/305825504_Fast_Memory_Efficient_Local_Outlier_Detection_in_Data_Streams/links/5a1ec39baca272cbfbc06f3d/Fast-Memory-Efficient-Local-Outlier-Detection-in-Data-Streams.pdf


texts: lags and rolling window in Azure AutoML Time series forecasting

texts:
I'm following this tutorial to try Machine Learning AutoML Forecasting.

In the several parameters we can submit to the AutoML experiment, we have these ones:

target_logs;
target_rolling_window_size;

Can you explain with an example how the several forecasting algorithms works when these two parameters are set?

Thank you

automl_advanced_settings = {
    'time_column_name': time_column_name,
    'max_horizon': max_horizon,
    'target_lags': 12,
    'target_rolling_window_size': 4,
}
automl_config = AutoMLConfig(task='forecasting',
                             primary_metric='normalized_root_mean_squared_error',

                             experiment_timeout_hours=0.3,
                             training_data=train,
                             label_column_name=target_column_name,
                             compute_target=compute_target,
                             enable_early_stopping = True,
                             n_cross_validations=3,
                             verbosity=logging.INFO,
                            **automl_advanced_settings)


texts: Why I get AttributeError: 'float' object has no attribute '3f'?

texts:
I am getting this error:

AttributeError: 'float' object has no attribute '3f'

I don't understand why I am getting it, I am following the example straight from the book "applied text analysis"

The chunk of code in python is:

total = sum(words.values())
for gender, count in words.items():
pcent = (count / total) * 100
nsents = sents[gender]
print(
"{0.3f}% {} ({} sentences)".format(pcent, gender, nsents)
)

I see that pcent clearly will return a float, why the author tries to apply .3f what  am I missing?


texts: What is the optimal value of a Markov Decision process with Single actions at each state?

texts:
I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows:

There are two states ($a$ and $b$) stepping to $a$ is terminal.
All rewards are zero, discount for stepping from $b \to b$ is $1$, and discount for all other steps is zero
The possible scenarios are: $a\to b$ (probability $1$), $b\to a$ (probability $p$) and $b\to b$ (probability $1-p$).

The first question I have is whether or not its true that the optimal values for each state here are zero? If not how did you derive this?

Second question I have is if we want a parameter $\lambda$ and have a feature $\phi$ such that

$\lambda \times \phi(a)$ and $\lambda \times \phi(b)$

approximate the optimal values at the states a and b and we attempt to approximate such a $\lambda$ by means of TD(0) starting with $\lambda_0 = 1$ how can I find the expected value of $\lambda$ after one episode of updating in terms of $p$? ($E[\lambda_T]$ where $T$ is a random number representing the duration of the episode)

By an episode I mean if we go back from state b to a the episode is over.


texts: I tuned my hyperparameter with random search and i used cv=5. Is it important to validated the hyperparameter with model and testdata or is it okay to use the given back accuracy from random search?


texts: Difficulty interpreting word embedding vector similarity (spaCy)

texts:
I calculate vector similarities like this:

nlp = spacy.load('en_trf_xlnetbasecased_lg')
a = nlp("car").vector
b = nlp("plant").vector
dot(a, b)/(norm(a)*norm(b))
0.966813

Why are the vector similarities so high for unrelated words for the embedding? This is not the only pair for which they are abnormally high. I also had a similar experience with fastText, so I am wondering, am I misunderstanding something?

Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?


texts:
I hope I can make this clear with few lines of code/explanation.
I've a 16K list of texts, labelled over 30 different classes that were ran through different classifiers; my Prediction and the Ground truth match on average at 94%.
I am now after measuring something extra (not sure what should I measure on top of an F1_score minimum as I'm still learning, though) and I came across the log_loss from sklearn, which result I understand it range between 0 and 1.
When ran against my prediction, however, the results is 1.48xxx, which is in fact higher.
In trying to understand what was wrong.
I have explored the result of ComplementNB.predict_proba that is required for the log_loss, and the value matches the one of my prediction array.
Below some code:
from sklearn.metrics import log_loss

y = ... # This is my array of value that is my source of truth

labels = numpy.unique(y)
label_ary = [idx for gt in y for idx, lbl in enumerate(labels) if gt == lbl]

print(f'The log loss is {log_loss(label_ary, clf.predict_proba(X.toarray()))}')

Whether I use label_ary or y, in both the circumstance I am obtaining the same value, meaning that some conversion inside the log_loss is already happening.
I'm not sure whether it me misinterpreting the results, or the specific of the function.
What am I doing wrong?
Thanks


texts: How do I store a question-and-answer combination for machine learning?

texts:
I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the question word-by-word and have the algorithm note some of the previous words to implicitly learn how these words combine and affect the answer. For example, the question "How's the weather in Philadelphia?" gets broken up into "How's","the", "weather", "in","Philadelphia".

The problem is, not all my questions or answers are of the same length. I could have questions like, "How's the weather?" or "What color of pants am I wearing?" Plus, the vocabulary consists of about 300 words, making a one-hot encoding scheme based simply on these words impractical.


texts: Text mining in Amazon product review using R. I wasn't able to extract the particular product's review

texts:
Text mining on Amazon product review using R Program. I wasn't able to extract the particular product's review(i.e.If iphone 11 has 6k review, I need to extract all of it.) I'm getting only one column labelled x.

Please let me know where I need to make necessary changes. I need those for performing sentiment analysis.
install.packages("rvest")

library(rvest)

install.packages("xml2")

library(xml2)

install.packages("magrittr")

library(magrittr)

url <-"https://www.amazon.in/Apple-iPhone-11-128GB-Black/product-reviews/B07XVLW7YK/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews"

apple <- NULL

for(i in 1:100)

{

murl <- read_html(as.character(paste(url,i,sep = "=")))

rev <- murl%>%

html_nodes(".review-text")%>%

html_text()

apple <- c(apple,rev)

}

write.table(apple,"apple1.txt",row.names = TRUE,col.names = TRUE,fileEncoding = "utf-8-mac")

apple<-read.csv(file.choose())

View(apple)


texts: t-SNE: Why equal data values are visually not close?

texts:
I have 200 data points that have the same values on all features.

After t-SNE dimension reduction they doesn't look so equal anymore, just like this:

Why aren't they on the same point in the visualization and even seems to be distributed in two different clusters?


texts: How to build this simple generative model?

texts:
I'm watching a lecture and it shows (schematically) a model with 2D input (sampled from Gaussian) followed by 12 conv layers, and output an RGB image.

That's all about it, no training at all! he relied on initializations only. He did it just to illustrate the concept of NN power to produce complex distributions.

How can I dot that in Keras?


texts: inputs and outputs of a fully connected layer for a note classifier

texts: i'm trying to make an object classifier of different musical note types and I'm having a problem understanding fully connected layers. for example I have a 3d array of binarized images (WxLxNo.OfImages) and i pass that array to the conv layer, relu and pooling. a lot of the videos and other tutorials said that I make a vector using the images but is the input per image, category or the whole training data? My second question is how many neuron should be on the last layer? should it be the number of categories in my training dataset?


texts: Is Recursive Feature Elimination finding best features subset?

texts:
On a set of 9 features I have applied Recursive Feature Elimination (RFE) algorithm using SVM estimator, following approach from (1). When requesting a subset of size 1 to be found, then RFE returned feature X.
However, when I trained SVM over each feature individually, I found another feature Y to have higher accuracy than SVM trained over X.
I thought that RFE finds features with the highest accuracy.
Is my understanding of RFE wrong?
(1): Gene selection for cancer classification using support vector machines


texts:
I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features.
Moreover, the documentation is what looks more ambiguous. Here are the first two lines from the documentation:

Encode categorical integer features as a one-hot numeric array. The input to this transformer should be an array-like of integers or
strings, denoting the values taken on by categorical (discrete)
features.

First line says it

encodes categorical integer features

and the next line says

input should be array like of integers or strings.

When i tried it, i still got the value error.
print(X.columns)
encoder = OneHotEncoder(categorical_features=[1,4,5])
encoder.fit(X)

Index(['age', 'sex', 'bmi', 'children', 'smoker', 'region'], dtype='object')
ValueError: could not convert string to float: 'female'

I am aware of the means to handle encoding of string features with LabelEncoder, ColumnTransfomer and pd.getDummies() but specifically want to understand about this.


texts: How SHAP value explains contribution of features for outliers event?

texts:
I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). I already created a simple synthetic dataset with 7 outliers.

I didn't get how 4.85 calculated as output model.

outliers_df = temp_dataset.loc[synth_data_df[outdet.label_feature] == 1]
outliers_df

#Here is the frame of 7 outlier cases I input to SHAP:

+----+-------------+--------------+---------------+
| id |      NF1    |      CF1     |outlier_scores |
+----+-------------+--------------+---------------+
|  1 |       904   |     2        | -0.192221     |
|  2 |       951   |     3        | -0.203116     |
|  3 |       125   |     5        | -0.076722     |
|  4 |       755   |     2        | -0.237189     |
|  5 |       983   |     2        | -0.213205     |
|  6 |       906   |     3        | -0.202074     |
|  7 |       999   |     5        | -0.266428     |
+----+-------------+--------------+---------------+

I applied IsolationForest using class OutlierDetector in my implementation to isolate outliers:
outdet = OutlierDetector(synth_data_df, label_feature="label",
                         normalize_outlier_scores=True, standardize_outlier_scores=False,
                         n_estimators=100, max_samples=1., contamination=0.1)
outdet.train_IF()

I got SHAP for each rows:
outdet.shap_values_outliers = outdet.explainer.shap_values(outliers_df)
outdet.shap_values_outliers
array([[-6.0205, -0.7785,  0.    ],
       [-6.3379, -0.7031,  0.    ],
       [-2.9014, -1.0911,  0.    ],
       [-6.6259, -1.1502,  0.    ],
       [-6.445 , -0.8171,  0.    ],
       [-6.2393, -0.7787,  0.    ],
       [-8.1958, -0.1862,  0.    ]])

This is simple shap.force_plot() for first row for 3 features. I already concatenate outlier_scores to the frame after I trained my model with NF1 and CF1 as follow:

How to get $\text{basic}_\text{value} = 11.65$ via a script, as explained here?

General interpretation considering $\text{basic}_\text{value} = 11.65$ and $\text{Model}_\text{output} = 4.85$ for first event or observation considering no features pushing the prediction higher are shownred by red color?

Reason for getting $\text{SHAP}_\text{outlier_scores} = 0$ ?

Edit
I would also like to share the following plot, which may help to see if we can understand the contribution of one of these two features to explain outliers like  $\text{SHAP}_\text{NF1} = [-8, -6]$:
outdet.plot_shap_values()


texts: Why is the accuracy on the test dataset very low when training a neural network on an IMU dataset?

texts:
I am trying to train an IMU (Inertial Measurement Unit) dataset. The dataset contain 6 features (3-gyro, 3-accelerometer) and 1 label column. I have build a neural network via Conv1D, LSTM and Dense layers respectively (in Keras Framework). The train set accuracy achieve %89 accuracy but validation accuracy is stucked on %27. I know it means overfitting. I tried to use dropout layers and hyperparamter tunning but validation accuracy is still low. The model can't predict good solutions if I test on untrained data(test set). What can I do to increase accuracy?
You can check my notebook and dataset here.
Will the accuracy on the test dataset increase if we add noise on the tabular data?
Or, do I need to have more data to make a generalization and have good test accuracy?


texts:
guys!

I have a dataset with a bunch of costumer-behavior features and the output being "Churned"/"Not churned". I applied a simple Random Forest Classifier and got a nice performance.

With this, I can predict whether or not a given user will churn. But I need to understand what are the patterns among churned users and patterns among non-churned users. How could I achieve that? (Where I could present something like "Usually, users that churn do this, that and that")

PS: No need for a full explanation, I'd happy enough if you give me some directions to what to study so I can achieve this knowledge

Many thanks in advance!


texts:
I built a model for time series in order to forecast new values.

What is the best way to choose the correct model?

Are better criteria like AIC and BIC or the comparison between prediction errors? In this last case I have to split dataset in train and test while in the first case is not necessary, reducing time of implementation.

Thanks!


texts: How to store and analyze classification results with Python?

texts:
I'm applying ML for classification task in Python using sklearn/pandas. I'm going to try various things to get the best results, and I wonder how do I effectively store and analyze all the parameters and results of the classification? Parameters include:

Number of training examples (which can be extended as I get more labeled data).
Set of features.
Classification algorithm.
Algorithm hyperparameters.

Results include:

Precision/recall for each of the classes.
Overall precision/recall.
Support for each class, etc.

Of course, I can manually copy the parameters and results to an Excel spreadsheet every time, but it's not an optimal solution. Are there any Python libraries (or modules of sklearn/pandas) which allow to easily store and display the parameters and results for later analysis? How do you solve this task?


texts:
Given a stream of data containing a vector of independant variables $X$ and a dependant variable $Y$, I would like to capture the probability distribution of $Y|X$, such that a synthetic data stream can be created that is realistic. In other words:

Model a set of distributions  $Y|X$
Sample $Y|X$ randomly to produce 'realistic' synthetic data.

I can assume the target distribution is always normal. However, the mean and the variance will certainly depend on $X$.
$Y|X \sim N(\mu(X),\sigma^2(X))$
There are various linear regression techniques that produce a normal distribution as an output (GLM's, Bayesian LR). However, these algorithms always assume a constant variance $\sigma^2$.
I know I can approach this problem by training a mixture density network(https://cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.6-MixDensityNetworks.pdf). But I feel this may be overkill for my application. Can anyone recommend some traditional ML (non-DNN) techniques for approaching a problem like this? Could Quantile regression be applicable somehow?


texts: Can anyone explain what is Saturating Gradient problem? It would be nice if anyone can provide math details as well. Thank you in advance!


texts: How to interpret upper-triangular matrix of cosine similarities


texts: Using LSTM's on Multivariate Input AND Multivariate Output

texts:
I have a very small dataset, only about 40 rows, that has historical usage data for a few categories (roughly 20). I strongly suspect that these categories are dependent in a partial-zero-sum-game fashion: if the usage of one category goes up, I'm expecting that of another to go down. My goal is to predict the next row of all categories.

I've looked at this post, but it's not predicting multiple variables. I also looked at this post, but it's still univariate output (albeit multiple time steps) and multivariate input. So far, I've been basing my approach on the typical LSTM post here at machinelearningmastery, but it's also a single-output-variable example, and a number of the functions used, such as scaler.inverse_transform don't appear to broadcast very well. I'm even having difficulties trying to scale back my full example to match his!

Any tips for scaling LSTM's up to multivariate output? Can the keras LSTM do this natively? If so, how would the code change?

Thanks for your time!