texts: Interpretation of the loss function for word2vec

texts:
I am trying to understand the loss function which is used for the word2vec model, but I don't really follow the argumentation behind this video https://www.youtube.com/watch?v=ERibwqs9p38&t=5s, at 29:30.

The formula which is unclear is the following:

$J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(p(w_{t+j}|w_t))}.$

$T$ is the number of words in the vocabulary
$w_t$ is a given word and we try to calculate the probablity that another word $w_{t+j}$ occurs within a window of +/- $m$ words ahead.
$\theta$ is the solution we're after. It is essentially a $2*d*Tx1$ dimensional vector which contains all the columns of the matrices V and U.

At a first sight it looks all pretty clear: we're iterating though the whole vocabulary and for each (fixed word then), we add up all probabilities that another word occurs within a window around that fixed word.

However, it fails apart for me when I consider that a word $w_t$ occurs in many positions in the corpus and might occur multiple times with a different word. E.g, the words 'deep learning' often occur together, which indicates that there's a contextual relation between them. Why would we only count them twice? It seems like the formula above is counting each pair $p(w_{t+j}|w_t)$ just twice (e.g. once for $p(deep|learning)$ and once for $ p(learning|deep)$). IMO we should need a correcting term that adjusts for the missing 'frequency', e.g.

$J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(\lambda(w_t,w_{t+j})p(w_{t+j}|w_t))}.$

In the case where $\lambda(x,y) = 1$, we get the formula above, but we could also be free to chose a function that boosts frequent occurrences (pairwise). The formula above could then be seen as a special case when you don't care that words that occur more often together get a boost.

On the other hand, when the formula above already accounts for multiple occurrences, then where is this visible?
The author then continues and defines $p(o|w)$ as $exp(o^T*w)/\sum(exp(u^T*w))$. In this particular case, I don't see that we're counting the dot-product as many times as the word $o$ is in the neighbourhood of the word $w$. Maybe the choice of $p$ is just very simplistic and represents a model where the fact that 2 words are in the neighbourhood (just somewhere in the corpus) is enough (bag of words model???). It's hard to see then that such models deliver good performance in NLP though.


texts: Are these images too 'noisy' to be correctly classified by a CNN?

texts:
I'm attempting to build an image classifier to identify between 2 types of images on property sites. I've split my dataset into 2 categories: [Property, Room]. I'm hoping to be able to differentiate between whether the image is of the outside of some property or a room inside the property.

Below are 2 examples of the types of image I am using. My dataset consists of 800 images for each category, and then a training set of an additional 160 images for each category (not present in the training set).

I always seem to be get reasonable results in training, but then when I test against some real samples it usually ends up classifying all of the images into a single category.

Below you can see the model I am using:

IMG_HEIGHT = IMG_WIDTH = 128
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(32, (11,11), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH, 3), padding='same'),
    tf.keras.layers.MaxPooling2D(11, 11),
    # tf.keras.layers.Dropout(0.5),
    # Second convolutional layer
    tf.keras.layers.Conv2D(64, (11, 11), padding='same', activation='relu'),
    tf.keras.layers.MaxPooling2D(11, 11),
    # tf.keras.layers.Dropout(0.5),
    # Flattening
    tf.keras.layers.Flatten(),
    # Full connection
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

from tensorflow.keras.optimizers import RMSprop

model.compile(
    optimizer=RMSprop(lr=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# now train the model
history = model.fit_generator(
    train_generator,
    validation_data=validation_generator,
    steps_per_epoch=75, #100
    epochs=5, # 15, or 20, and 100 steps per epoch
    validation_steps=50,
    verbose=1
)

Can anyone suggest the possible reasons for this? I've tried using different epochs and batch sizes, augmenting the images further, changing the Conv2D and Pooling layer size but nothing seems to help.

Do I perhaps not have enough data, or are they bad images to begin with? This is my first foray into ML so apologies if any of questions seem obvious.


texts:
So I saw this technique which is explained as an improvement for gradient descent algorithm,
which is based on coordinate transformation:

What I don't understand is why in the second slide multiplying by P from the left changing the $\theta^\_ $ into $\theta $
It is taken from here, 01:05:00:
lecture 4


texts:
I want to try the IBM speech to text API. I created an IBM cloud account and went to https://console.bluemix.net/catalog/services/speech-to-text. I see the error message:

  Please create a space in the selected org or choose another org.

Where can I create such a space? I wasn't able to find it yet.


texts: What are the assumptions of linear regression

texts:
Can anyone explain the assumptions of linear regressions?

If possible with an example?

Is this really important to check these assumptions before proceeding?


texts: Shall I install anaconda, with Python3 and R already installed?

texts:
On my Ubuntu 18.04, I have Python 3 and R installed. I am about to study some data science and just found https://www.anaconda.com/distribution/.
anaconda comes with Python and R and some packages. Will installing anaconda conflict with my existing installation of Python 3 and R?
Shall I install anaconda, or shall I install the packages manually and individually on demand?
How do people in data science install the tools?


texts: Determining usage/context of terms/acronyms/abbreviations in python

texts:
Background: I work for a major airline in the United States and have been given a pretty significant text mining and machine learning task. Since there are several unique terms, acronyms, and abbreviations in aviation, I have written a program that reads through the documents. The program is using 4 groups of terms; WORDS, STOPS, AIRPORTS, and SYNONYMS. WORDS, STOPS, and AIRPORTS are all sets where the first two were initialized from NLTK and the last was initialized with a manually built list of airports that my employer services. SYNONYMS is a nested dictionary 'trie' style structure where the branches are misspellings and the leaves are the terms that they represent. it was initialized as an empty dictionary. I have also created a command prompt UI that informs me when it finds a word that it doesn't know and allows me to correct the word or add it to one of the aforementioned groups. The documents are being tokenized by spaCy and fed into scikit-learn machine learning algorithms.

Problem: I've started noticing that some of the words that I previously accepted in WORDS or SYNONYMS are showing up with alternate meanings and could have a drastic impact on how well the algorithm operates. For example the word act can be the verb act; or it can be an abbreviation for actuator or active; or it can be an acronym for auxiliary central tank or a couple other things. At present, I'm using the UI to manually correct or ignore these occurrences, which is time consuming, cumbersome, and prone to errors.

Desired result: What I'm looking for is a method of determining whether 'lt' means left or light depending on context (lt wing vs. warning lt) with minimal manual entry. At a minimum I would like some high level ideas/discussion on how this may be accomplished in python. Optimally, I would like some example code, preferably using the same tools I already have; spaCy, NLTK, and sklearn but I am flexible and willing to learn.

Notes: I've considered a couple methods of doing this but I haven't been able to come up with any ideas strong enough to try. For example, I've considered marking the terms with the pos inside the sets by concatenating it with the word before adding it like 'rat:noun'. The issue with this is that rat:noun could be an animal or it could be an abbreviation for ram air turbine, which is also a noun.


texts: Differentiate twins, triplets images in Computer vision field

texts:
https://en.wikipedia.org/wiki/Computer_vision
https://en.wikipedia.org/wiki/Twin
https://en.wikipedia.org/wiki/List_of_triplets
Will there be challenges in Computer vision field to differentiate between twins, triplets images?


texts:
I am working on an LSTM autoencoder in keras. The aim here is to obtain a latent space representation for the time sequences which I intend to use for clustering.

My input sequences (each feature) have very low variance among them.
The input before normalization looks something like this:

This is one of the sequences, it has 4 features(the columns) and variable length (in this case 11 the number of rows).

The other sequences range from 11 to 200 in length. The number of features obviously remain constant. After normalization over the entire feature space (normalizing on each feature individually) these subtle differences in input sequences become even smaller. And I think the autoencoder is assuming this is noise and is not learning it (or rather behaving like a denoising autoencoder).

Any thoughts on how I can scale the data better? Should I make any changes to how I am treating the problem statement?

PS:

There is no problem with the code as I was able to generate very good latent representation on a toy dataset whose features were more evenly spaced out.
I have tried standardization (z score- subtracting by mean and dividing by standard deviation) but problem still persists.


texts: MOOC or book on Deep Learning in Python for someone with a basic knowledge of neural networks

texts:
I know the concept of a neural network, and I followed the Machine Learning course by Andrew Ng on Coursera, so I also coded some simple ones. However, I miss all the new tricks which are useful to prevent overfitting of Deep networks, such as for example:

using ReLU instead than tanh neurons
using dropout
more advanced learning methods than just vanilla stochastic gradient descent

etc. I would like to follow a MOOC which tells me how to use Keras for Deep Learning (I like Keras very much because IMO is much easier to understand than other packages, but I'm open to suggestions). I would also be content with a book, but I'd really prefer a MOOC. Doesn't have to be free. Can you indicate me one? The application is Data Science, but generic Deep Learning would do.

EDIT: to provide more context to the question, my main applications would be Internet of Things Analytics, i.e., applications on cloud platforms which collect real-time, streaming sensor data from industrial machines and allow to estimate their actual performance, predict the probability of a failure and the time before it happens, detect anomalies, etc. I don't need to develop the cloud platform: I just need to develop the "core" Analytics. Think of it as just applying Deep Learning to Time Series or to Classification problems. However, methods which can easily be retrained when new data arrive, without having to go through the full dataset again, would be preferred.


texts: How are precision and recall better metrics than accuracy for classification in my example?

texts:
I'm trying to understand precision and recall with an intuitive example, but my calculation doesn't seem right.
For example, there are 8 red balls and 2 blue ones. I'm stupid and just predict all of them are red. The accuracy would be 0.8 and looks good. But it doesn't reflect that it's just a lucky, stupid guess, and I missclassify all of the blue balls. If I'm not mistaken, we have 8 true positives (TP = 8), 2 false positives (FP = 2), and neither true nor false negatives (TN = 0, FN = 0) in this case. Then isn't precision = TP/(TP+FP) = 0.8 and recall = TP/(TP+FN) = 1? Don't they still look good?


texts:
I need a model that is able to receive as input an image of a nutritional information chart and tell the level of sugar that the product has. It would be a 3-class classification problem (low if sugar is below 5g, medium if it's between 5 and 22.5g and high if it has more than 22.5g). I have prepared all the data and I have 16000 images in total. However, I'm not able to train a proper model with the data. I have tried a simple convolutional neural network of 3 convolutional layers, the pretrained inception resnet v2 from keras, and even an attentional model (Github). The result is always the same, an accuracy equal to the proportion of samples from the most common class. So these models are unable to solve the issue and just bet for the most likely.
What kind of network could be able to solve this problem? I have never dealt with networks that have to "read" and classify text.


texts: Best practice for developing using Spark

texts:
I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ on my local machine.

I am wondering what the best way would be to go about development of an application to run on the data that is in the cluster. Things on my mind for consideration are the following;

IntelliJ is a nice IDE but I can't store the volume of data I want to look at on my local machine
I can run spark-shell on the edge node but its not great for data exploration and development

How is it best to bridge this gap between data and a nice development environment?


texts: Growth function of a 6-dimensional linear classifier

texts:
In our course, we are dealing with a d-dimensional classification problem
($\chi = \mathbb{R}^{d}$ as our input space, and $y = \{-1,+1\}$). Our hypothesis class $H$ consists of all hypotheses of the following form:
$h(x) = a\cdot \text{sign}(x_i - b)$, where $i = \{1,2,\dots,d\}$, $a \in \{-1,+1\}$, and $b\in\mathbb{R}$.

We have already shown that the growth function $m_{H}(3) = 2^3$ for $d=2$ by showing all 8 possible dichotomies for three chosen points. We further know that for a $d$-dimensional linear perceptron, the VC-dimension is always equal to $d+1$.

We know want to show that for $d=6$, $m_{H}(7) < 2^7$, i.e. that the VC-dimension of our hypothesis class is lower than 7.

Could you help us out with this?
Thanks a lot!


texts: Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)

texts:
I have a large Excel data set which I load as Pandas. In one column I have repeating strings/numbers (the information can be a text and or integers), in another column I have a unique identifier of that row, there is also a column with a due date. Using sort_values I can group the repeating values and sort according to the due dates. But now I would like to add a column containing numbers to get a ranking of the due dates of the first column.
example data:

[[Item, Kind, ID, Client, Duedate],[A, 3, 689, XY, 01.04.2019], [A, 3, 689, XY, 02.04.2019],[B, 3, 689, XY, 01.04.2019],[C, 3, 689, XY, 01.04.2019],[A, 3, 689, XY, 01.04.2018],[C, 3, 689, XY, 01.04.2019]

I sort them so that the output is all the A's first, then all the B's, and within that group they are sorted according to priority. But now I want to extend my data set into looking like this:

[Item, Kind, ID, Client, Duedate, **priority**],[A, 3, 689, XY, 01.04.2018, **0**],[A, 3, 689, XY, 01.04.2019, **1**], [A, 3, 689, XY, 02.04.2019**2**]

if tried looping over the values, but it doesn't seem to work, and I could not find anything on Google to help me. Any hint would be much appreciated, I am stuck with this for 10 days now..


texts: ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required

texts:
I am new to this logistic regression and finding this error. Please help.
solver='lbfgs'
def lr_model(X_train, y_train):

    # import the LogisticRegression class
    from sklearn.linear_model import LogisticRegression

    # create the model object
    model = LogisticRegression()

    # fit the model to the training data
    model.fit(X_train, y_train) #error here
    time.sleep(0.1)

    #return the fitted model
    return model

# Test
tmp_model = lr_model(X_train[0:12], y_train[0:12] ) #showing error here
print(tmp_model.predict(X_train[4:5]))
print(tmp_model.predict(X_train[5:6]))


texts: Why does changing the cluster number change the plot in Kmeans?

texts:
This might be a dumb questions but I can't find the answer to it. I don't have the perfect mathematical understanding of kmeans, so apologies if it is.

I'm just wondering why I see a different plot when I change the number of clusters in a kmeans plot?

Here's the code that I'm using:

set.seed(1)
k <- kmeans(data, centers = x)
plotcluster(data, k$cluster)

I vary x to see how the plot looks like. Below are the results for x = 3 and x = 4. Apologies for the poor formatting.

I'm wondering why both plots look different if I'm only varrying the number of clusters. Is it because the principle components that are being show, dc1 and dc2, are different as you change x, so as to maximize the variance displayed?

Another quick question is - can you determine the cluster # by how "neat" the clustering looks in the plot? I know there are various methods of doing it, I'm just wondering if the plot is indicative of good/bad clustering as well.

any help is appreciated!


texts: Classification based on color clustering

texts:
I need to classify some domain specific images by analysing their color distribution.
I have annotated data; this last classification step is supervised.
After some preprocessing and masking and other unrelated tasks, I extract the dominant colors in the image's ROI, by clustering the pixels values with K-means.
I end up with K clusters, their size (i.e. the number of pixels that fall into that cluster) and their centroid (A color is a 3d vector in the Lab* color space, this shouldn't really matter, it could be RGB).
Based on the total number of pixels (which varies from image to image) I convert the absolute cluster size into percentages, and the clusters are sorted by size.
For each image, I end up with features that look like this (for example, with K=3)
image1.png (class A)

Cluster number
Cluster size
Cluster center L*
a*
b*

0
81.3%
19.79
5.58
-6.14

1
14.7%
35.63
8.38
5.50

2
5%
19.84
5.73
-3.35

I don't know how to treat this data and what kind of model to use.
This is because both the color center and the cluster size are a continuous value, and there are multiple clusters.
I have looked into Computing Image Similarity based on Color Distribution but I have continuous colors and not categorical colors.
I have looked into Classification based on a Clustering Result but I have K clusters per image and not just 1.


texts: How to compare 2 elements of a column with each other?

texts:
I am using R. I have a column with 4 elements {car, bus, walk, train}. I would like to compare the car and bus with each other with the help of the 3 other columns. How can I do this?

For example, suppose that the other column is male/female, I would like to know how many are male or female for car and bus.


texts: Selection of Features and Data in Random Forest

texts:
First, I am confused whether at each node in all the trees, do we randomly pick features from the lot to be pitted for best split or does each tree get a random subset of feature and then all the nodes in that tree have to work with those features?

Now, regardless of the case whether we pick random features at tree or node level, why do choose random subset of data for each tree? Because if the features are different at tree/node level, the trees will be different. Why do we need to add another level and make the distribution of data also random to the trees and not just feed the complete original dataset to the trees.


texts: Parameter initialization in a genetic algorithm

texts:
I'm using a neural network in a genetic algorithm. The neural network has 4 inputs (values between 0 and 1) and 4 outputs, corresponding to the probabilities of different actions. The neural network has 58 parameters.

At first, I create a random population : each individual has 58 random parameters. Parameters are chosen randomly with the default method of keras, on python (values between -1 and 1). Is it the good method ? Maybe the best solution needs to have parameters with values higher than 1 for example, but with my method, only values between -1 and 1 exist in the "gene pool". So, a parameters equal to 3.4 can't appear for example.

I tried to train the same neural network with labeled data and gradient descent, in order to have an idea of parameters range. After training the model, I obtained some parameters with values >1, or <-1. I thought I could use those parameters as an initialization for my genetic algorithm. But how could I get different individuals ? If the 1st parameter of my trained model equals 2.5, do I have to set the 1st parameter of the different individuals to 2.5 +- 20% for example ?


texts: Supervised learning for a turn-based game?

texts:
So I have 4GB of turn-by-turn data for many games of a particular strategy game. It appears that most people interested in using ML to build an AI for turn-based games use reinforcement learning to build a model on the fly.

Since I already have really good data, can I use supervised learning to solve this task?

EDIT: I was considering using regression to assign a score to a given action based on its likelihood of eventually resulting in a win; is this the right way to think about it?


texts:
i plotted the ROC curve for  RandomForest Classifier and this is what i get :

The shape looks weird to me , can somebody help me to make sense of it , and is this shape 'common' to not say normal?

thank you.


texts: Is there a counting sketch optimized for intersections?

texts: Popular counting sketches(loglog, hyperloglog, etc) feature natural union operations. Are there any known counting sketches that feature natural intersection operations?


texts: How to recommend items after finding similar users in recommendation system

texts: As the title explains my problem, I'm done with creating a recommendation system that can give me similar users for any given new user. The problem I face is, If I extract the list of products that these similar users liked the most, how do I weight these items and recommend, say only 3 of these items.


texts: Why does my manual derivative of Layer Normalization imply no gradient flow?

texts:
I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through the operation, which can't be true.
Here's my calculations:
$\textrm{Given a vector of real numbers $X$ of length $N$, indexed as $x_i$,}\\
\textrm{we define the following operations:}\\
\mu =\frac{\sum_{k=1}^{N}{x_k}}{N}\\
\sigma = \sqrt{\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N}}\\
y_i=\frac{(x_i-\mu)}{\sigma}\\
\textrm{We seek to calculate the derivative of $y_i$ w.r.t $X$. That is,}\\
\frac{dy_i}{dX} = \sum^{N}_{k=1}\frac{dy_i}{dx_k}\\
\textrm{By the quotient rule:}\\
\frac{dy_i}{dx_j}=\frac{(x_i-\mu)'\sigma-(x_i-\mu)\sigma'}{\sigma^2}\\
(x_i-\mu)'=\delta_{ij}-\mu'\\
\mu'=\frac{1}{N}\\
\implies(x_i-\mu)' = \delta_{ij}-\frac{1}{N}\\
\sigma'=\frac{1}{2}(\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N})^{-\frac{1}{2}}*[\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N}]'\\
[\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N}]'=\frac{1}{N}\sum_{k=1}^{N}2*(x_k-\mu)(\delta_{kj}-\frac{1}{N})\\
\qquad =\frac{2}{N}\sum_{k=1}^{N}(x_k-\mu)\delta_{ij}-(x_k-\mu)\frac{1}{N}\\
\textrm{Note that $\delta_{kj}$ is only 1 when when $k=j$ and 0 otherwise, so we can further reduce:}\\
\qquad =\frac{2}{N}((x_j-\mu)-\sum_{k=1}^{N}(x_k-\mu)\frac{1}{N})\\
\qquad =\frac{2}{N}((x_j-\mu)-\frac{1}{N}\sum_{k=1}^{N}(x_k)+\frac{1}{N}\sum_{k=1}^{N}\mu)\\
\qquad =\frac{2}{N}((x_j-\mu)-\mu-\frac{1}{N}N\mu)\\
\qquad =\frac{2}{N}(x_j-\mu)\\
\textrm{Thus plugging that back into $\sigma'$ we get:}\\
\sigma'=\frac{1}{2}(\frac{\sum_{k=1}^{N}{(x_k-\mu)^2}}{N})^{-\frac{1}{2}}*\frac{2}{N}(x_j-\mu)\\
\quad=\frac{1}{N}(\frac{1}{\sigma})*(x_j-\mu)\\
\quad=\frac{(x_j-\mu)}{N\sigma}\\
\textrm{Now that we have all the components we can return to the derivative $\frac{dy_i}{dx_j}$:}\\
\frac{dy_i}{dx_j}=\frac{(x_i-\mu)'\sigma-(x_i)\sigma'}{\sigma^2}\\
\qquad=\frac{(x_i-\mu)'\sigma}{\sigma^2}-\frac{(x_i-\mu)\sigma'}{\sigma^2}\\
\qquad=\frac{\delta_{ij}-\frac{1}{N}}{\sigma}-\frac{(x_i-\mu)\frac{(x_j-\mu)}{N\sigma}}{\sigma^2}\\
\qquad=\frac{\delta_{ij}-\frac{1}{N}}{\sigma}-\frac{(x_i-\mu)(x_j-\mu)}{N\sigma^3}\\
\qquad=\frac{1}{N\sigma}(N\delta_{ij}-1-\frac{(x_i-\mu)(x_j-\mu)}{\sigma^2})\\
\qquad=\frac{1}{N\sigma}(N\delta_{ij}-1-\frac{(x_i-\mu)}{\sigma}\frac{(x_j-\mu)}{\sigma})\\
\qquad=\frac{1}{N\sigma}(N\delta_{ij}-1-y_iy_j)\\
\textrm{Finally, returning to $\frac{dy_i}{dX}$:}\\
\frac{dy_i}{dX}=\sum^{N}_{j=1}\frac{1}{N\sigma}(N\delta_{ij}-1-y_iy_j)\\
\textrm{Note that we are adding $N$ once (when $i=j$) and $(-1)$ $N$ times, so we can simplify to:}\\
\frac{dy_i}{dX}=\frac{1}{N\sigma}(N+(-1)N-\sum^{N}_{j=1}y_iy_j)\\
\quad=\frac{1}{N\sigma}(-\sum^{N}_{j=1}y_iy_j)\\
\quad=\frac{1}{N\sigma}(-y_i\sum^{N}_{j=1}y_j)\\
\quad=\frac{-y_i}{\sigma}\frac{(\sum^{N}_{j=1}y_j)}{N}\\
\quad=\frac{-y_i}{\sigma}\mu_y\\
\textrm{BUT by properties of data following a standard normal distribution $\mu_y=0$, so}\\
\frac{dy_i}{dX}=\frac{-y_i}{\sigma}0\\
\quad=0\\
\textrm{Which means no gradient flows through a layer normalization}\\\\$
I'm almost certain I've simply made a mistake somewhere, so if someone could point it out I'd greatly appreciate it. Thanks!


texts: Can a trained recognition model be used to generate a sample?

texts:
Suppose we have trained a cat classification network. It takes in an image (as a vector) x and returns $\hat{y}\in(0,1)$. The loss function is the typical cross entropy function. Shouldn't it be possible to now perform gradient descent on the space of images to obtain an example of a "cat picture" that our network really thinks is a cat?
I expect that this process would give us something very ugly, and not actually cat-like, but I am just curious if this works.


texts: Can this problem be solved using deep learning?

texts:
I want to predict price of used cars. I have data like this:

Is this problem suitable for deeplearning or Should I use XGBOOST, RandomForest etc.?

I used one hot approach for nominal features and scaled numeric values between 0-1. Is there anything I can do to improve my data?

I used deeplearning4j on java and predict prices with 10-15% error but I want to predict prices with 1-2% error


texts:
helo,
i have a very weird problem.
my validation accuracy graph is very jumpy, and i dont know how to fix it
this is the graph:

this is a multi label problem. i calculate accuracy with 0.5 threshold
i think that the problem related to my loss function
(i needed to minimize even one class success in the model prediction, and not only when he success in all of them)
i have a costume loss function:
    def loss_function(predictions, labels, batch_size):
        eps = 0.000000000000000000000000001
        losses = 0
        k = len(lebels[0])
        ones = torch.ones(1, 1).expand(batch_size, k).cuda()
        loss1 = -((ones-labels)*(((ones-predictions)+(eps)).log())).sum(dim=1)
        prod = (ones-labels)*k - labels*((predictions+ eps).log())
        loss2 = torch.min(prod, dim=1)[0]
        losses = (loss1 + loss2).sum()
        return losses / batch_size

i hope someone can understand the problem
thank you very much.


texts: How do I convert strings in a dataframe column to int or float

texts:
Encoding the column seperately works ,but when I try it on the dataset directly it throws an error.
The error is something like this.
ValueError: could not convert string to float: 'g'


texts: If this is “big-endian”, then what is this?

texts:
If an ISO 8601 date is “big-endian”, i.e. the most significant bit is first and the least significant is last, then what do we call data structures where the most significant part is first/at the top, and the least significant is last/at the bottom?

ISO 8601 date example: 2019-02-01T2157
data structure example: atom -> neutron -> quark
data structure example: body -> hand -> finger

I would like to be able to refer to this terminology when talking about any natural hierarchy such that any one branch makes ‘common sense’ from left to right (or top to bottom).


texts: Best machine learning algorithm for understanding specific conditional structures

texts:
Being still a bit new to neural networks, I wish to use some form of machine learning but I am not sure which one is most suited.

I have a toy-example of what I am working on. Tips are much appreciated.

Assuming there are chunks of data (not pixels of an image :)) which all are based on a root variable, being 'a'.
This 'a' is the 'thing' and the other variables are aspects.

Also assuming it's a finite set of things and that each thing 'a' has four ('b', 'c' and 'd') aspects.

I wish to make a program that learns that thing ('a':1) often occurs with ('b':0.2), ('c' :0.8), and ('d':0.3).
There is a catch however, I want there to be a certain conditional structure underlying the co-occurrence of a and the respective values of (b, c, d).

For instance:

('a':1) often     (prob. 0.7) occurs with ('b':0.2), ('c':0.8), and ('d':0.3)
('a':1) sometimes (prob. 0.6) occurs with ('b':0.2), ('c':0.5), and ('d':0.7)
('a':1) sometimes (prob. 0.4) occurs with ('b':0.3), ('c':0.4), and ('d':0.2)
('a':1) rarely    (prob. 0.2) occurs with ('b':0.5), ('c':0.3), and ('d':0.8)

For a given ('a':2) this would all be different.
There is no implied necessity for the values of b, c and d to sum to 1. They are mere placeholder values representing aspects (0.2 could be green, 0.8 purple, etc.).

I want to make sure the program understands that there is an inherent dependency between b, c and d, as well as an 'a':x given (b, c, d).

If anyone has suggestions based on this, I would be very grateful.
Thanks!

EDIT: I am working in python and was thinking about dictionaries as the data-structure, but if there are better solutions (also depending on the algorithm perhaps) I am all ears.


texts:
I have a batch of links with the following naming convention:

https://api.cilabs.net/v1/conferences/ws15/info/attendees?page=1; and the page number incrementing.

I would like to extract the data from those web pages into a CSV file.


texts: Macro and micro average for imbalanced binary classes

texts: Micro and macro averaging are metrics for multi-class classification. However, for binary classification when data are imbalanced, it seems that micro and macro precision have different results. My question is that: does it make sense to use micro and macro precision in binary classification problems when classes are imbalanced?


texts: diseases risk prediction using social media comments

texts:
I am trying to build one risk calculator using diabetes patients comments on social media platform.

I have list of comments given by diabetes patients for their experience on diabetes.

When any new user come on my platform, I collect what symptoms he is facing.

I match user symtoms with comment data symtoms and try to calculate the risk score.

approach 1.

(Numbers of comments in database having user entered symptoms) / (Total comments entered by diabetes patients)

approach 2.

(Numbers of comments in database having diabetes as a symptom and also user entered symptoms) / (Total number of comments in database having diabetes as a symptom)

Is there any better approach to calculate the risk score using user entered symtoms and diabetes patient comment data?


texts:
I'm trying to make a "car sound detector" I have the data from https://urbansounddataset.weebly.com/urbansound.html site. Which has 1000 labeled sounds for 10 different classes(Car sound, dog bark, drill..etc). So using a binary classifier I want to detect if a given sound sample is a car -> label 1 or any other sound from that data set -> label 0.

So data is highly unbalanced. 1000 1-labeled car sounds against 9000 0-labeled other sounds.

To make a more balanced set, for example in both test and training sets I can create %50 / %50  split using oversampling(not sure its a good idea). But in a real life scenario (a microphone listening environment sounds) the ratio of a car sounds is not gonna be %50. Maybe for every 100 sounds the mic picked only one percent will be a car sound.

So what would be a better/realistic to way to prepare and split the data for test and training set?


texts:
Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate

I need to learn $$\tilde{A}$$ such that $$\min_{\tilde{A}}||\tilde{A}^TB-C||$$

and $$\min_{\tilde{A}}||\tilde{A}-A||$$


texts: 'RandomForestClassifier' object has no attribute 'oob_score_ in python

texts:
I am getting:

AttributeError: 'RandomForestClassifier' object has no attribute 'oob_score_'.
But I can see the attribute oob_score_ in sklearn random forest classifier documentation.

param = [10,15,20,25,30, 40]
# empty list that will hold cv scores
cv_scores = []
# perform 10-fold cross validation
for i in tqdm(param):
    clf = RandomForestClassifier(n_estimators = i, max_depth = None,bootstrap = True, oob_score = True)
    scores = clf.oob_score_
    cv_scores.append(scores)

ERROR

AttributeError                            Traceback (most recent call last)
<ipython-input-37-d6c9c54f299e> in <module>()
     15 for i in tqdm(param):
     16     clf = RandomForestClassifier(n_estimators = i, max_depth = None,bootstrap = True, oob_score = True)
---> 17     scores = clf.oob_score_
     18     cv_scores.append(scores)
     19

AttributeError: 'RandomForestClassifier' object has no attribute 'oob_score_'


texts: Returned loss value is different than the loss printed with verbose

texts:
Could someone explain why the loss returned is different than the loss printed during the evaluation?
They are the same in the Tensorflow documentation https://www.tensorflow.org/guide/keras/train_and_evaluate
Code:
results = model.evaluate(test_data, test_target, verbose=2)
print("test loss, test acc:", results)

Output:
45/1 - 0s - loss: 1.2592 - mae: 0.7602
test loss, test acc: [1.05335361427731, 0.76020277]


texts: Clustering of documents using the topics derived from Latent Dirichlet Allocation

texts:
I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(Ideally I would like to use a good one for overlapping clusters so any recommendation is welcomed). I managed to get the topics but they are in the form of:

0.041*Minister + 0.041*Key + 0.041*moments + 0.041*controversial + 0.041*Prime

In order to apply a clustering algorithm, and correct me if I'm wrong, I believe I should find a way to represent each word as a number using either tfidf or word2vec.

Do you have any ideas of how I could "strip" the textual information from e.g. a list, in order to do so and then place them back in order to make the appropriate multiplication?

For instance the way I see it if the word Minister has a tfidf weight of 0.042 and so on for any other word within the same topic I should be to compute something like:

0.041*0.42 + ... + 0.041*tfidf(Prime) and get a result that will be later on used in order to cluster the results.

Thank you for your time.


texts: Keras error "Failed to find data adapter that can handle input" while trying to train a model

texts:
I've been following a tutorial on training a model and I've stumbled across an error that I've been struggling to find a solution for.

The code for the model training is bellow:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D
import pickle

X = pickle.load(open("X.pickle", "rb"))
y = pickle.load(open("y.pickle", "rb"))

X = X / 255.0

model = Sequential()
model.add(Conv2D(64, (3, 3), input_shape = X.shape[1:]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size = (2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size = (2, 2)))

model.add(Flatten())
model.add(Dense(64))

model.add(Dense(1))
model.add(Activation("sigmoid"))

model.compile(loss = 'binary_crossentropy',
             optimizer = 'adam',
             metrics = ['accuracy'])

model.fit(X, y, batch_size=32, validation_split=0.1)

And the result I get when I run this code is:

  ValueError                                Traceback (most recent call
  last)  in
       28              metrics = ['accuracy'])
       29
  ---> 30 model.fit(X, y, batch_size=32, validation_split=0.1)

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\training.py
  in fit(self, x, y, batch_size, epochs, verbose, callbacks,
  validation_split, validation_data, shuffle, class_weight,
  sample_weight, initial_epoch, steps_per_epoch, validation_steps,
  validation_freq, max_queue_size, workers, use_multiprocessing,
  **kwargs)
      731         max_queue_size=max_queue_size,
      732         workers=workers,
  --> 733         use_multiprocessing=use_multiprocessing)
      734
      735   def evaluate(self,

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py
  in fit(self, model, x, y, batch_size, epochs, verbose, callbacks,
  validation_split, validation_data, shuffle, class_weight,
  sample_weight, initial_epoch, steps_per_epoch, validation_steps,
  validation_freq, **kwargs)
      215           validation_data=validation_data,
      216           validation_steps=validation_steps,
  --> 217           distribution_strategy=strategy)
      218
      219       total_samples = _get_total_number_of_samples(training_data_adapter)

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py
  in _process_training_inputs(model, x, y, batch_size, sample_weights,
  class_weights, steps_per_epoch, validation_split, validation_data,
  validation_steps, shuffle, distribution_strategy)
      468                      'at same time.')
      469
  --> 470   adapter_cls = data_adapter.select_data_adapter(x, y)
      471
      472   # Handle validation_split, we want to split the data and get the training

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py
  in select_data_adapter(x, y)
      446         "Failed to find data adapter that can handle "
      447         "input: {}, {}".format(
  --> 448             _type_name(x), _type_name(y)))
      449   elif len(adapter_cls) > 1:
      450     raise RuntimeError(

  ValueError: Failed to find data adapter that can handle input: , ( containing values of types {""})

Thank you in advance if you can help me figure out where the problem is.


texts: How does a Faster R-CNN gets trained and how does the Region Proposal Network fits into that?

texts:
so a Faster R-CNN consists of 3 parts:
-the Base CNN
-the Region Proposal Network with RoIpooling
-and the classification CNN at the end

So I asked myself if all these three Networks are trained seperatly and get  united at the end. Or if the whole constillation is trained together
(so that the input flows through all three Networks and so does the backpropagation)?

If the sencond is the case, how does the RPN fits into that? Because the output of the RPN with RoI-pooling are multiple smaller regions which then get forwarded one after another to the last CNN for classification and so it's no more a continous united CNN, right?

I would be really happy to get some answers because I somehow can't find any else where, thanks :)


texts: Assign new point to a class using spectral clustering

texts:
Say I used spectral clustering to cluster a data-set $D$ of points $X_0 - X_n$ into a number $C$ of clusters. How can I efficiently assign a new single point $X_{n+1}$ to his convenient cluster?

Do I have to do the classification from the beginning (destroy all the clusters and apply the algorithm to the data-set $X_0 - X_{n+1}$), or is there an optimized way to extend to the point $X_{n+1}$?


texts: Metrics values are equal while training and testing a model

texts:
I'm working on a neural network model with python using Keras with TensorFlow backend. Dataset contains two sequences with a result which can be 1 or 0 and positives to negatives ratio in dataset is 1 to 9. Model gets the two sequences as input and outputs a probability. At first my model had a Dense layer with one hidden unit and sigmoid activation function as output but then I changed my models last layer to a Dense with two hidden unit and softmax activation function and changed my dataset's result using Keras to_categorical function. After these changes the model metrics which contains Accuracy, Precision, Recall, F1, AUC are all equal and has a high and wrong value. Here are the implementation I used for those metrics
def recall(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1(y_true, y_pred):
    precisionValue = precision(y_true, y_pred)
    recallValue = recall(y_true, y_pred)
    return 2*((precisionValue*recallValue)/(precisionValue+recallValue+K.epsilon()))

def auc(y_true, y_pred):
    auc = tf.metrics.auc(y_true, y_pred)[1]
    K.get_session().run(tf.local_variables_initializer())
    return auc

and here is the training result
Epoch 1/5
4026/4026 [==============================] - 17s 4ms/step - loss: 1.4511 - acc: 0.9044 - f1: 0.9044 - auc: 0.8999 - precision: 0.9044 - recall: 0.9044
Epoch 2/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9087 - precision: 0.9091 - recall: 0.9091
Epoch 3/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9083 - precision: 0.9091 - recall: 0.9091
Epoch 4/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9090 - precision: 0.9091 - recall: 0.9091
Epoch 5/5
4026/4026 [==============================] - 15s 4ms/step - loss: 1.4573 - acc: 0.9091 - f1: 0.9091 - auc: 0.9085 - precision: 0.9091 - recall: 0.9091

after that I tested my model using predict and calculated metrics using sklearn's precision_recall_fscore_support function and I got the same result again. metrics are all equal and has high value (0.93) which is wrong based on the confusion matrix I generated

What am I doing wrong?


texts: How to make R or Python as fast as SAS for ODBC Oracle queries?

texts:
I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS.

I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The query took 8-15 seconds in SAS, 20-30 seconds in Python, and 50-70 seconds in R. Does anyone know why?

R Packages Used:

First I used the RODBC package in R to query to the Oracle database. Then I tried the ROracle package, but both packages were much slower than SAS.

Python Packages Used:

For Python, I used Oracle's cx_Oracle package for the query.

Thanks a lot,
Sean


texts:
I want to know how to use FREAK feature extraction in python,
I read the documentation but I need some examples.
I wrote this code :

freakExtractor = cv2.xfeatures2d.FREAK_create()
keypoints,descriptors= freakExtractor.compute(image,None)

but the descriptor are None, why?

I used keypoints detected by SURF instead of None parameter:

surfDetector = cv2.xfeatures2d.SURF_create()
keypoints = surfDetector.detect(image,None)
freakExtractor = cv2.xfeatures2d.FREAK_create()
keypoints,descriptors= freakExtractor.compute(image,keypoints)

but the output was very very large, and hard to process.
How can I fix this?
Thanks...


texts:
So I made a convolution neural network to classify between different phonemes. My input datasets are a series of 0.4-second long spectrograms, the labels are each an individual phoneme that happens at every 1/100th of a second. So a 10-second long video would have 1000 labels.

If I pick frame 100, the input would be a spectrogram starting from frame 80 and ending at frame 120. The longest phoneme lasts for 20 frames but the extra length is kept for context.

The structure of the model is one convolution layer with 30 filters, RELU, max-pooling, and finally one fully-connected layer to act as the output layer. There is also a 35% drop-out on the convolution layer.

My dataset is a 26-minute video of me speaking the first third of the sponge bob movie script (not very professional, I know).

I used incremental learning by splitting the dataset into 1-minute pieces. For each training iteration, it made 10 batches, with each batch being 64 random frames from 1 specific minute of the dataset. If the model got above 70% on one 'minute', and the model accuracy didn't change by over 3% within 30 batches, it moved on to the next minute. Unfortunately, even if the model got around 70% on the training data, it still underperformed on the testing data as it consistently got around 30%.

I have run out of ideas to solve this sever overfitting problem, any suggestions would help.


texts:
I have an array of edges and weights:

[['a', 'b', 4],
 ['a', 'c', 3],
 ['c', 'a', 2],
 ...]

I have about 100,000 edges and weights are between 1 and 700, most around 100.

I am thinking of using Markov Cluster Algorithm however wanted to reach out to see if this is the best to use. What about Affinity Propagation? In either case, what is the workflow? Do you typically have a way to measure how well clustered the results. Is there an equivalent to a silhouette score? Is there a way to visualize the clusters?


texts: GPU Accelerated Data Processing for R in Windows

texts:
I'm currently taking a paper on Big Data which has us utilising R heavily for data analysis. I happen to have a GTX1070 in my pc for gaming reasons. Thus, I thought it would be really cool if I could use that to speed up some of the processing for some of the stuff my lecturers have me doing, but it really doesn't seem easy to do this at all. I've installed gpuR, CUDA, Rtools, and a few other bits and bobs, and I can get it to create gpuMatrix objects from genomic expression data, for example, but I have yet to find a function which both works with the gpuMatrix objects and also provides any noticeable difference in performance. Perhaps this just relates to limitations inherent to the gpuR package - some other packages do seem to talk about having functions which sound like they would be more like the sort of thing I'm looking for, which brings me to the question:

Almost all of those packages are exclusively for Linux, is it particularly hard to implement GPU support for R in windows? Or is there some other reason that there are so few packages available to do this in Windows? In some sense I'm just curious, but it would also be very cool to really get it working. It surprises me that there is so little available for Windows, usually it's the other way around.


texts:
My setup is this:

Suppose I have transactional data over a large period of time. The parties of each transaction are labled, and I use Louvain algorithm for detecting communities (and sub-communities) in each arbitrary timestep (a day, 24 hour period).

I also used the labels of the parties and tf-idf to give some literal, summarized description for each community, in each timestep. This has allowed me to manually focus on sereval communities that I find interesting for my specific research, which regulary appear in each timestep (they are somewhat consistent in terms of the nodes that make them up - mostly the same nodes each day). It might also be the case that such a community won't appear at all in some timestep (usaully, it wasn't "consolidated" enough that day, so the Louvain was only able to detect isolated components of it).

Based on previous knowledge, I can also label each timestep with some label which is relevant for my research. For example, the labels might be: "Wedding", "Funeral", "Birthday", "Ordinary day". This is crucial: By distinguishing communities that were detected during an ordinary day from those that were detected during a special day, my goal is to recognize what distinguishes them - and eventually use it for a predictive model.

Finally, In each timestep, I also computed centrality measures for each node in each community, such as degree but also betweenness and closeness.

Given this setup,

I realized I'm dealing with kind of a new dataset: The community instances (i.e a specific community in a specific timestep).
This has led me to think that representing each community instance as a feature vector will allow me to cluster them together, in an attempt to achieve my goal.

There are some direct propeties which can be included in such a vector, such as: number of nodes, number of edges, number of sub-communities, average degree, etc.

A less naive vector representation could be as an ordered list of number of nodes in each sub-community (of the parent community instance), and zero's elsewhere. (the vector length is the maximal number of sub-communities found). A more sophisticated method would be to caclulate the variance (and mean) of the centrality measures of the nodes in each community instance, and use that as features.

In general, I would like to know if there were any successful study cases based on a similar approach (embedding communities instances as feature vectors), and if so, what features were used?
Alternatively, are there any flaws in this kind of approach? or more specifically, in the features I suggested?

Any help would be appreciated.


texts: Marginal contribution of a text document

texts:
I'm trying to build a Shapley value (marginal contribution) of a text document in terms of information content, given that there are several documents on a given topic.
For example, we have 3 reports describing the ocean:
A: {The ocean is blue.}
B: {The ocean is blue and salty.}
C: {The ocean is salty and a home for fish}
If I'm correct, a first step in the algorithm is identifying a set of features (individual pieces of information): {blue, salty, a home for fish}
Moreover, say each feature has an equal weight.
Then I can compute Shapley values of 1/6 for A, 1/3 for B, 1/2 for C.
I wonder if there are well-established algorithms / papers that deal with this problem. I'm coming from economics myself (I'm familiar with Shapley values abstractly), so apologies if the question is too trivial.
Thank you!


texts:
I am new to Machine Learning. I want to develop Curriculum Vitae recommender system. I want to determine how similar 2 CVs are, and given a random CV, it suggest which cluster of CVs it belongs to?

This is what I've already done, following a blog post:

I have a folder containing lot of CVs or resume text documents in plain text format (.txt).
I have done pre-processing on this data, like tokenization, stop words removal, stemming.
I extracted the Candidate's name, email-id, contact number, education and experience.

I am confused with how to train the data and how do I create a model for that. More specifically, I have the following questions:

Now how to create a model on text data?
Which algorithm I should apply on this data?

Please anyone answer. Your help will be appreciated.

Thanks.


texts: Clustering after PCA: Use the standardized data, or take into account the variation explained at each PC?

texts:
I am interested in clustering daily gridded data.

Because of the many dimensions (gridpoints), I first perform PCA to reduce the dimensionality and keep the n-first PCs that account for at least 85% of the variation of the actual data. Then I use these n PCs as inputs to k-Means clustering.

My question is if I should use the standardized (mean=0, std=1) PCs as inputs to the k_Means clustering, or adjust the PCs based on the % of total variation that each PC accounts for. The adjustment can be something like PC[i] * Variation[i] /sum(Variation)

The 1st option results in PCs with the same variance, thus the clustering is unbiased. The 2nd option will end up with a bias towards the PCs that explain most of the variation.

Based on my understanding it is better to work with unbiased data. Nevertheless, in the case of PCA, the PCs are by default of varying importance. Would this support the use of the adjusted PCs?


texts: How to normalize the data correctly in spam dataset

texts:
I'm working on the spam dataset to classify the inputs into binary classes.
my problem is that: the observations in the dataset are floats small numbers in the first 53 column, and the 54 is float larger numbers, while the last two columns are integers.

My Question:
How to Normalize this dataset
correctly, so all the observations have the same importance?
import pandas as pd
spam = pd.read_table("spambase.data",sep=',',header=None)

one proposed approach which didn't seem very convenient to me, because it normalize the whole row input is that:

#========================
# Normalization Function
#========================
def Normalize(x):
    '''
    ==================================
    Normalization Function
    ==================================
    -----------
    Parameters:
    -----------
    @Parameter x: Vector
    ---------
    Returns:
    ---------
    Normalized Vector.
    ================================
    '''
    norm=0.0
    for e in x:
        norm+=e**2
    for i in range(len(x)):
        x[i]/=sqrt(norm)
    return x


texts: What is the right way to keep track of the different things we try?

texts:
In Machine Learning we usually try many combinations of different features, filters we apply to the data, transformations on the features or the target variables and different versions of hyperparameters.

This fact makes it difficult to keep track of what works and what doesn't if we are not exhaustive with how we keep track of the different combinations we try.

I am wondering if there are any best practices around this problem. My current approach is to keep track of the different combinations naming files making reference to the parts that compose it, for example a hyperparameters pickle file I would name it booster_params_{}_{}_{}_{}.pickle'.format(filter_name, features_name, model_target, params_iteration)

where filter name is the set of filters I'm applying to the data, features name refers to the set of features used, model target to the target I'm modeling and params iteration refers to the version of the hyperparameters.

This seems like an overkill, and that is why I'm looking for ideas on how to tackle this problem.


texts: Getting monthly revenue predictions for outlets

texts:
I am often presented with a task of predicting monthly revenues of retail outlets. Say I have a training set of N outlets, each associated with a series of historical monthly revenues (target) and a set of features which are time-independent (e.g. location (and all associated geofeatures), outlet type, trading area etc.) Then I have a test set of M outlets for which I need to predict monthly revenues for a specified time period (e.g. Q1-Q3 2019).

The usual approach I take consists of adjusting historical revenues on the training set for trend, seasonality and inflation, then take the average for each outlet and use it as a target, train my regression model (could be anything but mostly LightGBM), get predictions for the test set. These predictions are, again, averages for each outlet adjusted for trend, seasonality and inflation, so I need to deadjust them to obtain monthly predictions. So basically the scheme is: monthly historical revenues -> adjusted average as a target -> adjusted average as a prediction -> reconstructed monthly predictions (hope this makes sense).

The whole approach is pretty tedious and feels unnatural. I wonder whether there are frameworks that would allow the use of historical monthly revenues "as is" as a target and produce predictions for the specified time range.

There is, of course, panel regression, but 1) my features don't change over time so the idea seems superficial and 2) gradient boosting in panel regressions isn't exactly a hot topic so implementations are rare (if any exist).


texts: Distance to the Center of Vocabulary of Word Embeddings

texts:
Suppose I:

Generate a set of word embeddings (using word2vec or similar) based on a large but specific corpus
Compute the centroid of all the words in the set
Find the word(s) with the smallest (say Euclidean for instance) distance to that centroid

Would I be correct to assume that this word (or words) represent the core concept or something like the main theme of the corpus ?
Now suppose I were to do the same thing but diachronically, training different sets of word vectors on separate corpora for the last 5 decades. (One set of embeddings for the 2010's, one for the 2000's, etc...). Could I capture something like a shift in zeitgeist over time?
I am aware of some previous research on semantic drift using embeddings, like Histwords [https://nlp.stanford.edu/projects/histwords/]. However they track the position of a single, predetermined word over time. I would be more interested in the "discovery" of central concepts at certain points in time in a specific discursive corpus.


texts: What affect will replacing words with bigrams have on TfIDF?

texts:
Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by calculating the cosine similarity between the documents' TfIDF vectors.

If I now compile a set of bigrams based on frequency, and replace instances of two words in each document with the concatenation of the bigram, what affect will this have on

1) Calculating TfIDF vectors?

2) Calculating document similarity?

To be explicit, lets take an example in python:

doc1 = ['the', 'car', 'drove', 'from', 'new','york', 'to', 'washington'] # a single text document
top_n = {('new', 'york'), ('drove', 'from')...} # set of top n bigrams by frequency
new_doc = replace_ngrams(doc1) # function that replaces words with concatenated bigrams
print(new_doc)
>>> ['the', 'car', 'drovefrom', 'newyork', 'to', 'washington']


texts: Having difficult interpreting the eigenvectors for a simple 3x2 matrix

texts:
I calculated the eigenvectors and eigenvalues from a covariance matrix given a data matrix of 3 columns and 2 rows.

I am trying to interpret results but I can't understand on how to interpret them.

Create a 2x3 matrix:

# Create a 2x3 matrix

data = np.around(np.random.uniform(size=(2,3)) * 100)

The data looks as follows:

    [
      [ 4., 65., 77.],
      [68., 12., 89.]
    ]

# Here each row represents one data point
# and columns represent the features in the data set
# So there are 3 features and 2 data points

Calculate the mean for each feature in the data set.

mean = np.mean(data, axis = 0)

Center the data around origin, by subtracting mean from the data set.

difference = np.subtract(data, mean)

Now, calculate the covariance matrix:

cov = np.dot(difference.T, difference)

The cov matrix looks as follows:

[
    [ 2048. , -1696. ,   384. ],
    [-1696. ,  1404.5,  -318. ],
    [  384. ,  -318. ,    72. ]
]

As I understand about the covariance matrix, it explains the variance between all feature-pairs. Since there are 3 features, it gives out a 3x3 matrix explaining the variance between all possible pairs.

Finally, calculate the eigenvectors and eigenvalues:

val, vec = np.linalg.eigh(cov)

The vec matrix looks as follows:

[
    [ 0.60999981,  0.21639063,  0.76228297],
    [ 0.77451164,  0.040441  , -0.63126559],
    [ 0.16742745, -0.97546892,  0.14292806]
]

How do I interpret the the vector matrix? I understand what are eigenvectors physically. They do not change in position when an object undergoes a transformation but only a scalar change by their eigenvalues.

What are some possible ways, I could use this vec matrix?


texts: What happens in community detection when weights of edges are not considered using the Louvain method

texts:
My question is related to the following posts, but doesn't quite answer my question:
Effect of weights on the Louvain communities detected
https://stackoverflow.com/questions/49151996/meaning-of-weights-in-community-detection-algorithms/49311269#49311269
If weights are not considered (or provided in original graph creation), is community detection using the Louvain algorithm simply detecting communities based on the connections that form between nodes (in other words, k-number of communities) without considering the distance between them (edge weights, as there are no weights to determine how strong connections are)?
Thank you.


texts: What does "batch" and "batch_size" mean in word2vec skip-gram model?

texts:
I am reading tensorflow documentation now and I can not understand what does "batch" and "batch_size" mean in explanation skip-gram model. Please, can someone explain me?
Here this paragraph:

Recall that skip-gram inverts contexts and targets, and tries to predict each context word from its target word, so the task becomes to predict 'the' and 'brown' from 'quick', 'quick' and 'fox' from 'brown', etc. Therefore our dataset becomes
(quick, the), (quick, brown), (brown, quick), (brown, fox), ...
of (input, output) pairs. The objective function is defined over the entire dataset, but we typically optimize this with stochastic gradient descent (SGD) using one example at a time (or a 'minibatch' of batch_size examples, where typically 16 <= batch_size <= 512).


texts:
I searched for other similar questions on Google and here, but couldn't find any.
I'm consistently getting results of less quality when making deep learning with GPU, compared to CPU. Of course, it is much faster using GPU, but my results converge to lower quality on many different aspects: subjective quality of embedding, validation accuracy, loss, etc. It doesn't depend on the number of epochs, ie. results won't converge toward the same value: one is always inferior to the other. It's disconcerting when trying to achieve solid, reproducible SOTA results, where .1 or .2% is quite important (litteraly from top2 to top1).

I have a RTX2070 and a good CPU. Is it inherent to modern GPU ? Are approximations done using GPU that could explain this phenomenon ? Thanks for your inputs.


texts: What is the best hardware/GPU for deep learning?

texts:
What is the best GPU for deep learning currently available on the market?

I've heard that Titan X Pascal from NVidia might be the most powerful GPU available at the moment, but would be interesting to learn about other options.

And additionally - what might be considered a good machine for data-intensive computing, in terms of processor parameters, RAM etc. ?


texts: Working with inaccurate (incorrect) dataset

texts:
This is my problem description:

"According to the Survey on Household Income and Wealth, we need to find out the top 10% households with the most income and expenditures. However, we know that these collected data is not reliable due to many misstatements. Despite these misstatements, we have some features in the dataset which are certainly reliable. But these certain features are just a little part of information for each household wealth."

Unreliable data means that households tell lies to government. These households misstate their income and wealth in order to unfairly get more governmental services. Therefore, these fraudulent statements in original data will lead to incorrect results and patterns.

Now, I have below questions:

How should we deal with unreliable data in data science?
Is there any way to figure out these misstatements and then report the top 10% rich people with better accuracy using Machine Learning algorithms? -How can we evaluate our errors in this study? Since we have unlabeled dataset, should I look for labeling techniques? Or, should I use unsupervised methods? Or, should I work with semi-supervised learning methods?
Is there any idea or application in Machine Learning which tries to improve the quality of collected data?
Please introduce me any ideas or references which can help me in this issue.

Thanks in advance.


texts:
So I recently came along kNN k nearest neighbour.
When looking at its disadvantages, most of the literature mentions it is costly, lazy, requires full training data plus depends on the value of k and has the issue of dimensionality because of the distance.
Other than that I have following hypothesis.

1- It ignores the fact that dimensions can be inter related and instead assumes they are independent (as we are just calculating distance)
2- Has the issue of normalization of data... if the data is not normalized distance can be biased towards a specific dimension

I will like to have a comprehensive analysis on the disadvantages of kNN apart from those mentioned above and if they are wrong then why.


texts: Clustering of sparse matrix with many co-variates

texts:
I have a 2M x 2000 sparse matrix where rows represent an item and columns represent dimensions. I want to understand whether there are meaningful clusters in the data and I started to explore the dimensions to transform and normalise the data.

Of the 2000 attributes to an item, many are co-variant (rho > .5).
Are there clustering techniques that handle co-variants well automatically, without having to remove them manually?


texts: Predictive output with your own model built

texts:
I would need to better understand how can be created a machine learning algorithm from scratch using an own model developed based on boolean values, for example # of words in a text, # of punctuation, # of capital letters, and so on, to determine if a text is formal or informal.
For instance: I have
Text
there is a new major in this town
WTF?!?
you're a great person. Really glad to have met you
I don't know what to say
BYE BYE BABY

I created some rules to assign a label on this (small) train dataset, but I would need to understand how to apply these rules to a new dataset (test):

if there is an upper case word then I;
if there is a short expression, like don't, 'm ,'s, ... , then I;
if there are two symbols (punctuation) close to each other, then I;
if a word is in list of extra words, then I;
otherwise F.

Suppose that I have a dataframe to test and to assign these labels (I or F):
FREEDOM!!! I don't need to go to school anymore
What are u thinking?
Hey men!
I am glad to hear that.

how could I apply my model to this new dataset, adding labels?
Test                                                  Output
FREEDOM!!! I don't need to go to school anymore       I
What are u thinking?                                  I
Hey men!                                              I
I am glad to hear that.                               F

Update after mnm's comment:
Would it be considered a machine learning problem the following one?
import pandas as pd
import numpy as np
data = { "ID":[1,2,3,4],
        "Text":["FREEDOM!!! I don't need to go to school anymore",
    "What are u thinking?",
    "Hey men!","
    I am glad to hear that."]}

# here there should be the part of modelling
df['upper'] = # if there is an upper case word then "I"
df['short_exp'] = # if there is a short exp then "I"
df['two_cons'] = # if there are two consecutive symbols then "I"

list_extra=['u','hey']
df['extra'] = # if row contains at least one of the word included in list_extra then 'I'

# append cols to original dataframe
df_new = df
df_new['upper'] = df1['upper']
df_new['short_exp'] = df1['short_exp']
# and similar for others

It is not clear, however, the latest part, that one based on condition. How can I predict the new values for the other texts?


texts: Why is not AUC or other metrics used for splitting nodes in decision trees?

texts:
There are common ways to split a tree in decision trees and all their variants:

Gini Index
Entropy
Misclassification

Why there is not a method which uses directly AUC or accuracy (or whichever the modeler need) to split the nodes.

Is it because of common use, or there is a mathematical explanation for it?


texts:
My attempt:
(a) I solved that $a=\ln{\frac{P(X|C_0)P(C_0)}{P(X|C_1)P(C_1)}}$
(b) Here is where I'm running into trouble. I'm plugging the distributions into $\ln{\frac{P(X|C_0)P(C_0)}{P(X|C_1)P(C_1)}}$ and I get $a=\ln{\frac{P(C_0)}{P(C_1)}}+\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)$.
I can see that $b=\ln{\frac{P(C_0)}{P(C_1)}}$ and $w^Tx=\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1)-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)$.
I'm not sure how to simplify $w^Tx$ so that I can solve for $w$. Or is there something that I did wrong?


texts: Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

texts: If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be distributed across the cluster? Or do i have to use pyspark dataframe operations and pyspark ml packages for operations to be distributed?


texts: MLP regressor output invariant under different test data

texts:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor

#CSV of data
filepath = 'H:\\Data\\'
file = 'ScaledData'

#Read in data
features = pd.read_csv(filepath + file + '.csv')

#Features that we want to use are called the labels, we remove them from the features
labels = np.array([features['xVector'], features['yVector'], features['zVector'], features['length'], features['timeFromBeginning']])
features= features.drop('xVector', axis = 1)
features= features.drop('yVector', axis = 1)
features= features.drop('zVector', axis = 1)
features= features.drop('length', axis = 1)
features= features.drop('timeFromBeginning', axis = 1)
feature_list = list(features.columns)
features = np.array(features)

# Instantiate model with 1000 layers
rf = MLPRegressor(100)

# Train the model on training data
rf.fit(features, labels.transpose());

# Predict future events
futureEvents = np.array([521553,521554,521800])
predictions = rf.predict(futureEvents.reshape(-1, 1))
print(predictions)

I am trying to predict values of future events. There are 521553 data points that I am trying to learn based on and I am trying to predict future behavior. The data is all scaled to be between -1 and 1. Changing the event number that I am predicting the location and length and time of, I barely get a change in the predicted values even though the input values have a range over all values (and are rapidly varying). I have read somewhere that I should have a final linear layer though I didn't understand why to do this. I have simplified the code to what I think is the bare minimum required to show the problem.


texts: Best CNN architecture for binary classification of small images with a massive dataset

texts:
The title has it all...

Any tip is welcomed.

Should I use a very deep convolutional neural network ?

Use a large amount of filters ?

Parallel layers ?

Dataset examples:

1) "Good"

2) "Bad"


texts: Methods for Clipping Weights when training a Keras GAN

texts:
I'm trying to reduce some of the volatility I'm seeing while training a DCGAN in Keras and have decided to try to implement some type of weight clipping which was recommended in several places while researching this question.
One of my tutorials suggests a few extra lines of code in the training of the Discriminator:
for l in self.critic.layers:
     weights = l.get_weights()
     weights = [np.clip(w, -clip_threshold, clip_threshold) for w in weights]
     l.set_weights(weights)

On the other hand, there is a built-in function called tf.keras.constraints.MinMaxNorm in Tensorflow 2.2.0 that would seem to have the same effect. I wanted to check my assumptions: Are these different approaches to containing weights while training? How are they different?


texts: Similar algorithm to apriori to find unpopular sequential patterns

texts:
I am working with a dataset that looks like similar to this one but it is way larger (approx. 30.000 arrays):
sequences = [[car, house, bike, train],
             [car, house, bike],
             [apartment, train, car],
             [building,flower, bike, train],...]

It is a bit difficult to explain but there are some items in the data, that occur pretty often and some dont. Nevertheless the items that occurs less often occur in sequences that I want to find as well. So in this case I would like to find all transportation possibilities that occur together, but I also want to find all housings that occur together or all plants ect. I know that plants and buildings dont occur very often in general so the support value of these itemsets in apriori are penalized by this. All my results somehow contain transportation possibilities as their support values are higher as they occur often within the data set.
Currently I am using MLxtend's apriori but I would be glad if there are algorithms that I could use that do not penalize less often occuring items in the data set.
Is there some library/algorithm I could use to solve this problem?
Thanks in advance!


texts: Estimation of Bayes Error / Human-level error

texts:
In one of course Andrew Ng mentioned that model (Machine learning) error can't outperform Bayes / human level performance and hence the bias can't avoided.. (Unavoidable bias)

How do we determine / estimate Bayes error or human level error so that we could stop optimizing the model for reducing bias further?

Examples / concepts for classification problem would be appreciated...


texts:
First of all I know the question may be not suitable for the website but I'd really appreciate it if you just gave me some pointers.

I'm a 16 years old programmer, I've had experience with many different programming languages, a while ago I started a course at Coursera, titled introduction to machine learning and since that moment i got very motivated to learn about AI, I started reading about neural networks and I made a working perceptron using Java and it was really fun but when i started to do something a little more challenging (building a digit recognition software), I found out that I have to learn a lot of math, I love math but the schools here don't teach us much, now I happen to know someone who is a math teacher do you think learning math (specifically calculus) is necessary for me to learn AI or should I wait until I learn those stuff at school?

Also what other things would be helpful in the path of me learning AI and machine learning? do other techniques (like SVM) also require strong math?

Sorry if my question is long, I'd really appreciate if you could share with me any experience you have had with learning AI.


texts:
I have data on store level purchases, Panel-level purchases and demographic information of loyalty cards.

In the store purchases information, the data consists of a product code which can be assigned to brands/products (Pepsi, Coke, Budweiser etc.) for many different products. Along with how many items of that product was sold that week and the price of each product.

In the Panel information, I have data on the product code, the number of items purchased in each store by that Panel ID and total number of $ spent on that product in that store.

In the demographic information I have data on each Panel ID:- how many people live in the household, panel_ID income, number of children, education etc.

I want to apply conditional probabilities on the product information, such as given that panel_ID bought product A what is the probability that they will buy product B.

However for just beer data I have 250 different brands and 300,000+ Panel_ID purchases.

I am thinking of applying Naive Bayes as a starting point, however what other Machine Learning methods would you suggest for such a problem?


texts:
We're build an item-item recommender based on the text descriptions of the items. Our initial approach was to calculate the TF-IDF vectors for each item. We used a hashing tf with 5000 possible hashes for the words. Then approximate all-pairs using a sampling technique (DIMSUM).

We have an mxn matrix where m is the number of words, n is the number of items. The naive approach of calculating all column cosine similarities won't work here since we have n=10^7 m=10^4.

Our first attempt was using DIMSUM http://stanford.edu/~rezab/papers/dimsum.pdf which is an all-pairs sampling technique. The problem with DIMSUM is it works for data where m >> n. Our matrix is short and wide n>m.

My Question:
What is a good approach for estimating all-pairs similarity of items based on words. Where number of items is 10^7, number of words is 10^4. We only want items pairs above a certain threshold of similarity.

We don't have to do it this way, our task is to recommend a small set of items given one item, we have ways of doing this using collaborative filtering, but we want to also handle new items that we have no user data for so we're trying to find a way to use tf-idf vectors for that case.


texts: selecting a number of neurons specifically for RL

texts:
If I would like to use DQN to train my Reinforcement-Learning agent, how do I select the number of neurons?

In Supervised Learning, selecting too few or too many can result in either  a too low potential (lack of neurons) or in a poor generalization due to overfitting (for example, due to too many neurons). We can use Genetic Algorithm to determine the best combination of hyperparameters instead of trying by hand.

However, in RL - is generalization even a thing? After all, we are training a spider to run by moving its limbs in a most efficient way, an arm to grab a box in the fastest way, etc.

Once learnt, are we even allowed to change the environment and expect agent to perform well, generalizing to new environment?

And thus, is selecting neurons even a thing for RL, or can we just give it a maximum possible number of neurons and layers, letting it learn correct movement of creature's limbs?


texts: Forecasting multiple time series with a single model

texts:
I have a dataset with sales numbers for ~500 different markets (assume different cities or regions) and need monthly forecasts for each market. Instead of building 500 different models, I'm interested in just training a single model with a single set of params to prevent me from having to manually feature engineer each market.

Can someone point me to some resources for this problem? I went through the similar Stack questions and they weren't as helpful for this particularly.

Specifically, I know that LSTMs are sometimes used here, but would classical approaches be effective as well?


texts: How to identify new clusters that the training data has never seen

texts:
I have to identify the different operational states of a server. I have readings related to the different sensors of the server ( like temp sensor,fan speed sensor,job load sensor etc).The data I have information about some operational states (normal, high temp , high temp and high fan speed) etc. What ML algorithms should I use to identify if any other operational state,state which the training data has not seen, comes up?

I have used several clustering algorithms. I expected Gaussian Mixture models to work well, but they fail to indicate a new operational state of it comes up.

I used LSTM and looked at the residuals, but had to look at the residuals of each parameter to identify diff states.


texts:
I have vector c(1,3,4,5,7).
I want convert it in this way:
c('apple','pie','nuts','cookies','some_other_class')

Do you have any short ways to do it? Like a switch method or something like that?


texts: Why transpose of independent feature matrix is necessary in case of linear regression?

texts:
I can follow classical linear regression steps:

$Xw=y$

$X^{-1}Xw=X^{-1}y$

$Iw=X^{-1}y$

$w=X^{-1}y$

However, on implementing in Python, I see that instead of simply using

w = inv(X).dot(y)

they apply

w = inv(X.T.dot(X)).dot(X.T).dot(y)

What is the explanation of the transpositions and the two times multiplication here? I'm confused...


texts: Should augmentation also be performed on the validation set when the dataset is imbalanced?

texts: I am training a CNN on images (2 classes) and I have an imbalanced dataset (1:7 ratio). I am trying to tackle this by performing offline image augmentation. Should I perform augmentation also on the validation set or is it ok if the validation set remains imbalanced?


texts: I was looking to learn about Bayesian theory in decision tree and how it avoids overfitting but couldn't find any tutorials for someone just starting. Do you know any resources to learn about it?


texts: Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

texts:
So I just used BERT pre-trained with Focal Loss to classify Physics, Chemistry, Biology and Mathematics and got a good f-1 macro of 0.91. It is good given it only had to look for the tokens like triangle, reaction, mitochondria and newton etc in a broader way. Now I want to classify the the Chapter Name also. It is a bit difficult task because when I trained it on BERT for 208 classes, my score was almost 0. Why? I an get that there are lots of information also like nacl: sodium chloride , bohr model 9.8 m/sec etc which I think BERT is not trained for. I want to ask few questions.

Is BERT useful in these conditions? Is it trained on scientific terms. I mean can it get the context of Schrödinger equation to Plank's Constant? If not, I don't think I should use it because I don't have enough data to re-train BERT. Anything but BERT
Can I use FastText or GloVe? Cn they get the meaning or context?
Or should I simply create my own embeddings in pytorch/keras and keep nacl,fe,ppm as they are and hope either of Transformer or Attention will capture it?

Please help. I have a data of 120K questions/data points.


texts:
In this paper, in figure 4:

the decision boundaries are shown for different $W1$ and $W2$s. If the softmax classification forces either $W_1.x > W_2.x$ or $W_1.x < W_2.x$ for class 1 and class 2 respectively, then the decision boundary is $(W1-W2).x = 0$ which means that that the decision boundary is perpendicular to $W1-W2$. However in this figure only in the case where $||W1|| = ||W2||$ this is visually correct. In other cases, this is not the case. If one draws $W1-W2$, it won't be perpendicular to the decision boundary.

Am I thinking right?


texts: Long term time series forecasts with small dataset

texts:
I have a small dataset which has timestamp and temperature values for 6 months(I.e. one temperature value per day). I would like to forecast 2-3 months of temperature.
I would like to know, what kind of models would be suitable for such purpose.

I have used LSTM but the forecasts degrades as I go further  and the data size is pretty small for LSTM to be trained.


texts: How to handle large systematic missing data in time series?

texts:
I have this time series, where on the weekends, the dependent variable values are missing.
It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an industry metric. Currently, on Saturdays and Sundays, no real values are recorded, officially, they perform Last Observation Carried Forward (LOCF) for Saturdays and Sundays (That is, Friday's value is used for Saturday and Sunday).
This is very systematic and almost 30% of the data is LOCF and on those specific dates. I suspect that this is ruining my time series forecasting using ARIMA.
What should I do? I have thought about using multiple imputation, performing random forest regression on the valid data and fill in those 30%/performing KNN regression for the same. (given that random forest and KNN are very related)
I haven't encountered other people having this specific issue and implementing a specific solution.


texts: Methods of disaggregating data to smaller units?

texts:
I have a relatively straightforward question that I know poses some difficult challenges.
Let's say I have a state-level rate of X. I would like to disaggregate the state-level rate to the county-level. I realize this is can be dangerous (ecological fallacy), but I have seen some studies use the technique with a set of assumptions.
For example, if I know that each county is a certain proportion of the entire state population, I could take that proportion and multiply it by the state-level rate of X to get an (incredibly) naive county-level rate of X.
I'm trying to find more information on ways to make this approach 'less' naive, but I can't seem to get any momentum. I've tried using the terms 'disaggregating' and 'weights', but I can't seem to tap into the right body of literature.
Does anyone know of any methods/body of work that have attempted to handle this problem?


texts: How does implicit quantile network (IQN) keep the quantile value for a state action pair monotonous?

texts:
In my view, the implicit quantile network proposed by https://arxiv.org/pdf/1806.06923.pdf is used to estimate the quantile value of each state-action pair. Essentially the network is the approximated quantile function. Therefore, it should be monotonous (i.e., for a specific state-action pair, a larger quantile should correspond to a larger network output/quantile value ). However, the paper seems not to mention how to deal with this issue.
Do I get something wrong? Appreciate any comment!


texts: Why my svm.SVC.fit( ) (linear kernal) run so long time?

texts:
I am using sklearn.svm.SVC( ) to train & test my dataset. 80% are used for training, 20% are used for testing.

Here is my Python code:

data = pd.read_csv(trainPath, header=0)

X = data.iloc[:, 5:17].values
y = data.iloc[:, 17:18].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

print(X_train.dtype, y_train.dtype)  # float64 int64

clf = svm.SVC(kernel='linear').fit(X_train, y_train.ravel())
print('done')

y_pred = clf.predict(X_test)

print("Accuracy:", metrics.accuracy_score(y_test, y_pred))
print("Precision:", metrics.precision_score(y_test, y_pred))
print("Recall:", metrics.recall_score(y_test, y_pred))
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
print(tn, fp, fn, tp)

For data.shape = 30,000 x 13, it runs around 15 mins.

For data.shape = 130,000 x 13, it runs more than 1 hour.

Why it runs so long time, I don't think it is normal.

i5, 2.8GHz, 16.0 GB memory


texts: How to gauge overfit with MLPClassifier and cross_val_score?

texts:
I'm learning sklearn.

When using MLPClassifier.fit() and MLPClassifier.predict() I would do a manual validation (looking for overfit) by running the training set again through the prediction and accuracy (as follows)...

from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import cross_val_score

target = ["target"]
features = [c for c in shuf_df.columns if c not in target]
neurons =[8, 16, 32, 64, 128, 256]
for activation in ["logistic", "relu"]:
    for neuron in neurons:
        mlp = MLPClassifier(hidden_layer_sizes=(neuron,), activation=activation)
        start = datetime.now()
        mlp.fit(train_df[features], train_df[target])
        nn_predictions = mlp.predict(test_df[features])
        accuracy_test = accuracy_score(test_df["target"], nn_predictions)
        nn_predictions = mlp.predict(train_df[features])
        accuracy_train = accuracy_score(train_df["target"], nn_predictions)
        print("({}) [{}] neuron:{}, accuracy_test:{}, accuracy_train:{}".format(datetime.now() - start,
                                                                                activation,
                                                                                neuron,
                                                                                accuracy_test,
                                                                                accuracy_train
                                                                                )
        )

...which results in...

(0:00:00.750631) [logistic] neuron:8, accuracy_test:0.8611111111111112, accuracy_train:0.9700765483646486 # Probably OK
(0:00:00.860471) [logistic] neuron:16, accuracy_test:0.8916666666666667, accuracy_train:0.9930410577592206 # Approaching overfit
(0:00:01.491433) [logistic] neuron:32, accuracy_test:0.8972222222222223, accuracy_train:0.9993041057759221 # Probably overfit
(0:00:01.951523) [logistic] neuron:64, accuracy_test:0.9166666666666666, accuracy_train:1.0 # overfit
(0:00:02.449780) [logistic] neuron:128, accuracy_test:0.925, accuracy_train:1.0 # overfit
(0:00:03.304685) [logistic] neuron:256, accuracy_test:0.925, accuracy_train:1.0 # overfit
(0:00:00.846773) [relu] neuron:8, accuracy_test:0.8583333333333333, accuracy_train:0.9436325678496869 # Probably OK
(0:00:00.905262) [relu] neuron:16, accuracy_test:0.8777777777777778, accuracy_train:0.9986082115518441 # Probably overfit
(0:00:01.531930) [relu] neuron:32, accuracy_test:0.8972222222222223, accuracy_train:1.0 # overfit
(0:00:01.695193) [relu] neuron:64, accuracy_test:0.9083333333333333, accuracy_train:1.0 # overfit
(0:00:01.503808) [relu] neuron:128, accuracy_test:0.9027777777777778, accuracy_train:1.0 # overfit
(0:00:02.060312) [relu] neuron:256, accuracy_test:0.9194444444444444, accuracy_train:1.0 # overfit

How would I determine overfit when using cross_val_score? (as follows)...

target = ["target"]
features = [c for c in shuf_df.columns if c not in target]
neurons =[8, 16, 32, 64, 128, 256]
for activation in ["logistic", "relu"]:
    for neuron in neurons:
        mlp = MLPClassifier(hidden_layer_sizes=(neuron,), activation=activation)
        cv_scores = cross_val_score(mlp, shuf_df[features], shuf_df[target], cv=4)
        print("[{}] neuron:{}, cv_scores:{}".format(activation,
                                                    neuron,
                                                    cv_scores,
                                                    )
        )


texts: Converting pixel values to temperature OpenCV

texts:
I am converting a pixel values to temp (values of a thermal image). But after conversion output image is not except able. Here is my code:
import cv2
import numpy as np
import matplotlib.pyplot as plt

#Thermal image
img = cv2.imread('IR2.bmp',0)

print(img)
pixel=np.asarray(img)

# storing in pixel array
for i in range(0,img.shape[0]):
   for j in range(0,img.shape[1]):
      print(pixel[i][j])

# converting pixel value to temperature values
r,c=(230,304)

arr = np.zeros([230,304])
for i in range(0,img.shape[0]):
   for j in range(0,img.shape[1]):
        arr[i][j] = 9.2+(pixel[i][j]/255)*26.2

print(arr)

# showing the converted image

cv2.imshow('converted',arr)

cv2.waitKey(0)
cv2.destroyAllWindows()


texts: How do I transform a file to .txt file using pandas?

texts:
I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv().
Is there something similar for transforming the file into .txt() file using pandas?


texts: Export pandas to dictionary by combining multiple row values

texts:
I have a pandas dataframe df that looks like this

name    value1     value2
A       123         1
B       345         5
C       712         4
B       768         2
A       318         9
C       178         6
A       321         3

I want to convert this into a dictionary with name as a key and list of dictionaries (value1 key and value2 value) for all values that are in name

So, the output would look like this

{
 'A': [{'123':1}, {'318':9}, {'321':3}],
 'B': [{'345':5}, {'768':2}],
 'C': [{'712':4}, {'178':6}]
}

So, far I have managed to get a dictionary with name as key and list of only one of the values as a list by doing

df.set_index('name').transpose().to_dict(orient='list')

How do I get my desired output? Is there a way to aggregate all the values for the same name column and get them in the form I want?


texts: Network Visualization - R code - Which Package is needed?

texts:
I'm trying to create a network visualization to study the flights to and from a certain airport.

My data consists of 123000 rows in this format:

Origin_ID    Destination_ID    Frequency
  1726            3504           40000
  3504            4517              40
  5616            7205              38
               ...

I'm trying to create a image similar to this:

Basically, I want to be able to see the strength of the relationship (based on the frequency) and identify which combinations are most common in that airport.

Can it be created in R using this format? Or do you suggest any other open source platform to create this visualization?


texts: Visualizing deep neural network training

texts:
I'm trying to find an equivalent of Hinton Diagrams for multilayer networks to plot the weights during training.

The trained network is somewhat similar to a Deep SRN, i.e. it has a high number of multiple weight matrices which would make the simultaneous plot of several Hinton Diagrams visually confusing.

Does anyone know of a good way to visualize the weight update process for recurrent networks with multiple layers?

I haven't found much papers on the topic. I was thinking to display time-related information on the weights per layer instead if I can't come up with something. E.g. the weight-delta over time for each layer (omitting the use of every single connection). PCA is another possibility, though I'd like to not produce much additional computations, since the visualization is done online during training.


texts: Support Vector Machine analysis - Python, how do I determing whether to use linear, square, or other types of SVM models?

texts:
how do I determine whether to use linear, square, or other types of SVM models?

under which criteria should an SVM model be applied in the first place?


texts:
I am not very good at understanding statistics jargon

I tried to read articles on statistics every article says distribution and show a picture of bell curve. I understand that x axis of bell curve will take the value of data but does y-axis always has the count/frequency od the data

Is Distribution and Histogram same ?

I did find an article on difference between distribution and histogram

https://asq.org/quality-resources/histogram

But it says histogram is most commonly used for frequency distribution

So does that means there are other types of distribution ?
Or
Does distribution of variable always means frequency distribution ?


texts: How to adjust the Regression with ANN for last part of function

texts:
The Blue dots represent the required function and the Black ones represent the predicted function

I am using keras with the following code:-

# Set callback functions to early stop training and save the best model so far
from keras.callbacks import EarlyStopping, ModelCheckpoint
callbacks = [EarlyStopping(monitor='val_loss',verbose=1, patience=5,min_delta=0.002,mode='auto',restore_best_weights=False),
             ModelCheckpoint(filepath='best_model.hdf5', monitor='val_loss', save_best_only=True)]

##The Architecture
np.random.seed(0)
model = Sequential()
model.add(Dense(500,kernel_initializer='normal',activation='tanh',input_dim=1))

model.add(Dense(300, kernel_initializer='normal',activation='relu'))

model.add(Dense(500, kernel_initializer='normal',activation='relu'))

model.add(Dense(1, kernel_initializer='normal',activation='linear'))

model.compile(optimizer='adam',loss='mean_absolute_error',  metrics=['mae','accuracy'])

his = model.fit(X_train,y_train,epochs=150,batch_size=10,validation_data=
(X_val,y_val), verbose=1,callbacks=callbacks)

I tried various architecture but still not getting the result please guide


texts:
I have learnt from some examples the existence of regularization option at ANNs (concretely, at Keras implementation). As far as I know, regularization in general is a kind of "penalty" on parameters to prevent model complexity and overfitting.

Accordingly, W_regularizer and b_regularizer options in Keras are for weight and bias parameter regularization, unless I am mistaken. But what is activity_regularizer for? How is it related to the weight/bias regularization? And more generally: what is a good practice to using all these regularization possibilites (apart from the blind brute force tuning)? Because of ANNs/CNNs are produce very low overfitting measured on the validation set, it seems me that regularization is not a really useful tool with neural nets.


texts:
I'm looking for an (ideally free) API that would have time series avg/median housing prices by zip code or city/state.

Quandl almost fits the bill, but it returns inconsistent results across different zip codes and the data is not as up to date as I'd like (it's mid November, and the last month is August).

I also looked at Zillow, but storing their data is against TOS, and at 1,000 calls daily--it would take forever to pull in the necessary data.

Any suggestions (even if they aren't free) would be much appreciated!