texts: Alternative methods of population-level classification

texts:
In a recent study, I am generating a series of logistic regression and random forest models to predict some binary outcome. For the sake of understanding, suppose this outcome is the occurrence of diabetes in 1,000 patients at a local hospital. I know the occurrence of diabetes in this population, and can thus generate a number of classification models of this outcome using this population.

Suppose I then want to use these predicted models to project the probability of diabetes in a second population where I don't observe diabetes classification (say, N = 10,000). I can use my models generated above to predict diabetes on a person-by-person basis in this second population.

For this particular study, I'm most interested in predicting the level of diabetes at the population-level of this second population. For example, I may want to know the % of this second population that potentially has diabetes.

In the literature, it appears that most studies use some threshold (say, 50%) and sum up the # of observations with a predicted probability greater than the threshold. However, I've found other studies that have simply summed the probabilities to generate the # of, say, diabetes patients in this second population.

I'm not overly familiar with the machine learning literature, but do both of these approaches make sense? Both approaches make sense and likely lead to similar estimates with well-balanced data. However, unbalanced data may lead to distributions where the choice matters (see attached). Thoughts?


texts:
I have two encoder-decoder models.

*First model:

*Second model:

When I check the performance of the models I get approximately the same performance time (First model ~ 42 sec, Second model ~ 40 sec). I train my model on GPU and check performance on CPU. I test it only on one large image where the size is 12348x12348.
I was expecting the larger model that has more parameters to train (second model) to give me longer run time. Anyone can help me to understand why it is not the case here? Am I doing something wrong?


texts:
I hope this is the right subforum as I did not really find a suiting one.
I'm mining data from a website that does specifically exclude some crawlers from it's site in the robots.txt like this:

User-agent: *
Disallow:
User-agent: Proxetrics2000
Disallow: /

While I'm pretty sure I'm not Proxetrics2000, how can I know which User-Agent to comply to?  Let's say I'm mining too frequently and the admin wants to restrict me from certain sites. He would want to add my name to

User-Agents: dnns
Disallow: /

Is there a way to tell the website "a name" of my crawler? I'm looking forward to your answers. To me this seems an oddly obvious questions, however I did not find anything while searching for it.

Cheers,

Dennis


texts: About Natural Question (NQ) benchmark in NLP

texts:
I recently learned that there is a benchmark called NQ.
https://ai.google.com/research/NaturalQuestions/visualization
Unlike other QA benchmarks which relevant document is povided with query, it has to find information from millions of corpus by itself.
For example, if question is "when are hops added to the brewing process?" other QA benchmark also provide only 1 document about brewing. While NQ provide whole wikipedia text and model has to find most relevant document and answer.
When I tried all the example in the NQ with google search engine it gave me the answer every time. Then

How does google search engine is managing those questions so well(Since NLP such as BERT is pretty computationally expensive I think it is not likely google is running whole model for every search)
If google is doing it in other ways than neural network.(Or mixing it with other method) Do we need to bother with method such as REALM(https://arxiv.org/pdf/2002.08909.pdf) which seems to be computationally heavy?


texts: How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)?

texts:
Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit an LSTM neural net. From $X$ I make a new input data and its corresponding output data. So, we have:
$X = ( (X_1, ..., X_m), ... , (X_{n-m}, ..., X_{n-1}) )$
$Y = (X_{m+1}, ..., X_n)$
$\text{Card}X = \text{Card}Y$
Let's set $p$ the size of my test set. If I use Python's notation, we have:
$X_{train} = X[:-p]$
$X_{test} = X[-p:]$
Idem for $Y$ ... Now, I am wondering how to standardize my data. I think that standardizing $X$ before splitting the data into train and test sets could lead to over-fitting since we a transformation that involves all $X_i$. Basically, I am not sure that the sum (mean, standard deviation) will drown the information. In this case I think it could be better to just compute the mean and the standard deviation in the training set and use them to standardize both of the train and test sets. For me it makes no sense to standardize them separately since $\text{Card}X_{test} << \text{Card}X_{train}$. But may be I am wrong. I would also like to know whether I have to standardize $Y$ and $X$ or just $X$. When I am working with MLP neural net I used to just normalize the input data..

So, thank you first for reading and if you have any ideas or remarks, any questions to ask, please let me know. I can explain more it is up to you :)

P.S. I don't find a 'standardization' tag and I thus use the tag named normalization.


texts: How do I calculate the weight values for a piece-wise linear ReLU network approximating function?

texts:
I'm currently studying ReLU neural networks and their ability to approximate functions. Specifically, I'm looking at using ReLU networks to approximate a function using piece-wise linear components. I've been referencing this paper and specifically the excerpt in the snippet below.

Let's say I have a known function, how would I go about actually generating a ReLU network that approximates it? That is, I want a ReLU neural network that outputs a linear piece-wise function. For example, looking at the right plot on the image below:

My issue here is that even when I know the function, it's very difficult to to determine the weights to turn on the ReLU with the correct slopes at the correct time.

I can cause delays using the ReLUs by modifying the b parameter (per the equation), however, it seems like the current slope must be calculated such that all the previous ReLU slopes are negated.. this is due to the fact that once a ReLU turns on for some X input, it doesn't turn off as X increases.

So my question: Assuming I know a function perfectly (can calculate slopes, etc.) and I want to approximate it piece-wise linear using a ReLU network, how do I actually determine the weight values (i.e. slopes of each ReLU)? This appears to be a non-trivial task.


texts: saving the images to a folder with custom filenames

texts:
I'm new to Python and need assistance. I am performing segmentation with my own medical data set. The test images in my folder are named like "1.0.2.34.56_1.png". I would like to remove the actual image extension and append the image names with "_mask.png". The current code throws an error.

def labelVisualize(num_class,color_dict,img):
    img = img[:,:,0] if len(img.shape) == 3 else img
    img_out = np.zeros(img.shape + (3,))
    for i in range(num_class):
        img_out[img == i,:] = color_dict[i]
    return img_out / 255

def saveResult(save_path,npyfile,flag_multi_class = False,num_class = 2):
    for i,item in enumerate(npyfile):
        img = labelVisualize(num_class,COLOR_DICT,item) if flag_multi_class else item[:,:,0]
        io.imsave(os.path.join(save_path,"%d_mask.png"%i),img)
saveResult("data/membrane/test/",results)


texts: VC-dimension of the infinite set of convex bodies?

texts:
I try to find and prove the VC-dimension of the infinite set of (uni-directional) convex bodies. From intuition, it's clear to me that it goes to infinity, but I don't know the correct way to prove it formally.
Any clue for direction?


texts: Using a GAN discriminator as a standalone classifier

texts:
The goal of the discriminator in a GAN is to distinguish between real inputs and inputs synthesized by the generator.

Suppose I train a GAN until the generator is good enough to fool the discriminator much of the time. Could I then use the discriminator as a classifier that tests whether an input belongs to a single class?

For instance, if I train StyleGAN to be able to synthesize photorealistic cats, could I use the trained discriminator to detect whether an image is a cat or not?

My thinking is that perhaps the discriminator would be more accurate than other classifier models because it has effectively trained on many, many more inputs thanks to the generator.

On the other hand, perhaps the discriminator is somehow worse because it has been trained overwhelmingly on cat-like images (assuming the generator has gotten pretty good), and hasn't seen a wide variety of negative examples. It is concerned less with "is this a cat?" than "what are the tell-tale signs of this being synthetic?"


texts: Calculating correlation of slightly out of sync data

texts:
I am trying to do some analysis on some data that comes from special glasses that track a few things including pupil size and gaze velocity. I  would like to calculate the correlation between two glasses on two different people. At the moment I cannot use df.correlate() because the timestamps are not identical and therefore the data looks something like this:

index | ts | r_person | l_person
-----------------------
0     | 23 | 3.0      | NAN
1     | 25 | NAN      | 3.2
2     | 28 | 3.1      | NAN
3     | 32 | 3.0      | NAN

I was wondering if there was still any way to calculate a correlation directly.

At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.

This would be less trivial than it seems because even at the start it wasn't always one data point R one data point L and after cleaning the data it has become less so. In other words multiple NAN values might appear in the same column as you can see in the example. I can still deal with that by just spreading out the average. My second technique was going to try and merge the values together if they were close enough. Bearing in mind the data was collected at 50hz.

My question is whether anyone has a quicker or better way of aligning the data without losing it or changing it too much?


texts: Python how to add a condition to the groupby while calculating the median of a field?

texts:
I need to add a new condition to the following line:

train["Age"].fillna(train.groupby("Ticket")["Age"].transform("median"), inplace=True)

Currently, its taking the median of the age based on tickets class.

I need to add to this groupby("Ticket") another condition, where train["PrSb"]==0.


texts: Optimizing regression weights for NN outputs with PyTorch

texts:
So I'm basically trying to fit a regression on the relation of the input and output of a neural network model. Then the idea is, that these estimated regression weights should be optimized to some specific target value (let's say matrix of ones). So the weights are optimized, but have a direct relation to the neural network weights. For a simple problem, I can find a solution on paper, but I fail to implement this. The loss is not decreasing, but oscillating.
# Some arbitrary PyTorch model
my_nn = Net()

# Our target regression weights
target = torch.ones(2, 2)

# Optimizer for NN weights
optimizer = optim.Adam(my_nn.parameters(), lr=0.0001)

# Arbitrary loss function
criterion = nn.MSELoss()

# Training loop
my_nn.train()
for epoch in range(500):
    print('Running Epoch: ', epoch)
    # zero the parameter gradients
    optimizer.zero_grad()

    # output of model is 2-dim
    y_pred = my_nn(X)

    # weights for regression model
    w1 = Variable(torch.randn(2, 2).type(torch.float), requires_grad=True)
    # optimizer for regression model
    optimizer1 = optim.SGD([w1], lr=0.001, momentum=0.9)
    for nest_epoch in range(1000):
        optimizer1.zero_grad()

        y_hat = torch.mm(X, w1)
        loss = criterion(y_hat, y_pred)

        # Need to add retain_graph, as it doesn't work otherwise
        # but it does work, this loss will get close to 0
        loss.backward(retain_graph=True)
        optimizer1.step()

    # this loss does not decrease
    loss = criterion(target, w1)
    print('Loss:', loss.item())
    loss.backward()
    optimizer.step()
```


texts:
I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is there any way to transform this nominal values into numeric ones in a way that the algorithm will work?

I've used preprocessing.LabeledEncoder(), but then when I apply the algorithm it finds anomalies - the values that most differ from the mean it seems.

Are there any examples of an algorithm or another way to transform the data?


texts: What is the best solution to replace NaN values?

texts: I'm thinking about using the normal distribution of a specific column that has missing values and replace them by random values generated using the normal distribution function of numpy on that specific column ? Replacing by zeros or the mode doesn't really make sense sometimes... When is it relevant to do so ?


texts: How does bottleneck layer reduce computations without compromising with performance?

texts:
I was reading an article explaining the google's inception model . There it was mentioned , that to reduce the number of computations , we use a bottleneck layer. But I was surprised , if the model was somehow compromising the performance , as the bottleneck layer introduces a layer that has very less number of features. From what I understand , as we decrease the number of features (dimensions) , there is a loss of information .
Also when I read a little bit more , I got to know that it has to do something with encoders, but I am unable to connect the dots.
So how does the bottleneck layer works exactly ?


texts: Get number of correct predictions for each class in Keras

texts:
I am having unbalanced dataset(1:93) and want to use kappa's metric.
However, for that I need to capture how many correct predictions are made for each class.

I have tried understanding from here and other google links.

Is it possible to capture class wise #correct predictions made?


texts: Audio files and their corresponding spectrograms for image classification process

texts:
Suppose I have a dataset of audio files that I have to use for whale sound classification. I am choosing the strategy of treating it as an image classification problem by using their corresponding spectrogram (frequency vs time plot) images. The image shown below shows an example how the whale calls look like in the Label B(B is a species of whale and C stands for negative samples) of the spectrogram.

Since the audio files will be of varying length, the pre-processing step would involve padding all the shorter length samples with zero to have a fixed length for all files. So the spectrogram images of all those shorter samples will have the whale call in the beginning (or somewhere) with the majority of the frequency-time area as mere noise from the padding. (The above example divided the audio sample into some frames (to divide them into positive and negative classes) and labelled them as B,C.)

If we were to use the spectrogram images as such, this would hinder the generalization of our CNN model to a large extent.

Or if we save the output from the pre-processing to .npy format (binary form), I guess this could go unnoticed (or not?). What will be the consequences of saving the images in .npy format and then using in our model

I am not sure whether I am correct with my reasoning or not. Can anyone help me out?


texts: Code or Package to cluster sequences (or time series) of different lengths based on HMM?

texts:
Is there any existing code or packages in Python, R, Java, Matlab, or Scala that implements the sequence clustering algorithms in any of the following 2 papers?

1) 'Clustering Sequences with Hidden Markov Models' by Padhraic Smyth (1997): https://papers.nips.cc/paper/1217-clustering-sequences-with-hidden-markov-models.pdf

The paper gives a probabilistic model-based approach to clustering sequences (or time series), using hidden Markov models (HMM).

2) 'Visual Cluster Exploration of Web Clickstream Data' by Jishang Wei, Zeqian Shen, Neel Sundaresan, Kwan-Liu Ma (2012): http://www.cs.tufts.edu/comp/250VIS/papers/VAST2012-ClickStream.pdf

The paper is quite relevant to 1), as it maps each high-dimensional sequences (each sequence may have different length or in other words different dimension) to a 2D map (self organizing maps) where the distance metric is no longer Euclidean distance as that present in the conventional Kohonen Self-Organizing Maps, but instead the distance metrics becomes the log likelihood of how each sequence fits in a candidate hidden Markov model (HMM). Then on the 2D self-organizing map, K-means is used to cluster the map's nodes.

I haven't found existing package or code that implements the above clustering algorithms. There's hmmlearn (https://github.com/hmmlearn/hmmlearn) Python package to fit sequences to HMM, and there's existing Python package to implement SOM (self-organizing maps, such as this one https://github.com/stephantul/somber), but I wonder if there's existing code to implement clustering algorithms for sequential data points, based on updating the distance metric of HMM's likelihood function (or negative log likelihood, etc)? It can be Python, R, Java, Matlab, or Scala, or any other languages.

Thanks!


texts:
Recentl I saw that there is some basic overlapping of functionality between keras and sklearn regarding data preprocessing.
So I am a bit confused that whether should I introduce a dependency on another library like sklearn for basic data preprocessing or should I stick with only keras as I am using keras for building my models.
I would like to know the difference for scenarios like

which is good for production
which will give me better and faster response
is there any issue with introducing dependency over other libraries for just 1 or 2 functionality
which has a better compatibility with other tools like tensorboard or libraries like matplotlib, seaborn etc.

Thanks in advance.


texts: How to measure a almost (?) ordinal classification?

texts:
I have a model where I predict classes to define instructions for a trader robot. The classes are -2, -1, 1 and 2 (strong sell, light sell, light buy and strong buy) and I'm using a simple confusion matrix to asses the performance of the model, however I would like to know if there are other ways that takes into account the "distance" between the classes.
I've found some content about ordinal regression, but I already made my classification in other way (not sure if this content applies to me though).
Any suggestions?


texts:
I am analyzing a problem where i have 5 diseases and the measure of effectiveness of a remedy in its 7 first applications. The data is organized as an Excel spreadsheet as follows:

(The spreadsheet has dosage % values until 1, e.g 98, 97, 96, ..., 3, 2 ,1)
I have all these values for diseases 2,3,4 and 5. I just don't know how to deal this Dosage (%) feature since it is combining interval data (e.g. 401-500) with real values (e.g 100, 99, 98..). My task is to create a dataset based in this spreadsheet. My first guess is transform these intervalar values into real values and replicate samples, but i think there is a more clever way of dealing with it.
Thanks in advance.


texts:
I recently wrote this as a list of what I have seen / can think of as problem sources that make it hard to reproduce (replicate?) an experiment. I think I have seen most of them, except for hardware errors.

My thought was this: A typical computer vision experiment might run a training on a GPU for many hours / several days. Even if a bit-flip happens only once every $10^9$ FLOPs, that would be 11500 errors per second with an Nvidia Titan GTX 1080 Ti. I don't know how this error would affect later calculations (how the problem is numerically conditioned).

So: Are there any reports on Hardware errors affecting experiments?

(Blog posts, journal articles, posters?)


texts: What software is being used in this image recognition system?

texts:
I was wondering if anyone knew which piece of software is being used in this video? It is an image recognition system that makes the training process very simple.

http://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn#t-775098

The example is with car images, though the video should start at the right spot.


texts: Linear Discriminant Analysis + bayesian theorem = LDA classifier??

texts:
I am new to machine learning and as I learn about Linear Discriminant Analysis, I can't see how it is used as a classifier.

I can understand the difference between LDA and PCA and I can see how LDA is used as dimension reduction method. I've read some articles about LDA classification but I'm still not exactly sure how LDA is used as classifier.

From what I understand, we consider the features vector x as multivariate gaussian distribution and use the bayesian rule to calculate which category gives the max probability for x, then it belongs to that category.

Is this understanding generally correct?


texts: gensim : KeyError: "word 'sso' not in vocabulary"

texts:
I am using Google pretrained word embeddings(word2vec) to train my model. while using getting an error as below :

KeyError: "word 'sso' not in vocabulary"

i am trying this below code:

import gensim
model = gensim.models.KeyedVectors.load_word2vec_format('D:/NLP  JIRA/GoogleNews-vectors-negative300.bin', binary=True)
print(model)
words = list(model.wv.vocab)
print(words)
print(model[words])
for words in op:
        print(words)
        obj = model[words]
        print(obj)

input i am passing as

op = [['observations', 'sso', 'clienttool', 'testing'], ['logouri', 'column', 'clients', 'table', 'accepting', 'values', 'null']]


texts: XGBoost regressor hyperparameter tuning with hyperopt leads to overfit

texts:
Using hyperopt to hyperparameter tuning on XGBoost regressor, I am receiving overfiting on the train set.
There is any suggestion how to solve it ?
I have used cross validation with early_stopping_rounds and it still doesn't improved.
I have tried to tune the gamma,  colsample_bytree, subsample, max_depth, min_child_weight and eta and still couldn't reached to good enough results without overfitting.
Would appreciate any help.
Thanks,
Roi.


texts: Brain Tumor Classification(yes/no) - Model performs extremely well on train and validation set but performs badly on test images from the internet

texts:
I am trying to do image classification on brain tumor dataset consisting of 400 normal brain mri images and 1.6k brain tumor mri images. In image preprocessing, I am applying histogram equalization followed by erode and then dilate. I used resnet101v2 as the base model and then added a dense layer of 256 nodes. I ran it for 20 epochs. Since it is of medical field, I am focusing more on the recall metric. All 3 : accuracy, precision and recall are great on training and validation set as we can see below.

But when I try for 20 images from the internet consisting of 10 yes and 10 no images, the recall falls to 60%. I didnt go for balancing the classes here by upsampling the "no" class as already the model is predicting "no" for tumor images.

Can anyone explain what is the problem here as I cant see any form of overfitting (although the validation metrics have higher values than training set metrics).Is the problem due to my model or the image processing I have done?
https://github.com/AryanMethil/Brain_Tumor_Detection/blob/master/Brain_Tumour_Detection.ipynb


texts: GlobalAveragePooling2D in inception v3 example

texts:
I'm a complete begginer at Keras. In the Inception v3 example at https://keras.io/applications/

# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)

# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# let's add a fully-connected layer
x = Dense(1024, activation='relu')(x)
# and a logistic layer -- let's say we have 200 classes
predictions = Dense(200, activation='softmax')(x)

As at first step it adds a GlobalAveragePooling2D layer, which is described as:

  Global average pooling operation for spatial data.

What GlobalAveragePooling2D does and why the example uses it instead of something like Flatten? Which information is averaged?


texts: Using partially defined features in an unified deep learning model

texts:
Suppose we have two types of feature A and B. A is defined for all kinds of samples while B is only defined for some of the samples. Here, B is partially defined does not mean B is missing value (such as NaN) for the rest of the samples. In fact, we can not define a well-defined feature B in such samples.
For example, for stock market and commodity market, price feature (A) do exist in both of them. However, only commodity market has the concept of inventory feature (B).
My question is that how we can use feature A and B simultaneously in an unified deep learning model (instead of using them separately in two different models and combine their outputs thereafter)? Is there any frames or neural network structures can handle such problem ?


texts: Clustering of very high dimensional data and large number of examples without losing info in dimensions

texts:
I'm trying to get a grasp on scalability of clustering algorithms, and have a toy example in mind. Let's say I have around a million or so songs from $50$ genres. Each song has characteristics - some of which are common across all or most genres, some of which are common to only a few genres and some of which are genre-specific.

Common attributes could be something like song duration, artist, label, year, album, key, etc. Genre-specific attributes could be like lead guitarist, trombone player, conductor, movie name (in case of movie soundtracks), etc. Assume that there are, say, $2000$ attributes across all possible genres.

The aim is to identify attributes that characterize subgenres of these genres. So of course, let's say for rock I can just collect all the attributes for all rock songs, but even that set of attributes may be too broad to characterize the rock genre - maybe there are some that are specific to subgenres and so I won't have the desired level of granularity.

Note that for the purpose of this example, I'm not assuming that I already know the subgenres a priori. For example, I'm not going to categorize songs into subgenres like post rock, folk rock, etc. and then pick out attributes characterizing them. I want to discover subgenres on the basis of clustering, if that makes sense.

In a nutshell, about a million songs belonging to $50$ genres and all songs collectively have $2000$ attributes. So for each song I'll make a vector in $\mathbb{R}^{2000}$ - each dimension corresponds to an attribute. If that attribute is present for that song, the corresponding element of the vector is $1$, otherwise $0$ (e.g. a jazz-related attribute will be $0$ for a rock song). Now I want to do genre-wise clustering. For each genre, not only do I want to cluster songs of that genre into groups, but I also want to get an idea which attributes are the most important to characterize individual groups.

On the basis of this clustering, I can identify subgenres (e.g. of rock music) that I can characterize using a subset of the $2000$ attributes.

My first question is: is there a better way to initialize the problem than forming 2000-dimensional vectors of ones and zeros?

Secondly, given the vast number of dimensions and examples, what clustering methods could be tried? From what I've surveyed, there are graph-based clustering methods, hierarchical, density-based, spectral clustering and so on. Which of these would be best for the toy example? I've heard that one can project the points on to a lower-dimensional subspace, then do clustering. But I also want which attributes define different clusters. Since attributes are encoded in the dimensions, with dimensionality reduction techniques I'll lose information about the attributes. So now what?


texts: SelectKBest and Correlation returns me excatly same feature selection. How?

texts:
Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But when I look the features ranking I saw that selectKBest with chi-square are generating excatly same results as Correlation. Is it possible or am I doing something wrong?

My all features consist of 64bit float continuous numbers, between [-8,11] and my target column is binary which can be only 0 or 1.

Updated on 05.09.19: I am STILL searching how can it be possible? I mean I can guess that both methods based on same formula and developed by same person but I need a proof for understand that clearly.

Correlation Function:

cor = data.corr()
# Class is my target column
cor_target = abs(cor["Class"])
# Want to get correlation values for every feature without target column
relevant_features = cor_target[cor_target > 0].drop(labels=["Class"])
#Top 1000 features
relevant_features = pd.Series(relevant_features, index=data.columns).nlargest(1000).index.values

SelectKBest function:

bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances_chi = pd.Series(fit.scores_, index=dataValues.columns).nlargest(1000).index.values

And the result relevant_features and feat_importances_chi have excatly same results.


texts: What's the best way to generate similar words?

texts:
Hi all I'm fairly up to date with all the NLP tasks out there (nlpprogress.com, paperswithcode.com) and great tools like (nltk, flair, huggingface etc). I want to take a single word, and predict a similar word, a little like the old "Google Sets" feature except extrapolating from a single example. I'm thinking GPT-3 might be the best bet with some seed text like
here is a list of similar things: banana,

and ask it to predict the next word.
transformer.huggingface.co is promising enough (though hilariously inadequate in itself) that I'm thinking GPT-3 indeed may well be the answer.
But the alternative is to navigate a treebank, through "type of" relationships… much, much faster and cheaper.
I've tagged this "semantic similarity" but really I don't want the relationship to be "similar", rather "is part of same set of".
thoughts most appreciated from actual practitioners in this space rather than hobbyists like me :)


texts: Decision tree and SVM for text classification - theory

texts:
I used 4 classifiers for my text data: NB, kNN, DT and SVM. As for NB and kNN I fully understand how they work with text - how we can count probabilities for all words in NB and how to use similarity metrics with TF-IDF vectors in kNN I don't understand at all how decision tree and support vector machine work with text data. I implemented all algorithms in Python so all I need is some resource or explanation how the other two classifiers work with text...
I understand DT with non-text data - it seams logical for example nodes with checking if some data is more/less than some number. But with text I get confused. Does it operate on text or with numerical vectors? The same applies to SVM...


texts: Algorithm for generating classification rules

texts:
So we have potential for a machine learning application that fits fairly neatly into the traditional problem domain solved by classifiers, i.e., we have a set of attributes describing an item and a "bucket" that they end up in. However, rather than create models of probabilities like in Naive Bayes or similar classifiers, we want our output to be a set of roughly human-readable rules that can be reviewed and modified by an end user.

Association rule learning looks like the family of algorithms that solves this type of problem, but these algorithms seem to focus on identifying common combinations of features and don't include the concept of a final bucket that those features might point to. For example, our data set looks something like this:

Item A { 4-door, small, steel } => { sedan }
Item B { 2-door, big,   steel } => { truck }
Item C { 2-door, small, steel } => { coupe }

I just want the rules that say "if it's big and a 2-door, it's a truck," not the rules that say "if it's a 4-door it's also small."

One workaround I can think of is to simply use association rule learning algorithms and ignore the rules that don't involve an end bucket, but that seems a bit hacky. Have I missed some family of algorithms out there? Or perhaps I'm approaching the problem incorrectly to begin with?


texts: Is it possible to decompose a scalar value to a inter-dependent vector neural network?

texts:
My data contains a scalar feature $r$, I found this feature is important for training my deep model. My idea is supposing there is a 3-layer MLP $f(x), x \in \mathbb{R}^{n}$, where $n=1$. It outputs a vector with dimension $m$ where each value in $[0, 1]$.

For my data, it inputs $r$ and outputs an m-sized vector.

So here is my decomposition idea make sense?


texts: Is a CNN architecture more susceptible to adversarial attack if it is built on ResNet?

texts:
The title kind of sums up my question. But more generally, is a CNN architecture more susceptible to an adversarial attack if it is built on top of another open source architecture, like ResNet. Can something that spoofed ResNet also spoof another model built on top of ResNet?

(Assuming you do not have access to the gradient of the secondary model, but of course do have access to ResNet)


texts: Build neural network to calculate points for a board game

texts:
I want to be able to calculate how much points each player has for a board game by taking a picture on the board game.
I do this as hobby, not for university or professional purposes. I will use it while playing with my friends and also I hope this will help me build some knowledge about ML.
I plan to do that with supervised learning. Take many pictures of the game and tell the machine what is the correct output.
The game scoring is not something straight-forward. Players may control different territories. A territory is controller if a player has units there even if there is enemy buildings. The game is Scythe. From my previous knowledge about ML I know that small changes in the image (for example the angle) from the images you gave to the machine to learn may bug it to recognise them.
Questions (I do not want someone to tell me how to do it but rather which approach is better so I can read more about it):

Is supervised learning the best option in this case?
I plan to build many micro neural-networks for the different cases - recognise who controls a territory and which territory (which territory seems very hard); separate network to recognise if there are resources on some players territory (this seems very hard, as the combination of resources might be from 0 to many; shall I just put different combinations of resources (+numbers of them) and take many pictures?). There are lots of combinations. I cannot take a pic on every of them.


texts:
The docs for sklearn.neighbors.DistanceMetric state that

  in order to be used within the BallTree, the distance must be a true metric

(i.e. be non-negative, 0 only if objects are equal, symmetric, and satisfy triangle inequality). My question is, how strict are those requirements? More specifically, would ball tree work for pseudometric (may be 0 for non-equal objects but satisfies other conditions)?


texts: Deciding the number of clusters in K-means clustering of descriptors

texts:
I am new to the Machine Learning area and I have a question to ask. But let me first post the problem.

Problem:
The problem is very simple. I want to classify images as Category-1("Images containing garbage") or Category-2("Images not containing garbage"). Garbage is used in every literal sense of the word.

Solution I opted for:
The solution also happens to be pretty straightforward for the most part. Extract points of interest using an Algorithm like SIFT,SURF etc etc. Then obtain the descriptors of these key points and cluster them using the K-means algorithm. Then using this clustered data generate the bag of words and proceed from there.

The thing which I am unable to comprehend is the number of clusters which I might need. Any help with the above example would be much appreciated.


texts: How to turn linear regression into logistical regression

texts:
I followed these articles to implement logistic regression.

I'm confused however because after training the model and getting the weights for my variables I don't now how to use the one-hot vector to turn this into confidence scores for the different classes.

I've got the formula: y' = x1W1 + x2W2 + x3W3 + b

I've got values for all Ws and b.

I've got my one-hot vector: [[1, 0, 0], [0, 1, 0], [0, 0, 1]]

How do I combine all this into confidence for each class?


texts: How to work with different Encoding for Foreign Languages

texts:
I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such as é, â, î or ô.
Let me explain my problem with the following example:
Consider these two words and their respective vectors, both of which are taken from model.txt:
etait -0.100460 -0.127720 ...

était 0.094601 -0.266495 ...

Both words signify the same meaning but the former is without the accents while the later has accents.
Now I'm trying to load this word embedding using the gensim.models.KeyedVectors in the following way:
model = KeyedVectors.load_word2vec_format(open(model_location, 'r',
                                              encoding='utf8'),
                                          binary=False)
word_vectors = model.wv

To which I get the following error:
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-82-e17c33c552da> in <module>
     10 model = KeyedVectors.load_word2vec_format(open(model_location, 'r',
     11                                               encoding='utf8'),
---> 12                                           binary=False)
     13
     14 word_vectors = model.wv

D:\Anaconda\lib\site-packages\gensim\models\keyedvectors.py in load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype)
   1547         return _load_word2vec_format(
   1548             cls, fname, fvocab=fvocab, binary=binary, encoding=encoding, unicode_errors=unicode_errors,
-> 1549             limit=limit, datatype=datatype)
   1550
   1551     @classmethod

D:\Anaconda\lib\site-packages\gensim\models\utils_any2vec.py in _load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype, binary_chunk_size)
    286                 vocab_size, vector_size, datatype, unicode_errors, binary_chunk_size)
    287         else:
--> 288             _word2vec_read_text(fin, result, counts, vocab_size, vector_size, datatype, unicode_errors, encoding)
    289     if result.vectors.shape[0] != len(result.vocab):
    290         logger.info(

D:\Anaconda\lib\site-packages\gensim\models\utils_any2vec.py in _word2vec_read_text(fin, result, counts, vocab_size, vector_size, datatype, unicode_errors, encoding)
    213 def _word2vec_read_text(fin, result, counts, vocab_size, vector_size, datatype, unicode_errors, encoding):
    214     for line_no in range(vocab_size):
--> 215         line = fin.readline()
    216         if line == b'':
    217             raise EOFError("unexpected end of input; is count incorrect or file otherwise damaged?")

D:\Anaconda\lib\codecs.py in decode(self, input, final)
    320         # decode input (taking the buffer into account)
    321         data = self.buffer + input
--> 322         (result, consumed) = self._buffer_decode(data, self.errors, final)
    323         # keep undecoded input until the next call
    324         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 7110-7111: invalid continuation byte

which I thought made sense if my file was encoded in a different format. However, using git I tried checking the encoding of the file using file * and got:
model.txt: UTF-8 Unicode text, with very long lines

Now, if I try to write the above code and have the encoding set to latin1, there isn't any problem to load this document but at the cost of not being able to access any of the words which contains an accent. Essentially throwing an out-of-vocab error upon executing:
word_vectors.word_vec('était')
How am I supposed to approach the problem? I've also got the .bin file of the model, should I try to use that to load my words and their corresponding vectors?


texts: Model not learning when using transfer learning

texts:
I am working on a personal project on image classification (two classes) and am trying to see how the MobileNet v2 structure would perform. While training the training accuracy is already quite high after the first epoch after which is continues to increase. The validation accuracy, however, seems forever stuck around the 50% mark.

I have tried both SGD, Adam, and RMSprop as optimizers, however they both produce the same results. I have also used both a MobileNet v2 with and without ImageNet weights without seeing any difference. Other CNN architectures work perfectly fine, achieving up to ~95% validation accuracy. I am using the following code for my model:

from keras.applications import MobileNetV2
from keras.models import Sequential
from keras.layers import GlobalAveragePooling2D, Dropout, Dense

basemodel = MobileNetV2(input_shape=(128, 173, 3), include_top=False, weights=None)
model = Sequential(
    [
        basemodel,
        GlobalAveragePooling2D(),
        Dropout(0.5),
        Dense(64, activation="relu"),
        Dropout(0.3),
        Dense(1, activation="sigmoid"),
    ]
)
model.compile(loss="binary_crossentropy", optimizer="sgd", metrics=["accuracy"])

Is there something obvious that I am missing or am I just using incorrect hyperparameters for my dataset?


texts: Hive External table does not showing in Namenode (Cloudera-QuickstartVm)

texts:
I have created external table in Hive in the hdfs path 'hdfs://localhost.localdomain:8020/user/hive/training'. If I apply describe command I can find the table path as shown below. But when I browse through the namenode web page, the table name does not showing up in the path.

hive> describe extended testtable4;
OK
firstname   string
lastname    string
address string
city    string
state   string
country string

    ***Detailed Table Information  Table(tableName:testtable4, dbName:default, owner:cloudera, createTime:1408765301, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:firstname, type:string, comment:null), FieldSchema(name:lastname, type:string, comment:null), FieldSchema(name:address, type:string, comment:null), FieldSchema(name:city, type:string, comment:null), FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country, type:string, comment:null)], location:hdfs://localhost.localdomain:8020/user/hive/training, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=,, field.delim=,, line.delim=
    }), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{EXTERNAL=TRUE, transient_lastDdlTime=1408765301}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE)
    Time taken: 0.7 seconds***


texts: What is the best strategy to use on data with many classification labels?

texts:
In general, what sort of supervised algorithms and techniques should I use on data that has the following charactersitcs:

2 potential classification labels?
3-5 potential classification labels?
6-10 potential classification labels?
10-50 potential classification labels?
50 or more potential classification labels?

My Primary Questions:

What algorithms learn most effectively at these different tiers of total possible class labels?
What algorithms generally make the best predictions with the smallest amount of data at each of these tiers?

I know at some point it makes more sense to use regression rather than a classifier. How many potential class labels would this be?


texts: Evaluating loss for non classifying convolutional neural network

texts:
Sorry if my question is kind of dumb, I am very new to this field.

I am trying to create a CNN that plays a variant of chess (for the examples, we'll use chess as it is close enough).

My network , which is a policy network, outputs a vector of planes of probabilities. (e.g 1st pawn layer, 2nd rook layer etc..) each layer contains scalars determining how "good" would a move of the piece to this case be, according to the network. My question is, given the input channels I. And expected outpus (0s everywhere and 1 for the move that was played), how to calculate loss for gradient descent ? (e.g : how to evaluate the "closeness" of a move to another.).

My wild guess is to weight each layer a "wrongness" factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really "wrong" and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ? And in general , how to compute the loss of a non classifying convolutional neural network ?


texts: How to draw a hyperplane using the weights calculated

texts:
In the simple example where I have n input neurons, I can consider this to be a point in a n-dimensional space.

If the output layer is just one neuron with value 0 or 1, if I get convergence, the neural net should define a hyperplane dividing the two classes of points - those mapped to 0 and those mapped to 1. How do I calculate this hyperplane using the (n,1) matrix of weights I calculated?


texts: Comparing text corpora sentence-wise in Python

texts:
I have two corpora (in the form of list of sentences); one is the "original" one, the other is a version of it with some minor changes (with contractions expaned, for instance):

s1 = ['He's a real stupid', 'There's no need to worry', 'All right, my friends']
s2 = ['He is a real stupid', 'There is no need to worry', 'All right, my friends']

I'd like to get some rendering of the corresponding sentence pairs where IS some difference between the sentences. Is there any elegant solution to this?


texts: Bootstrapping or Randomly Dividing Dataset to reduce variance?

texts:
If I have 10,000 training samples then what should I do:

Bootstrapping and train 10 classifiers on it and then aggregating

Or

randomly divide the dataset into 10 parts and train 10 classifiers on them and then aggregating. Which will be better?

Will the 2nd method reduce variance and will it be better than 1st method


texts: Data Scientist Prerequisites: What are the essential prerequisite statistics topics?

texts:
I have may years of experience with database application development mainly focused on SQL Server and Oracle.

I like to move into Data Science field and I my understanding is that I need a solid knowledge of Statistics and background - I wonder if that is correct?

I wonder what are the important statistic topics that I must know as data science prerequisites?

I am reading the following book:

Statistics in a Nutshell 2nd Edition (http://amzn.com/1449316824 )

Which of the following topics are important for data science or is there any other important topic that I must learn before I start Data Science?

Chapter 3 Inferential Statistics
Chapter 4 Descriptive Statistics and Graphic Displays
Chapter 6 The t-Test
Chapter 7 The Pearson Correlation Coefficient
Chapter 8 Introduction to Regression and ANOVA
Analysis of Variance (ANOVA)
Chapter 10 Multiple Linear Regression
Chapter 11 Logistic, Multinomial, and Polynomial Regression
Chapter 12 Factor Analysis, Cluster Analysis, and Discriminant Function Analysis

Thank you


texts:
I am learning about pandas and find an interesting case
i try to find total sales for every genre that exist in the data frame,I try looping a few time and get a dead end,maybe you can give me a few tips about this
here is my code
import pandas as pd

df = pd.read_csv('vgsales.csv')
dcheck = df.dtypes

#check value count in data frame
genre_values = df['Genre'].value_counts()

#check what value in dataframe
genre2 = df['Genre'].unique()

zero = 0

def multiply():
    while genre2['Racing'].isin(df['Genre']):
        df['Global_Sales'][0] + zero

result = multiply()


texts: Policy gradient: why does this converge with Adam and not SGD?

texts:
I am looking into policy gradient methods. I stumbled into this implementation:
https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a
It uses a Nesterov Adam optimizer.

If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.

However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?

NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.


texts: Interpreting Results of Multivariable Regression / how to transform variables to improve results

texts:
I am working on a project that predicts the Market Cap (value) of different crypto-currencies. My data is very small (51 observations) and I initially have 18 X-variables. I was hoping to get feedback on my modeling approach and results, and suggestions on improving the model (particularly by transforming variables / with a non-linear regression) technique. I will do my best to keep the post clear and brief, and hope it can be helpful to others working on a similar analysis. The post may seem long, but a lot of it is images. Also, I am doing this in R, and can share any code on request.

1.) My first action was to log-transform the Market Cap (Y-variable) into log(Market Cap). Here are 2 graphs, of Market Cap and of log(Market Cap) of my 50 observations:

...the outlier point, with a $190B market cap, is bitcoin, and the model badly overfit to this data point if I did not log-transform. For this reason, I think this first action of log-transforming the Market Cap is a good action.

2.) 12 of the 50 observations were missing values for 6 of the X-variables. Rather than throwing these observations away, I predict missing values for these 6 X-variables by fitting, for each of these 6 X-variables, a simple linear regression between the (a) the X-variable with missing values, and (b) one of the remaining 12 X-variables with no missing values. For (b), I chose the X-variable with the highest pearson correlation to the missing-value-X-variable. I use this simple linear regression to predict missing values.

3.) I then fit an initial linear model with all 18 variables. However, since 18 variables will almost certainly overfit to 50 data points, and because there is some multicollinearity to the X-variables, I then use stepwise regression with backward elimination to fit a model with fewer variables. The R summary output of the model returned using backward elimination, with 9 variables, is here:

and the R diagnostics plots from this model are:

4.) Lastly, here is a grid with variable distributions and correlations for these 9 predictor variables + the log(Market Cap):

as well as a table of Variance Inflation Factors

My thoughts on next steps are to fix the remaining multicollinearity issue by removing additional variables, and also remove some outlier data points indicated in the Residuals vs. Leverage graph. I also plan to test the model on a test data-set (I did a 40 / 10 split), 5 times using 5-fold cross validation.

I am interested in anyone's thoughts on transforming the X-variables, which I currently do none of. The grid of histograms / correlations / scatter plots shows that many variables are right-skewed. Additionally, I have graphed the simple linear regression fit between log(Market Cap) and each of the 9 remaining variables, and received these plots (for 4 of the 9):

And noticed for the most part that none of the X-variables seem to have a great linear-fit with log(Market Cap).

Any thoughts on next-steps on my model building would be greatly, greatly appreicated. Apologies again for the long post, but I felt that thorough / lots of images / plots would be helpful here. Thanks!


texts: Damerau-Levenshtein Edit Distance in Python

texts:
I found some python codes on Damerau Levensthein edit distance through google, but when i look at their comments, many said that the algorithms were incorrect. I'm confused.

Can someone share a correct python code on Damerau Levensthein Distance?

Thank you.


texts:
I was wondering if labels could be visualized below images in the image grid tool in Image Analytics?
I know, this feature might be not terribly useful for images in general, hence unlikely to be deployed in Orange. How hard it would be to modify  image_grid.py in order to display such image names? Any pointers would be useful.


texts: What would cause a Hierarchical Cluster to look like skewed (if this is the right term to use!)?

texts: I am surprised about this hierarchical cluster! For me, it looks somehow abnormal. Or, maybe normal but I am not able to identify why it looks like this. Any idea why data would be clustered in such a structure?


texts: What is "position" in CNN (im2latex) for Positional Encoding?

texts:
I'm trying to build a model that maps images of math
formulas into LaTeX markup.
I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder architecture that solves this problem.
In encoder after the work of CNN it's proposed (page 3) to add following signals to generated feature maps "in order to retain the spatial locality information":
$$PE(x,y,2i)=sin(x/10000^{4i/D})$$
$$PE(x,y,2i+1)=cos(x/10000^{4i/D})$$
$$PE(x,y,2j+D/2)=sin(y/10000^{4j/D})$$
$$PE(x,y,2j+1+D/2)=cos(y/10000^{4j/D})$$
Here the authors refer to positional encoding part of Transformer model "tailoring the 1-D positional encoding technique to 2-D".
In original Transformer model (from article "Attention Is All You Need") the position encoding is calculated like this:

Where "pos" is just an index number of word in given sentence (correct me if I'm wrong).
And here comes the question: what is "x" and "y" in first formula?


texts:
Quoting from this example of Predicting movie review with BERT and Tensorflow,

with tf.variable_scope("loss"):
        # Dropout helps prevent overfitting
        output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)

        logits = tf.matmul(output_layer, output_weights, transpose_b=True)
        logits = tf.nn.bias_add(logits, output_bias)
        log_probs = tf.nn.log_softmax(logits, axis=-1)

        # Convert labels into one-hot encoding
        one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)

        predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))
        # If we're predicting, we want predicted labels and the probabiltiies.
        if is_predicting:
            return (predicted_labels, log_probs)

        # If we're train/eval, compute loss between predicted and actual label
        per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
        loss = tf.reduce_mean(per_example_loss)
        return (loss, predicted_labels, log_probs)

How is the loss between actual and predicted labels being calculated?

The above example id for multi-class, single-label setting. If I want to calculate the loss for a multi-class, multi-label problem, how can I do the necessary computation in Tensorflow?


texts: Custom metrics for unbalanced classes problem in RandomForest or SVM

texts:
My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of >100k samples. Classifying foreground class as background is quite OK, while classifying background as foreground should be penalised.

I am using Scikit-learn's RandomForests, and I was experimenting with SVM and OneVsRest classifiers as well. I would like to specify the scoring metrics used for the method fit() of the model, so it will correspond to my goal (I imagine something like fitness function with evolution algorithms). However, API does not allow something like that.

So far I tried:

Use class_weight parameter of the model. If I set it so it represents the real world, then the classifier learns to classify everything as background having accuracy >99 %. If I set class_weight = 'balanced', then it seems better, but it has high false positive rate.
Use scoring method for GridSearchCV, which outputs values specified by me (even F1-score makes more sense than simple accuracy), but it is used only for the parameter selection and the final model is learnt by fit() method, which again ignores my scoring.

Is there a way to provide my own custom scoring function for the fit() method?


texts: Combining text (NLP), numeric and categorical data for a regression problem

texts:
I have a dataset

data = {
    points: 3.765,
    review: `Food was great, staff was friendly`,
    country: 'Chile',
    designation: 'random',
    age: 20

}

I am looking for a way to use these features to build a model to predict points.

Description seems to hold alot of information about points.

How do i feed this data into the model and also which model?

Note I dont want to use word2vec (embeddings)


texts: Clustering for mixed numeric and nominal discrete data

texts:
My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individual level.
Data is of shape (n=7219, p=105).
Couple things:

I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data. There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.

I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.


texts: How can I do classification with categorical data which is not fixed?

texts:
I have a classification problem with both categorical and numerical data. The problem I'm facing is that my categorical data is not fixed, that means that the new candidate whose label I want to predict may have a new category which was not observed beforehand.

For example, if my categorical data was sex, the only possible labels would be female, male and other, no matter what. However, my categorical variable is city so it could happen that the person I am trying to predict has a new city that my classifier has never seen.

I am wondering if there is a way to do the classification in these terms or if I should do the training again considering this new categorical data.


texts: I am an undergraduate that is currently working on a project in which I need to collect reviews and comments about mobile phones across several platforms and analyse the data to see if the sentiments and reviews have direct impact on the sales of the mobile. My problem is the best method to get this data from sites like GSM arena and the rest of them.


texts: SVM - Making sense of distance derivation

texts:
I am studying the math behind SVM.
The following question is about a small but important detail during the SVM derivation.
The question
Why the distance between the hyperplane $w*x+b=0$  and data point (in vector form) $p$, $d = \frac{w * p + b}{||w||}$  can be simplified to $d = \frac{1}{||w||}$?
My argument
Since data point $p$ is not on the hyperplane, then we have $w*p+b=k, k \ne 0$. Then $d=\frac{k}{||w||}$ but $k$ is not a constant as it depends on $w$. So optimizing $d=\frac{k}{||w||}$ is not the same as optimizing $d = \frac{1}{||w||}$.
Mentions of the distance calculation step in materials:

page 4 of https://www.ugpti.org/smartse/resources/downloads/support-vector-machines.pdf
slide 10 of http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf
slide 11 of https://www.cs.upc.edu/~mmartin/SVMs.pdf


texts: How is computed the train data set score and why?

texts:
In a lot of machine learning blogs or review, the training dataset accuracy (or other metric) is given alongside the test dataset score. Is this score calculated through the training, or is the training data given at the final model to see if it gives a good score ?

Moreover, what is the point of computing such a score, as the model has been trained with this data ?


texts: we are trying to build an in house OCR system to extract alphanumeric strings from images (kindly note, most of our clients cant afford to send their data onto the cloud so it rules out any API from the googles and microsofts..and also tesseract is way too slow for our purpose ..we are talking 10's of users simultaneously uploading images .. tesseract, i guess was never designed to be used as such). We have already started using a CTC based approach to train on our data sets but there's a fundamental/conceptual issue. By definition, CTC finds a happy path to the most probable unit (in this case, a word) but alphanumeric strings by definition DO NOT have a pattern ! So i have a feeling CTC will not work for us. But juxtaposed to that is the multiclass classification approach (where we just predict every individual char) which has this big problem about how to split any image into specific characters before passing it to a CNN layer. We are a little clueless about the approach to be taken here. Any pointers will be heartily appreciated :)


texts: How much data is created daily (source of IBM estimation)

texts:
I've stumbled upon this IBM text.

It states that:

  Every day, we create 2.5 quintillion bytes of data

It's very interesting statistic, but I couldn't find any source for this.

Does anyone know how they calculated this? Or maybe someone knows some papers showing similar/different results?

This paper from CSA has the same statistic, but again without any source.

Note: The quote from IBM's text refers to data created by humanity, not by IBM alone.


texts: Why is this binary state format so much more efficient than a non-binary one?

texts:
I am teaching myself Deep Q Learning (and machine learning in general) using python and tensorflow to play Snake. My initial learning came mostly from this article. I have rewritten it in my own coding style and it is working well.
Now I am experimenting with different parameters and state information. The state for my game is an array of 11 values that looks like this:
0 - Danger 1 step ahead
1 - Danger 1 step on the right
2 - Danger 1 step on the left
3 - Snake is moving left
4 - Snake is moving right
5 - Snake is moving up
6 - Snake is moving down
7 - Food is on the left
8 - Food is on the right
9 - Food is on the upper side
10 - Food is on the lower side

All of these values are binary, either 0 or 1. To experiment, I wanted to switch out elements 3 through 6 with the constants I use for direction.
DIR_UP = 1
DIR_RIGHT = 2
DIR_DOWN = 3
DIR_LEFT = 4

So the state object would be an array of 8 elements, element 3 being non-binary:
0 - Danger 1 step ahead
1 - Danger 1 step on the right
2 - Danger 1 step on the left
3 - Snake movement direction
4 - Food is on the left
5 - Food is on the right
6 - Food is on the upper side
7 - Food is on the lower side

To me, this seems to have all the same information but in a different format, but the DQN apparently doesn't think so. After 150 training iterations on the first format, I get a competent snake that can regularly score 40 or more. To get the same scores on format 2 I needed to run 800 iterations with a slower epsilon decay. And in testing (without any randomness) it still takes indirect routes and needlessly crashes into the wall now and then, which I thought would have been weeded out.
So my main question is: what am I failing to understand about state format that makes my first so much more efficient than my second?
And sub-questions: are states supposed to be binary? What if I have to account for something like distance that can't be easily represented that way?
Thanks so much for you time and input!


texts:
I have created some document embeddings which were then used further in text classification tasks. After revisiting my code I was unsure about the workflow I used to train the document embeddings.

At the moment I am creating the document embeddings based on the complete corpus available at the time of training. After the training is done, I evaluate the model by looking whether it creates useful similarities between the document embeddings. Those embeddings are used then in machine learning models and that's where the embeddings will be split into train, test and validation sets.

Now my question is: Where is the right time to split the data? Should I do it before creating the document embeddings to prevent data leakage? I have used the mentioned approach because I viewed the creation of the document embeddings as a preprocessing step, so the computer can work with textual data. However, after I have put some thought into it, I think it's the wrong approach. I wanted to hear from more experienced NLP practitioners how they approach this task. Sorry for this very basic question.

Thanks.


texts: How are deep-learning NNs different now (2016) from the ones I studied just 4 years ago (2012)?

texts:
It is said in Wikipedia and deeplearning4j that Deep-learning NN (DLNN) are NN that have >1 hidden layer.

These kind of NN were standard at university for me, while DLNN are very hyped right now. Been there, done that - what's the big deal?

I heard also that stacked NN are considered deep-learning. How is deep-learning really defined?

My background of NN is mostly from university, not from jobs:

studied applications of NN in industry
had about 5 courses on artif. intel. & mach. learn. - though maybe 2 of them on NN
used NN for small, simple project on image recognition - used 3 layer feed-forward NN
did not do real research (as in doctor thesis) on them


texts: ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). Any advice?

texts:
The error below presented itself when attempting to assemble a PCA.
My code:
  is_float = X.dtype.kind in 'fc'
    if is_float and (np.isfinite(_safe_accumulator_op(np.sum, X))):
        pass
    elif is_float:
        msg_err = "Input contains {} or a value too large for {!r}."
        if (allow_nan and np.isinf(X).any() or
                not allow_nan and not np.isfinite(X).all()):
            type_err = 'infinity' if allow_nan else 'NaN, infinity'
            raise ValueError(
                    msg_err.format
                    (type_err,
                     msg_dtype if msg_dtype is not None else X.dtype)
            )

The error:
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Any help or advice would be deeply appreciated!
Thanks!


texts: How to construct pipeline with different alternative transformations for different kind of features in Scikit-learn?

texts:
I try to construct a pipeline in sklearn where I do different (in some cases multiple) transformations on different kind (numeric/ordinal/binary nominal/non-binary non-ordinal nominal) features. An additional tweak is that I want to try out different (and sometimes None) kind of the specific transformations in the pipeline.
So far I have tried the following:
preprocess = make_column_transformer(
    (numerical_columns, make_pipeline(RobustScaler(), PolynomialFeatures())),
    (categorical_columns, make_pipeline(OneHotEncoder())),
    (ordinal_columns, "passthrough"),
    (binary_columns, "passthrough"),
)

search_pipeline = Pipeline([("preprocessing", preprocess),
                     ("dimred", PCA()),
                     ("classifier", RandomForestClassifier())])

search_parameters = [
            {"preprocessing__pipeline__robustscaler": [None]},
            {"preprocessing__pipeline__robustscaler": [RobustScaler()]},
            {"preprocessing__pipeline__robustscaler": [StandardScaler()]},

            {"preprocessing__pipeline__polynomialfeatures": [None]},
            {"preprocessing__pipeline__polynomialfeatures": [PolynomialFeatures(degree=2)], "preprocessing__pipeline__polynomialfeatures__interaction_only": [False, True]},

            {"dimred": [None]},
            {"dimred": [PCA()], "dimred__n_components": [.95, .75]},
            {"dimred": [LinearDiscriminantAnalysis()], "dimred__n_components": [.95, .75]},

            {"classifier": [KNeighborsClassifier(weights="distance")],
             "classifier__n_neighbors": [3, 7, 11]},
            {"classifier": [RandomForestClassifier(n_estimators=100, class_weight="balanced")],
             "classifier__max_depth": [5, 10, None]}
            ]

As you can see, for example I tried to apply different kind of scaler methods for numerical features:
None
RobustScaler
StandardScaler
However after running GridsearchCV:
CV = GridSearchCV(search_pipeline,
                  search_parameters, cv=5,
                  scoring="f1_weighted",
                  refit=True,
                  n_jobs=-1)

CV.fit(train_X, train_y)

I get error message:
ValueError: Invalid parameter robustscaler for estimator ColumnTransformer(transformers=[('list-1',
                                 ['income', 'reside', 'address', 'wireten',
                                  'tollten', 'equipten', 'cardten', 'longten',
                                  'age', 'employ', 'tenure'],
                                 Pipeline(steps=[('robustscaler',
                                                  RobustScaler()),
                                                 ('polynomialfeatures',
                                                  PolynomialFeatures())])),
                                ('list-2', ['region', 'custcat'],
                                 Pipeline(steps=[('onehotencoder',
                                                  OneHotEncoder())])),
                                ('list-3', ['ed'], 'passthrough'),
                                ('list-4',
                                 ['retire', 'callid', 'gender', 'marital',
                                  'tollfree', 'equip', 'callcard', 'wireless',
                                  'multline', 'voice', 'pager', 'internet',
                                  'callwait', 'forward', 'confer', 'ebill'],
                                 'passthrough')]). Check the list of available parameters with `estimator.get_params().keys()`.

I suspect that the syntax in search_parameters to access specific transformers' specific parameters is incorrect, but what is correct?


texts: Andrew Ngs Class - Why Did He Change up the Cost Function?

texts:
I am taking Andrew Ng's Machine Learning Intro class. Looks like he changed the cost function without any explanation in the second week. Specifically:

He no longer squares each deviation between the estimated and the actual value
He no longer divides the sum of deviations by 2 (I guess because he doesn't square it, but that leaves the question of why for both
He multiplies the whole thing by the first number in the sample for which he is finding the delta between estimated and actual. Why?

This image might explain it better for those familiar with the class:

Also, which of the two is "correct"? Or if there is no "correct" answer, what is the use case for each?


texts: Does the length of the text data affects the score of clustering

texts:
I'm learning about clustering with Pythons Scikit-Learn lib.
I have a list of sentences (strings) . Im wondering, does the length of the string affects the silhouette_score.

For example, I have sentences from 2 words to 35 words, and i have tried number of clusters from 2 to 60, and the biggest silhouette_score that I get is around 7. Can that affect the silhouette_score?
Is it better to filter my data, so that I can select sentences that are much closer by number of words, for example, to set number of words from 20-25, or 5-10?

This is how my code looks like:

list_of_comments = data

#cv = TfidfVectorizer(analyzer = 'word', max_features = 6500, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')
cv = CountVectorizer(analyzer = 'word', max_features = 8000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')

x = cv.fit_transform(list_of_comments)

my_list = []
list_of_clusters = []
for i in range(2,35):

    kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
    kmeans.fit(x)
    my_list.append(kmeans.inertia_)

    cluster_labels = kmeans.fit_predict(x)

    silhouette_avg = silhouette_score(x, cluster_labels)*100
    print(round(silhouette_avg,2))
    list_of_clusters.append(silhouette_avg)


texts: Question about Relative-Position-Representation code

texts:
In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py
In _relative_attention_inner method, which I think is one of the core code for https://arxiv.org/abs/1803.02155
x_t_r shape is [sequence_length, batch_size, hidden_size]
z shape is [sequence_length, sequence_length, hidden_size]
then tf.matmul x_t_r and z to get a tensor with shape [batch_size, sequence_length, sequence_length], How To Understand This Line? How To Understand The Insight of This Line?
x_tz_matmul=tf.matmul(x_t_r, z, transpose_b=transpose)

To be brief, I remove the heads in shape.


texts: How can i interpret cluster attributes from clusters in hierarchal clustering?

texts:
I have data with numerical columns for products bought e.g. price, etc, many are numerical but it is also crucial i incude the industry classification of product e.g/ fashion, gardenware etc.
If i use say gower distance then run this distance matrix through hierarchal clustering how can i interpret the results? if you look at this example here: https://www.kaggle.com/rajshree07/hierarchical-clustering-behavioral-segmentation you can see that a way to interpret the clusters is by plotting the data points against two features... But how can i interpret the clusters with ALL features?
Is it just a case of calculating summary statistics like mean, median, mode, min , max of each feature for each cluster assigned by the hierarchal algorithm? or is there another way


texts: How to tell how much information I lose when I simplify the graph data structure with respect to unsimplified graph?

texts:
I have the following problem: I have some sort of data (that I can't publish here, but they are in the form of points with XYZ coordinates) and I can represent them as a collection of graphs i.e. $Q = \{G_1, G_2 ... G_t\}$, where for every node there is an associated set of features, e.g. node $u_i$ has feature vector $\mathcal{F}_i$ and the features are changing between graphs (but graph structure does not). The resulting graphs are big in size with this approach. Therefore I decided to make the graphs smaller, by truncating some of the nodes and edges. And I would like to calculate how much information I lose when I simplify the graphs with respect to the not simplified graphs or original data. I would like to get something like "This graph explains 77% variance in the data" And the truncated graphs "This graph explains 55% variance in the data".

The question is then fallowing:
How to tell how much information I lose when I simplify the graph data structure.

Edit: Also the feature vector can be replaced with weighted edges. I think it can make the problem a bit simpler to solve.


texts: Machine learning - features engineering from date/time data

texts:
What are the common/best practices to handle time data for machine learning application?

For example, if in data set there is a column with timestamp of event, such as "2014-05-05", how you can extract useful features from this column if any?

Thanks in advance!


texts: Why would Pytorch be non-deterministic for batch only?

texts:
I am training a LSTM network on CPU and can achieve deterministic results when not using a dataloader. But when I used Pytorchs dataloader I achieve non-deterministic training error results, despite the actual batches being loaded from the dataloader being deterministic.

I have set pretty much every seed I could think of here

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled   = False

The two pieces of code are:

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    for batch_idx, (X_train, y_train) in enumerate(train_loader):

        # Zero out gradients
        optimiser.zero_grad()

        # Forward pass
        y_pred = model_test(X_train)

        # Loss Function
        loss = loss_fn(y_pred, y_train)

        # Backward pass
        loss.backward()

        # Update parameters
        optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())

and

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    # Zero out gradients
    optimiser.zero_grad()

    # Forward pass
    y_pred = model_test(X_train)

    # Loss Function
    loss = loss_fn(y_pred, y_train)

    # Backward pass
    loss.backward()

    # Update parameters
    optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())

I have seen some posts on Github talking about there being issues with determinism on GPU, but this is just on CPU.


texts: Getting parameters of the best model with crossvalidation in with SparkMLLib

texts:
I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is:
I have a logistic regression model for which I want to find the best regularization parameters (regParam and elasticNetParam). To do that, I use the CrossValidator which works and finds me a model better than all the other one I tried. The issue is that I do not know how to access the actual value of the parameters that were found by the cross validator. Below is the code I use to fit my cross validator:

from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.classification import LogisticRegression

lr_predictor = LogisticRegression(featuresCol='polyFeatures', labelCol='label', maxIter=10)
paramGrid = ParamGridBuilder() \
    .addGrid(lr_predictor.elasticNetParam, [0., 0.5, 1]) \
    .addGrid(lr_predictor.regParam, [0.1, 0.01]) \
    .build()
crossval = CrossValidator(estimator=LogRegPipeline,
                          estimatorParamMaps=paramGrid,
                          evaluator=BinaryClassificationEvaluator(),
                          numFolds=2)
cvModel = crossval.fit(train_set)
bestModel = cvModel.bestModel
# How to get the best parameters fitted by cvModel


texts: Learning to rank: construct absolute ranking using pair-wise ranking approach

texts:
I am learning about the "pairwise approach" for learning to rank. As far as I understood, the training output is a partial ranking function $r$ that:

given given some query $q$ and two document $d_i$ and $d_j$
predicts whether $$r(d_i)>r(d_j)$$ or not

However, for a IR system to work, the ranking should be absolute.

The natural question next is how to construct the absolute rank using only the partial ranks output by $r$?

But partial ranks does not guarantee absolute rank. For example

$$r(x)>r(y), r(y)>r(z), r(z)>r(x)$$ gives a cycle.

I guess I am misunderstanding about how pairwise approach works for IR system. Can anyone correct me?


texts: How to combine two different machine learning models, to get the combined result?

texts:
To further explain my question, I will explain my use-case. Say I have a model which is trained for how good/bad a food is for obesity based on its nutrition facts. And another model for, say hypertension. I wish to combine these models to be able to predict food which are good for a person suffering from both obesity and hypertension.
I do not wish to retrain a new model for both cases as eventually I will add more diseases and do not wish to train for each combination of diseases.
Ideally I would like to be able to combine different models (eg Logistic Regression, SVM) but as a stepping stone, I was looking to combine two models of the same type.
I am not sure if Stacking would be appropriate in this case.


texts: How to treat the undefined values which make sense?

texts:
I'm currently trying to create a few features to improve the performances of a model. One of those features that I would like to create corresponds to the difference in days between a customer's purcharse and his last one. To create this feature is not a problem. However, I don't know which value to set if this is the first purcharse of a customer. Which value should I set and, more generally, how to treat these cases ?

   customer_id date_purchase  diff_last_purchase  first_purchase
0            1    2018.02.12                 NaN               1
1            1    2018.02.18                 6.0               0
2            2    2018.02.25                 NaN               1
3            3    2018.03.15                 NaN               1
4            3    2018.03.18                 3.0               0


texts: What's the best way to approach a regression problem with a lot of features (over 300), many of which are categorical?

texts:
I have $20,000$ training examples of various candidate attributes (highest level of education, country, year of formal training completion, etc).

My output (prediction) will be for job_performance, which is measured between $700$ and $4,200$ (in my training data). Initially, I was thinking about putting together a fully connected neural network, but I attempted a similar problem in a Kaggle competition, but that didn't produce great results. What method(s) would you use to start?


texts: Learning rate and memory of Least Mean Square algorithm

texts:
I am studying neural networks.

  The smaller we make the learning rate, the longer the memory span over which the LMS algorithm remembers past data will be. (Section 3.5(page 103) of Simon Hykin's "Neural Network and Learning Machines":)

I don't understand why this is the case. Could someone explain?


texts: Predict customer action from previous buying history

texts:
I'm trying to predict what service a customer wants when he comes to our office from his previous transactions history. I have 7 years transactions data(3 crore txns) and good amount of customers are frequent ones.Each service is personal to each customers.

sample data

[
    {
        "customerId":"1xxxx",
        "txns":[
            {
                "serviceId":"12ds23",
                "date":"2016-08-03T08:43:33Z"
            },
            {
                "serviceId":"1dsd89",
                "date":"2016-09-03T08:43:33Z"
            },
            {
                "serviceId":"1dbbb89",
                "date":"2016-10-03T09:43:33Z"
            }
        ]

    },{
        "customerId":"2xxxx",
        "txns":[
            {
                "serviceId":"dds2dfsd",
                "date":"2016-08-03T08:43:33Z"
            },
            {
                "serviceId":"dsdsdsdf",
                "date":"2016-09-03T08:43:33Z"
            },
            {
                "serviceId":"sdfbb9",
                "date":"2017-10-03T09:43:33Z"
            },
            ...
            ...
        ]

    },
    ...
    ..
    ..
]

Can someone please advice which Machine learning technique or statistical approach would be best in this case.

I can think of a Decision tree classification/Logistic regression model taking date(month,day,day of week) as features for predicting the service he wants as Class labels


texts: How to present Market Basket Analysis Results?

texts:
I am working on a Retail Company's in-store transactions for 3 months. I have performed the Market Basket Analysis on the same and I'm getting hundreds if not thousands of association rules. I am using the apriori algorithm from mlxtend.frequent_patterns import apriori in Python and I have used different support values in apriori(basket_sets, min_support=0.01, use_colnames=True), all the way from 0.01 to 0.4.

If I use a support value too high, (for some stores there are no rules found), there are very few association rules, and if I use a support value too low, its very difficult to make sense of the association rules generated since there are too many of such rules.

Since I've chosen to go with the low support value, I wanted to understand ways in which I could present the rules which would make business sense to the data owners. If there is any literature (I've tried googling at least 10 different queries but all of them return "How to do Market Basket Analysis and its applications!") on how to present the Association results, that would be really good.

Thanks


texts: How can you include information not present in an image for neural networks?

texts:
I am training a CNN to identify objects in images (one label per image). However, I have additional information about these images that cannot be retrieved by looking at the image itself. In more detail, I'm talking about the physical location of this object. This information proved to be important when classifying these objects.

However, I can't think of a good solution to include this information in a image recognition model, since the CNN is classifying the object based on the pixel values and not on ordered feature data.

One possible solution I was thinking of was to have an additional simple ML model on tabular data (including mainly the location data), such as an SVM, to give give a certain additional weight to the output of the CNN. Would this be a good strategy? I can't seem to find anything in the literature about this.

Thanks in advance!

edit: Someone asked what I meant by 'location'. With the location I meant the physical location of where the image was taken, in context of a large 2d space. I don't want to go too deep into the domain, but it's basically an (x,y) vector on a surface area, and obviously this meta-data cannot be extracted by looking at the pixel values.

edit2: I want to propose an additional way I found that was useful, but was not mentioned in any answer.
Instead of using the neural network to predict the classes, I use the neural network to produce features.

I removed the final layer, which resulted in an output of shape 1024x1. This obviously depends on the design of your network. Then, I can use these features together with the meta-data (in my case location data) in an additional model to make predictions, such an SVM or another NN.


texts: How is hinge loss related to primal form / dual form of SVM

texts:
I'm learning SVM and many classic tutorials talk about the formulation of SVM problem as a convex optimization problem: i.e. We have the objective function with slack variables and subject to constraints. Most tutorials go through the derivation from this primal problem formulation to the classic formulation (using Lagrange multipliers, get the dual form, etc...).
As I followed the steps, they make sense eventually after some time of learning.

But then an important concept for SVM is the hinge loss. If I'm not mistaken, the hinge loss formula is completely separate from all the steps I described above. I can't find where the hinge loss comes into play when going through the tutorials that derive the SVM problem formulation.

Now, I only know SVM as a classic convex optimization / linear programming problem with its objective function and slack variables that is subject to constraints. How is that related to hinge loss??


texts: Neural networks, optimization math intuition

texts:
When I look into the following partial derivative, I see it as being the key element of any optimization algorithm out there. Correct me if I'm wrong, but this gets us the slope of the loss function, so we can go opposite to that slope, therefore minimizing the loss.

$$\frac{\partial \theta}{\partial \mathcal{L}}$$

where: $\theta$ is the weights, and the $\mathcal{L}$ is the loss;

Does that make sense? Is there any other calculation step that is arguably more fundamental to the optimization of neural networks other than this derivative?

This topic is specially important for me right now, because I was thinking of tattoing this derivative, as a cool A.I. tattoo, and I want it to be fundamental and simple.


texts: CNN regression. help to improve current model

texts:
I have time series grey scale images that show movement of fluid with different densities.
I want to predict a pixel value for time t, with (t-3),(t-2),(t-1) 2D images as inputs.
I am figuring out how to use what layers.
This is my current basic configuration. I believe this can be vastly improved.
model = Sequential()

model.add(TimeDistributed(Conv2D(16, (3,3), padding="same", activation='relu'), input_shape=training_input_nparray[0].shape))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))

model.add(TimeDistributed(Conv2D(32, (3,3), padding="same", activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))

model.add(TimeDistributed(Conv2D(64, (3,3), padding="same", activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))

model.add(TimeDistributed(GlobalAveragePooling2D()))
model.add(LSTM(1024, activation='relu', return_sequences=False))

model.add(Dense(1, activation='linear'))
model.compile('adam', loss='mean_squared_error')

Any help would be greatly appreciated. Thanks in advance :)
Training input array and output array shapes:
(1785, 3, 180, 206, 1)
(1785, 1)


texts:
I am trying to understand the purpose of a 3rd split in the form of a validation dataset. I am not necessarily talking about cross-validation here.

In the scenario below, it would appear that the model is overfit to the training dataset.

Train dataset {acc: 97%, loss: 0.07}
Test dataset {acc: 90%, loss: 8.02}

However, in this scenario it appears much more balanced.

Train dataset {acc: 95%, loss: 1.14}
Test dataset {acc: 93%, loss: 1.83}

Do I need validation data if my train and test accuracy/loss is consistent? Is the purpose of setting a validation split of 10% to ensure this kind of balance before evaluating the model on the training set? What does it prove?


texts: Image classification problem using convolutional neural networks

texts:
I am trying to solve this problem by using a convolutional NN to classify an image data set to check the type of disease it is. I have reached task 1b and trying to implement the training loop. However, I am getting an error and can't understand how to implement the training loop.
I am sharing the google drive link.
The code in task 1b where I am getting an error:
import datasets

ds = datasets.LesionDataset('/content/data/img',
                            '/content/data/img/train.csv')
input, label = ds[0]
print(input)
print(label)

error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-dcb680f1f34a> in <module>()
      4 ds = datasets.LesionDataset('/content/data/img',
      5                             '/content/data/img/train.csv')
----> 6 input, label = ds[0]
      7 print(input)
      8 print(label)

/content/drive/MyDrive/DL Assignment/datasets.py in __getitem__(self, idx)
     34   def __getitem__(self, idx):
     35
---> 36         image = self.img_dir[idx]
     37         label = self.labels_fname[idx]
     38

'datasets' is a .py file currently in the same folder in the link shared which has the following code: You can open the link to understand it better.
import collections
import csv
from pathlib import Path

import numpy as np
import torch
import torchvision.transforms as transforms
from PIL import Image
import pandas as pd

# TODO Task 1b - Implement LesionDataset
from torch.utils.data import Dataset
class LesionDataset(Dataset):
#The __init__ function should have the following prototype
  def __init__(self, img_dir, labels_fname):
    # is the directory path with all the image files
    # is the csv file with image ids and their corresponding labels
      self.img_dir = Path(img_dir)

      #image = Image.open('ISIC_0024306.jpg')
      #
      self.labels_fname = pd.read_csv(labels_fname)

      image = self.labels_fname['image']

#list of images store in list
  def __len__(self):

        return len(self.labels_fname)

  def __getitem__(self, idx):

        image = self.img_dir[idx]
        label = self.labels_fname[idx]

        return image, label

# TODO Task 1e - Add augment flag to LesionDataset, so the __init__ function
#                now look like this:
#                   def __init__(self, img_dir, labels_fname, augment=False):

# TODO Task 2b - Implement TextDataset
#               The __init__ function should have the following prototype
#                   def __init__(self, fname, sentence_len)
#                   - fname is the filename of the cvs file that contains each
#                     news headlines text and its corresponding label.
#                   - sentence_len the maximum sentence length you want the
#                     tokenized to return. Any sentence longer than that should
#                     be truncated by the tokenizer. Any shorter sentence should
#                     padded by the tokenizer.
#                We will be using the pretrained 'distilbert-base-uncased' transform,
#                so please use the appropriate tokenizer for it. NOTE: You will need
#                to include the relevant import statement.

I am stuck at this part. Can anyone suggest how to implement it?


texts: Categorical features for relief based feature selection

texts:
I am exploring Feature Selection methods and was wondering if the Relief based feature selection methods can be used for categorical or mixed-type features. From what I understand there is,

relief: to be used for a binary target/response variable
reliefF: to be used for multiclass target/response variable
RreliefF: to be used for continuous target/response variable

But what about the features/explanatory variables? Can they be categorical or mixed type? Hopefully, used through one-hot-encoding?
It seems that relief based measures use a distance measure to compute similarities based on the target/response variable. In which case does it only make sense to use continuous features?


texts: No registered env with id: BanditTenArmedGaussian-v0 for the package gym_bandits of OpenAI

texts:
I've made the following instruction :

git clone https://github.com/JKCooper2/gym-bandits.git
cd gym-bandits
pip install -e .

After installing :

import gym_bandits
import gym

Now we finally :

env = gym.make("BanditTenArmedGaussian-v0")

Even tough the first three steps was successfull, the last one doesn't work. Here is my error :

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
c:\windows\system32\gym\gym\envs\registration.py in spec(self, path)
    120         try:
--> 121             return self.env_specs[id]
    122         except KeyError:

KeyError: 'BanditTenArmedGaussian-v0'

During handling of the above exception, another exception occurred:

UnregisteredEnv                           Traceback (most recent call last)
<ipython-input-13-645d2b1dfa73> in <module>
      4 import math
      5 import random
----> 6 env = gym.make("BanditTenArmedGaussian-v0")

c:\windows\system32\gym\gym\envs\registration.py in make(id, **kwargs)
    143
    144 def make(id, **kwargs):
--> 145     return registry.make(id, **kwargs)
    146
    147 def spec(id):

c:\windows\system32\gym\gym\envs\registration.py in make(self, path, **kwargs)
     87         else:
     88             logger.info('Making new env: %s', path)
---> 89         spec = self.spec(path)
     90         env = spec.make(**kwargs)
     91         # We used to have people override _reset/_step rather than

c:\windows\system32\gym\gym\envs\registration.py in spec(self, path)
    129                 raise error.DeprecatedEnv('Env {} not found (valid versions include {})'.format(id, matching_envs))
    130             else:
--> 131                 raise error.UnregisteredEnv('No registered env with id: {}'.format(id))
    132
    133     def register(self, id, **kwargs):

UnregisteredEnv: No registered env with id: BanditTenArmedGaussian-v0

Does anyone know how to fix it ?

Thanks and regards.


texts: Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)

texts: What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision processes)?