texts: Find highest reward for epsilon-greedy bandit program

texts:
I started to learn reinforcement learning, the first example is handling bandit program using epsilon-greedy method,
In this example, there are three bandit machines used, the output is the mean value for all bandit machines and cumulative average with respect to the epsilon value
The code -
class Bandit:
  def __init__(self, m):
    self.m = m
    self.mean = 0
    self.N = 0

  def pull(self):
    return np.random.randn() + self.m

  def update(self, x):
    self.N += 1
    self.mean = (1 - 1.0/self.N)*self.mean + 1.0/self.N*x

def run_experiment(m1, m2, m3, eps, N):
  bandits = [Bandit(m1), Bandit(m2), Bandit(m3)]

  data = np.empty(N)

  for i in range(N):
    # epsilon greedy
    p = np.random.random()
    if p < eps:
      j = np.random.choice(3)
    else:
      j = np.argmax([b.mean for b in bandits])
    x = bandits[j].pull()
    bandits[j].update(x)

    data[i] = x
  cumulative_average = np.cumsum(data) / (np.arange(N) + 1)

  for b in bandits:
    print(b.mean)

  return cumulative_average

if __name__ == '__main__':
  c_1 = run_experiment(1.0, 2.0, 3.0, 0.1, 100000)
  c_05 = run_experiment(1.0, 2.0, 3.0, 0.05, 100000)
  c_01 = run_experiment(1.0, 2.0, 3.0, 0.01, 100000)

  # log scale plot
  plt.plot(c_1, label='eps = 0.1')
  plt.plot(c_05, label='eps = 0.05')
  plt.plot(c_01, label='eps = 0.01')
  plt.legend()
  plt.xscale('log')
  plt.show()

The output plot(cumulative average for different epsilon values (0.1, 0.05, 0.01) -

I understand form the output graph, that the cumulative average for the machines using epsilon value = 0.01 scored more rewards than the other epsilon value(0.1 and 0.05)
Here we are just comparing the different epsilon values,
But, how to decide which machine is the best which gives more rewards
I am looking forward to learn, thanks


texts: When can embeddings be useful for small input spaces?

texts:
I've noticed that some authors tend to use embeddings for almost everything. E.g. in the AlphaStar paper, or below an example for the OpenAI5 paper (link):

The Hero Embedding here encodes information about which hero out of 17 was picked. With 5 pickable heroes per game, that would yield a one-hot encoded vector of size 17 * 5 = 85. Instead, they chose to encode this information using an embedding. Does that not seem a bit overkill?

My understanding was that embeddings are very useful for large scale inputs, such as word-corpora, as they reduce the dimensionality nicely. What's the benefit of doing it on smaller inputs such as above? Is it just convenience? Or maybe I am misunderstanding what embedding means here?

EDIT: I wanted to add that EVEN if the Hero Embedding contains more unique information, such as health, mana, etc, then it would still be very small, likely under 200 values. So when does it make sense to start using an embedding, as opposed to using the values directly? What's the underlying thought?


texts: Checkers playing Neural Network evolved with Genetic Algorithm becomes too sensitive to input data changes

texts:
I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.
I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.
The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.
A bit more information:

Neural Network Structure:

Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).
I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.

Genetic Algorithm

30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).
The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:

$m' = m * e^{r*0.9} $
$w' = w * m' * r$

Where $m$ is the mutation parameter of that specific weight which starts off at 0.05, $r$ is a random standard Gaussian number and $w$ is the weight.
I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.


texts: How does the below calculation help to create a PNG from a floating point depth map?

texts:
What is the effect of max_val * (depth - depth_min) / (depth_max - depth_min) in the below function?
def write_depth(path, depth, bits=2):
    """Write depth map to pfm and png file.

    Args:
        path (str): filepath without extension
        depth (array): depth
    """
    write_pfm(path + ".pfm", depth.astype(np.float32))

    depth_min = depth.min()
    depth_max = depth.max()

    max_val = (2**(8*bits))-1

    if depth_max - depth_min > np.finfo("float").eps:
        out = max_val * (depth - depth_min) / (depth_max - depth_min)
    else:
        out = np.zeros(depth.shape, dtype=depth.type)

    if bits == 1:
        cv2.imwrite(path + ".png", out.astype("uint8"))
    elif bits == 2:
        cv2.imwrite(path + ".png", out.astype("uint16"))

    return

My question is not about syntax but about the reason or the effect of that calculation?

What effect on the array does subtracting the depth_min have? Other than changing the mean and median.  Why do this step?
What effect on the array does multiplying by max_val have?
What effect on the array does dividing by the range have?
Why do this instead of just converting each floating point to an integer?

I have seen the output of this function, which is a depth map where lighter gray pixels denote close objects and darker pixels denote far away objects.


texts: NoSQL Comparison - Is this part of my Job?

texts: For the more experienced Data Scientists here, i was asked to perform a case study on how Redis / HBase etc performs, compared to each other.How does data science play a role into this? Note that there will be no actual data involved.


texts: How to determine feature importance while using xgboost in pipeline?

texts:
How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline?

AttributeError: 'Pipeline' object has no attribute 'get_fscore'

The answer provided here is similar but I couldn't get the idea.


texts: any reason for this project to use hadoop/spark?

texts:
I am setting up for a couple self study projects to explore machine learning techniques.

1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I will be creating a bunch of calendar and weather features for the data, then trying to forecast using a variety of machine learning techniques.

2nd is some 13 million rows of text data (several paragraphs for each row)  for classification. (currently in solr database)

My compute rig is 6-core, 32g ram, gforce GPU. I plan to install Ubuntu 14.2.

I expect to be using python for file processing, scilearn, pylearn2 and word2vec for general exploration and training. R for getting a taste of the language.

Clearly data set 1 will require joining weather and calendar data to date/time and aggregation across time and location. I know how to stuff it all into a MySQL database and do the aggregations and joins there, but I have been reading about spark and wondering.

......

If I take the time to simulate a cluster using virtual box/hadoop/spark (for my learning experience, not performance), can/should I do the aggregations there and write the results to the distributed data store?

Since deep learning can not be run on spark, does that mean I would need to copy the aggregated data back out to the local file system to use some of those techniques?

For data set 2, I want to run the word2vec algorithm as found in the kaggle tutorial https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors. In that example, that is a deeplearning method, so I should just leave the data in the solr.. right?

In general I am looking for appropriate applications and insight into data flow from app to app to help me get to the part where I start trying various ML techniques.

Thanks for helping me along


texts: Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

texts:
To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated through many layers, the product of these gradients becomes very small.

I know there are other solutions like a rectifier activation function, but my question is why we could not simply use a variation of the often used tanh function.

If the activation function was of the form $\tanh(n x)$ then the maximum possible gradient is $n$. Thus if $n > 1$ we no longer have a case where the product of gradients necessarily goes to 0.

Is there some reason why such an activation function would otherwise fail?


texts: standardize dataset with numerical and dummy features

texts:
I have a dataset with both numerical and categorical features (variables), I converted all the categorical variables in dummies, then I split the train and test data.

Now I am at the step where I want to standardize the features before fitting the model.

Should I apply the standardisation to all the features or only to the numerical ones?
In this case, is it preferable to use the MinMax scaler on a range 0 to 1?


texts: Difference between Information Gain and Mutual Information for feature selection

texts:
What is the difference between information gain and mutual information?

At this point, I understand that information gain is calculated between a random variable and target class for classification while mutual information is calculated between two random variables.

Does mutual information become the same as information when it is calculated between a random variable and target class?


texts: What is the difference between sentiment analysis (e.g. on Facebook) and Twitter sentiment analysis?

texts:
I'm bit confused with the names of sentiment-analysis and twitter-sentiment-analysis.I want to do sentiment analysis for facebook fan page posts.so I decide to use an api but I did not found any api which gives sentiment analysis for facebook Posts.Every api service is made twitter.I found datumbox api which gives results for sentiment-analysis and twitter-sentiment analysis here.

for facebook sentiment analysis, do I need to go with sentiment-analysis api or twitter sentiment-analysis.(I know Name itself suggests sentiment is for twitter tweets).

Is their any api that provides sentiment analysis for facebook posts data ?

or

facebook trained datasets with the classes of POS,Neg,Neu?


texts: GAN: Discriminator converges, generator learns almost nothing

texts:
In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge.

I suspect this is due to the vanishing gradient problem. Theory: as the discriminator loss doesn't start out big to begin with, the generator will never be backpropagated small gradients into, and therefore it doesn't learn to generate better images.

What are some remedies to resolve this and make the generator actually work?


texts: The most used loss function in tensorflow for a binary classification?

texts:
I am working on a binary classification problem using CNN model, the model designed using tensorflow framework, in most GitHub projects that I saw, they use "softmax cross entropy with logits" v1 and v2 as loss function, my questions are:

1- Why this loss method is the most used one?

2- What is the type of this function because of the name I am confused about its type.

3- What is the equation for the function? in tensorflow website, the equation is not available.


texts: weighted cross entropy for imbalanced dataset - multiclass classification

texts:
I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.

I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.

I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.

Is there any standard way of handling this sort of imbalance?


texts: Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis

texts: I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn implementation, or maybe another library?


texts: Why this model does not converge in keras?

texts:
This case has an underlying story but I have essentially boiled it down to the simplest possible re-producible example I could.

Essentially let us think that I have up to 1000 nodes and each node represented by a small (this case is a 3-cell vector) vector and I concatenate and represent these nodes as a padded 3*1000 input vector and need to find out which one is more suitable. So the model is trying to predict 1000 float values, one for each node.

Let's imagine the function to score nodes is this arbitrary code:

def score_vector(v):
  a, b, c = tuple(v)
  if a == 0 or b == 0 or a - c < 2:
    return float(Defs.INVALID_SCORE)
  return float(a * math.sqrt(a - c) / math.log(b + 2, 5))

And essentially my model is supposed to learn this function plus an argmax to find the node that has the highest score. This looks to me like a pretty simple problem compared to the problems I have solved so far (but it is different too).

So my question is why doesn't this model converge? I am thinking it could be due to its differentiability but really kinda lost and started to doubt everything I know about NN (which is not a lot).

Here is the repro code:

import numpy as np
import math

from keras import Sequential, Input
from keras.layers import Flatten, Activation, Dense
from keras.optimizers import Adam

class Defs:
  VECTOR_SIZE=3
  NODE_COUNT=1000
  MAX_REAL_NODE_COUNT=400
  MIN_REAL_NODE_COUNT=20
  INVALID_SCORE=0

def score_vector(v):
  a, b, c = tuple(v)
  if a == 0 or b == 0 or a - c < 2:
    return float(Defs.INVALID_SCORE)
  return float(a * math.sqrt(a - c) / math.log(b + 2, 5))

def build_vector():
  a = np.random.randint(1, 100)
  c = np.random.randint(1, 50) if np.random.choice([False, True, True]) else 0
  b = 0 if c == 0 else np.random.randint(c, c*3)
  return [float(a), float(b), float(c)]

def build_vectorset_score():
  n = np.random.randint(Defs.MIN_REAL_NODE_COUNT, Defs.MAX_REAL_NODE_COUNT)
  vectorset = []
  for i in range(0, n):
    vectorset += build_vector()

  # pad it
  vectorset += [0. for i in range((Defs.NODE_COUNT-n) * Defs.VECTOR_SIZE)]
  scores = [score_vector(vectorset[i*Defs.VECTOR_SIZE:(i+1)*Defs.VECTOR_SIZE]) for i in range(0, Defs.NODE_COUNT)]
  index = np.argmax(scores)
  scores = [1. if index == i else 0. for i in range(0, len(scores))]
  return vectorset, scores

def build_model():
  model = Sequential()
  model.add(Dense(Defs.VECTOR_SIZE * Defs.NODE_COUNT, input_dim=Defs.VECTOR_SIZE * Defs.NODE_COUNT, activation='relu'))
  model.add(Dense(Defs.NODE_COUNT, activation='relu'))
  model.add(Dense(Defs.NODE_COUNT))
  model.add(Activation('softmax'))
  print(model.summary())
  model.compile(loss="categorical_crossentropy",
                optimizer=Adam(lr=0.001), metrics=['categorical_accuracy'])
  return model

if __name__ == '__main__':
  SAMPLE_SIZE = 1 * 1000
  X = []
  Y = []
  for i in range(0, SAMPLE_SIZE):
    x, y = build_vectorset_score()
    X.append(np.array(x))
    Y.append(np.array(y))
  model = build_model()
  model.fit(np.array(X),
                  np.array(Y), batch_size=100, epochs=200, verbose=1)


texts: Help with reusing glove word embedding pretrained model

texts: When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?


texts: Vanishing Gradient vs Exploding Gradient as Activation function?

texts:
ReLU is used as an activation function that serves two purposes:

Breaking linearity in DNN.
Helping in handling Vanishing Gradient problem.

For Exploding Gradient problem, we use Gradient Clipping approach where we set the max threshold limit of Gradient, similarly to ReLU which set the minimum Gradient limit which is 0.
So far I read, ReLU is considered as an activation function. In a similar fashion, can we use Gradient Clipping also as an activation function? If yes, any pros and cons of using it?


texts: How to create a barplot using multiple column

texts:
My data is about boardgames which contain 1 to many catgeories each like this :

categorie1 categorie2 categorie3
  Deduction   Medieval  Word Game
  Deduction   Medieval  NA
  Card Game   Medieval  Zombies
     Horror   NA        NA
     Horror   Medieval  Zombies

I would like to create a barplot showing the most common categories accross games but i can't figure out how with multiple columns instead of one. Is there a dplyr method ?


texts:
For a binary classifier, Is there any difference in Sigmoid activation function vs Softmax with cross entropy ?
In this answer how-to-maximize-recall it is stated that if we use softmax with cross entropy then Recall is improved. Although at most places it is stated that for a binary classifier Sigmoid or Softmax work same.


texts: Machine Learning For Keeping Stable Humidity

texts: I am working on a project building a system to automate much of the tasks in my home garden.  For a few weeks now I have had the system setup to control a humidifier as well as have a sensor to read the current humidity.  I have been reading the humidity every minute and my program has a min and max humidity set, if the humidity goes below the min it kicks on my humidifier and if it goes above the max it kicks on my exhaust fans.  I have also set the system up to log the temp and humidity reading every time it updates to a database.  What I am looking to do is write an algorithm that will take this data and calculate when it needs to turn the humidifier on/off to keep the humidity in the tent as stable as possible.  Right now if you look at the humidity data it is consistently being brought up to just under the max when the humidifier is running (I have it setup to shut off when it reaches 3/4 of the max humidity) and then dropping back down to the min humidity.  The humidifier is a warm vapour humidifier so it takes a few minutes to heat up every time it kicks on.  I want to write something that would look at past data, look at all these factors and calculate when it should turn the humidifier on or off to try and keep a steady base humidity but, I am not really sure where to start or what to google to work toward an answer.  Any info anyone could provide would be greatly appreciated.  I apologise if I am not giving a great description and feel free to ask me for more details.


texts: Ways of calculating the area of colored regions in a map

texts:
Background
I am a PHD student trying to improve my data science. One of my research projects, has me tasked with determining the size of the clusters in a colored image of regions. Here is an example image I am using. The coloring is natural as it represents the orientation of the microscope light. The light hits the surface in different ways resulting in the different colors. But I'm not trying to sum regions of similar colors, but instead just determine the area of each of those polygon regions. I don't have ground truth or labeled images.
Problem
I am trying to calculate the size of each of these colored regions.

Attempts to solve the problem
I don't have labeled images so I have not attempted any supervised approaches. And I don't know if my approach is wise, but my current approach has me solving two problems at once. First is detecting the edges, second is counting the pixels. Maybe there is a better approach?
To detect the edges I am using K means as provided by weka. The parameters I used are below.

weka.clusterers.SimpleKMeans -init 0 -max-candidates 100 -periodic-pruning 10000 -min-density 2.0 -t1 -1.25 -t2 -1.0 -N 70 -A "weka.core.EuclideanDistance -R first-last" -I 500 -num-slots 1 -S 40

Another way I tried solving this problem is by graphing the RGB values of the pixels to determine the co-variance of the pixels in order to determine the boundaries. Then I would add the pixels between the boundaries to determine the size of the region.
Issues with my approach
By using k means, I have to know ahead of time how many colored regions are in the image and I don't know this. By using the co-variance method, it determines the boundaries well in the color array, I don't have a way of knowing the area of the region.
Question
What would a more enlightened way of going about this be, that I could try next? What method would work best here?
More information
Image format: TIF
Compression: None
Size: 2000 x 2000 Pixels (4 MPixels)
Colors: 16.7 Million (24 BitsPerPixel)
Number of Unique Colors: 382,887


texts: Scikit-learn decision tree in production

texts:
I'm working at building a decision tree model that will be used in production.

In documentation here pickle is used to serialize the model however the concerns about this technique make me think there's maybe a better solution to export a model to production.

  pickle (and joblib by extension), has some issues regarding
  maintainability and security. Because of this,

  Never unpickle untrusted data as it could lead to malicious code being executed upon loading.
  While models saved using one version of scikit-learn might load in other versions, this is entirely unsupported and inadvisable. It
  should also be kept in mind that operations performed on such data
  could give different and unexpected results.

So my question is : Using scikit-learn, is there a safe and convenient technique to export the model into production.

PS: Converting the dot data to a python function can be a solution but i'm surprised there's no built-in solution for this.


texts: Is autocorrelation the same as multicollinearity?

texts:
I have a time series dataset that has 63 features with traffic_volume as the target.
print(main_data.columns)
Index(['air_pollution_index', 'clouds_all', 'humidity', 'temperature',
       'wind_direction', 'wind_speed', 'is_holiday_0', 'is_holiday_1',
       'is_holiday_2', 'is_holiday_3', 'is_holiday_4', 'is_holiday_5',
       'is_holiday_6', 'is_holiday_7', 'is_holiday_8', 'is_holiday_9',
       'is_holiday_10', 'is_holiday_11', 'weather_descr_0', 'weather_descr_1',
       'weather_descr_2', 'weather_descr_3', 'weather_descr_4',
       'weather_descr_5', 'weather_descr_6', 'weather_descr_7',
       'weather_descr_8', 'weather_descr_9', 'weather_descr_10',
       'weather_descr_11', 'weather_descr_12', 'weather_descr_13',
       'weather_descr_14', 'weather_descr_15', 'weather_descr_16',
       'weather_descr_17', 'weather_descr_18', 'weather_descr_19',
       'weather_descr_20', 'weather_descr_21', 'weather_descr_22',
       'weather_descr_23', 'weather_type_0', 'weather_type_1',
       'weather_type_2', 'weather_type_3', 'weather_type_4', 'weather_type_5',
       'weather_type_6', 'weather_type_7', 'weather_type_8', 'weather_type_9',
       'weather_type_10', 'dew_point_1', 'dew_point_2', 'dew_point_3',
       'dew_point_4', 'dew_point_5', 'dew_point_6', 'dew_point_7',
       'dew_point_8', 'dew_point_9', 'is_weekend', 'traffic_volume'],
      dtype='object')

All variables are univariate including the target. However, I would like to have differences between autocorrelation and collinearity to be clarified since this has been my first major time series project and some terms are certainly different than statistics. From what I've analysed, there are collinear variables present and they seem to be hindering the accuracy of predictions but visualising the auto and partial correlation doesn't seem to be suggest that. Can someone help me out on pointing out anything wrong with this data and does it need some removal of variables to make it more robust? I've ued XGBoost algorithms with a MAE of 1635 which is not a good score. The first plot is the correlation matrix while the rest are the auto and partial correlation plots.

Please note the partial and auto correlation plots relate to response variable only.


texts:
I have dataset of 50000 values (rows) and 1000 variables (columns). Since this is high dimensional, I am unable to work with just DBSCAN.
So I am trying to use PCA (principle component analysis). Since PCA reduces dimension to few 100's.
I request you to help you to help me with
EITHER
 1. Other high dimensional clustering algorithm
 2. How to interpret results of PCA


texts: What is new population in genetic algorithm?

texts:
Here is my (mis?)understanding of genetic algorithm:

Create n individuals. This is initial population
Calculate fitness of each individual in this population
for i in range(n):
    select two individuals randomly with replacement from population with
    probability of selection of each individual proportional to its
    fitness value
    cross over the two selected individuals to create two offsprings
    introduce mutation
Calculate fitness value for each offspring

Now you have an initial population of n and n offsprings.

What comes now?

Is my new population the n offsprings or the n fittest individuals of the 2n (n population + n offsprings) available?


texts: How to use keras.utils.Sequence with multiple files

texts:
I am looking at the documentation and this guide

https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence

https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly

And I need to write in

def __len__(self):

and

def __getitem__(self, idx):

Which gets the length of the dataset, and the getting a data sample using a particular index. So I am wondering if this is appropriate to use for multiple files, and if so, are there any best practices for this?


texts: Campaign Hypothesis Testing: Is using chi-square test appropriate?

texts:
I’ve just gotten the results back from a marketing campaign that consisted of testing new men’s and women’s apparel creative (test creative) against business as usual men’s and women’s creative (control creative).

I want to test to see if the lift in response rate was significant between...

1.The Men’s Test Vs Control Creatives

2.The Women’s Test Vs Control Creatives

3.All Test vs All Control

This is what my data looks like:

Name              Pannel    Sample_Size  RR_Numerator   RR_Dominator Res_Rate
Men’s In-Target     Test    236,119      1,345          236,119      0.57%
Men’s In-Target     Control 207,715      1,220          207,715      0.59%
Women’s In-Target   Test    364,966      3,915          364,966      1.07%
Women’s In-Target   Control 542,786      4,489          542,786      0.83%
Model Test          Test    601,123      5,538          601,123      0.92%
Business As Usual   Control 750,573      6,185          750,573      0.82%

My questions are:

1.Is the appropriate course of action / decision mechanism to test significance a Chi-Square test ?

2.Should I be adjusting for the unequal Test vs Control sample sizes?


texts: What are the performance measures in the neural networks field?

texts:
I constructed a neural networks in R using neuralnet package.

I want to test that using cross-validation, that is a technique based on using 4/5 of the dataset to train the network and the fifth one as the test set.

I wonder about what measures I should use to measure the neural networks performance in terms of predictability.

Could you suggest what measures are commonly used in the field and explain me why?

Any hint and ideas about that will be appreciated.


texts: Algorithms for Sentiment Analysis on Entity

texts:
I want to make sentiment analysis for an entity which was found, like Google NLP.

Entity should have magnitude and score.

Please share with me the possible research papers.

p/s please not propose to make sentiment for sentence where the entity is located and them assign to entity from such sentence.


texts: Time series packing algorithm for load balancing/smoothing

texts:
I'm looking for pointers to implement an algorithm which takes many data series (load profiles) and packs them into a fixed number of containers in order to minimise peaks and achieve the most even balance.
I have many (say 1000) series to work with and need these optimally packed into 50 boxes, accounting for changes over time.  My data is a fixed number (say 0-100) captured for every minute in a week, representing load over that period.
At this stage I'm interested in ideas on how to approach this task. I understand the basics of packing algorithms however I didn't want to embark on a brute force calculation if there was a more scientific way to do this. Historical data would be available for training if that was deemed the correct approach.


texts: How to split data frame into groups, combine rows

texts:
I have a large data set with 405 columns, many rows, and data from 15 sites.  Each site has 27 columns, each one one quadrats data. Rows are species data.

I would like to split the data into the 15 sites and be able to use functions such as adding or averaging together all 27 columns to get an idea of the species presence at each site. I tried creating a vector of sites and using this to split the data.

Example:

"n<-rep(27,15)                    #repeats 27 15 times (15 sites, 27 quadrats per site in 2018)
names<-c("BB", "BEP","BKP", "BP","BY",'DB','DP','H1P','LTP','NB','NP','NRP','OP','ZB','ZP')
sites18<-as.factor(rep(names[rep(1:15, n)]) )     #site name replicated 27 times

When I use the split function, it loses the species data and makes one long column for each site.

split18<-as.data.frame(split(t(p18),sites18))

I need another solution, perhaps something with an apply function, but I have been unable to find a good solution.


texts:
I want to implement RBF Network from scratch (in python) for classification problems. I get the main idea (compute centroids, RBF activation function, etc) but i don't understand how to build the output layer (mainly for a multiclass problem). I saw examples searching the web but they are all done with a single input (with linear regression).
Any material (mainly books) recommendation is welcome.


texts: artificial neural network for vehicular speed prediction based on volume

texts: I an trying to develop a neural network for vehicular speed prediction. my question is how can i train neural network with many input output combination of data?


texts:
I read about deconvolution/upsampling and the only reason to use it (according to the info in the web) is to enlarge a picture.
Talking about Fully Convolutional Networks isn't it better to just prevent "downsampling" by using zero-padding?
I understand that the results would differ but I haven't found other reasons to choose deconvolution.
The only disadvantage of zero-padding I see is that more pixels is stored in hidden layers (though we don't need a deconvolutional layer in this case). Plus we don't loose the information from the borders.
So is it a good practice to use zero-padding in FCN instead of upsampling?


texts: Ways to deal with longitude/latitude feature

texts:
I am working on a fictional dataset with 25 features. Two of the features are latitude and longitude of a place and others are pH values, elevation, windSpeed etc with varying ranges. I can perform normalization on the other features but how do I approach latitude/longitude features?

Edit: This is a problem to predict agriculture yield. I would think lat/long is very important since locations can be vital in prediction and hence the dilemma.


texts: R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

texts:
I am new to R and usually rely on stata.

I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. The variables have not been standardised. The data is at an Individual level.

Questions I have are the following:

How can I aggregate the surveys?
How can Identify the number of surveys which have provided information on a particular variable i'm interested in like education?

Thank you very much,

raj


texts: Compare two columns and visualize third column based on comparison in Power BI

texts:
I am new to Power Bi, learning myself and I got a dataset with some questions. Most of the questions are about comparing two columns and visualize the effect on the third column.
For example , lets assume we have three columns A,B and C with some values. The question asked is '

When B is higher than A, report on Column C
A histogram of Column C, average of Column C

Is it possible to plot such questions on Power Bi? I know they have filter column but it doesn't take a column , it asks for a number. How can I achieve desired results in Power Bi?


texts: Searching documents at scale: how to maintain cleaned documents?

texts:
I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word "hello" -- not "Hello", "hello", "hello.", and so on).

How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)? I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.


texts: Multivariate, Multi-step LSTM time series forecast

texts:
I'm trying to predict the Pollution using a Multivariate and Multi-step LSTM code, I've been following this tutorial.

I've been following the code until the end, but couldn't understand where the code writer determined the Pollution column to be the code output (predicting the Pollution column instead of an other column?) I'm new to python, and it got me confused.

At first, I thought that this is the part of the code where we define it, but I was wrong:

# invert scaling for forecast
pred_scaler = MinMaxScaler(feature_range=(0, 1)).fit(dataset.values[:,0].reshape(-1, 1))
inv_yhat = pred_scaler.inverse_transform(yhat)
print(inv_yhat.shape)
# invert scaling for actual
inv_y = pred_scaler.inverse_transform(test_y)
print(inv_y.shape)

Any explanations on how he determined the pollution column out of the 7 other columns to be the output?


texts: sklearn.model_selection: GridSearchCV vs. KFold

texts:
Here is the explain of cv parameter in the  sklearn.model_selection.GridSearchCV:

cv : int, cross-validation generator or an iterable, optional
Determines the cross-validation splitting strategy. Possible inputs for cv are:

integer, to specify the number of folds in a (Stratified)KFold

For example, can I replace
CV = 5

to
CV = KFold(n_splits=5, random_state=None, shuffle=False)

or replace in the opposite way. CV is called in the function
GridSearchCV(estimator=XXX, ... , cv = CV)

I am not sure whether any difference?


texts: Clustering tendency using hopkin's statistics

texts:
As per the references
1.   http://www.sthda.com/english/wiki/assessing-clustering-tendency-a-vital-issue-unsupervised-machine-learning
2.   http://www.listendata.com/2016/01/cluster-analysis-with-r.html

The Hopkins statistics value close to '0' is also clusterable. Is this correct? Or the value should always be close to '1' for clusterable?


texts: Created a Machine Learning model on some data. Used this model for predicting test data. Model has learnt only from training data. How this model can be retrained as new data come in for prediction. I don't know whether it is true that retraining a model is necessary so that it can predict new data more accurately.


texts: How to calculate steady/incremental growth?

texts:
I have timeseries data for stocks at minute intervals.
What is the best way to calculate incremental growth, for example if I have a stock's price from 9am to 2pm at minute intervals, how can calcule the incremental growth. I don't want to simply calculate the difference between the price at 2pm and 9am, I want to be able to identify stocks which have risen steadily between the two time intervals


texts: Classification based on a Clustering Result

texts:
Here is the Sample data:

Values    Attribute1    Attribute2    Attribute3    Attribute4
  2.5       1980            A            1.5           C
  1.8       2000            A            0.4           B
  12.4      2017            S            18.5          D
  0.4       1987            R            10            P
......

In my real data, I have more than 60 attributes. What I need to do is:

Category Values into different groups
classify these groups using Attribute1 - Attribute4.

Some of the difficulties are:

Values mean something in my application. For example, Values>10 will be put into one group. So, it might be not ideal to use a clustering algorithm such as density based to divide Values into groups.
When I use the groups based on Values and all the attributes to run a classification, I might need to choose some important attributes.

What I want is to give Values(meaningful data) to a clustering algorithm. So, that I can get best results/Insights from that Clustering.

So, here are my Questions:

How to cluster?
How to choose attributes?


texts: Training Word2Vec with names instead of sentences

texts:
I have scientific database with articles and coauthors.
using this database I am training word2vec model on co-authors.
Use use case here is to disambiguate authors.
I was wondering my approach here can be improved or any suggestions will greatly be appreciated.
Code


texts: NLP - How can I encode single words from one feature (No word frequency)

texts:
I'm constructing a pandas data-frame as an input for some sklearn machine learning models. It is a supervised learning problem that consists in classify 'words' included in the body-content of documents to different labels. I got categorical, numerical and text data. The text data is already tokenised. So in feature 'text' I have single words. Meaning that, there is one word per row. E.g., 'downtown road', '5678-VB', '3552', 'product#3'. The data frame looks like the following:

label                               316511 non-null object
num_feature_1                       316511 non-null float16
num_feature_2                       316511 non-null float16
num_feature_3                       316511 non-null float16
num_feature_4                       316511 non-null float16
text                                316511 non-null object
length                              316511 non-null float16
start_with_capital                  316511 non-null uint8
is_upper                            316511 non-null uint8
ending_with_colon                   316511 non-null uint8

I already encode every-feature but the words (I got current word, previous word and next word as three different features). I have thought about using sklearn.preprocessing.LabelBinarizer for one-hot-encode the words. However, I got 316511 instances/words and 161880 unique words so I created a total of 161920 features. This increased the weight of the pandas data-frame up to 47GB. This is not feasible for my laptop/cloud instance so I had to find another way to handle this large dataset. I found that HashingVectorizer as another way to encode large dataset in a more optimised way  But still, this uses frequency of words and for this problem case it does not make sense to apply frequency.

Anyone got some other idea on how to encode text/strings on this fashion to input sklearn-models?

****EDIT: Just to have a small view about 5 entrances with few features of the dataset:

+--------+-----------------------+--------------+-------------+------------+---------------+
|        | actual_label          |num_feature_1 | text        |   is_upper |num_feature_2  |
|--------+-----------------------+--------------+-------------+------------+---------------|
|  22337 | Sold-to-party Address | 0.989258     | 6B          |          1 | 0.0400085     |
| 199218 | Material Description  | 0.956543     | 32X10       |          1 | 0.0999756     |
|  33579 | Sold-to-party Address | 0.989258     | Hatfield    |          0 | 0.160034      |
|   6506 | Sold-to-party Address | 0.956543     | 26          |          0 | 0.0400085     |
| 199066 | Material Description  | 0.93457      | scanalatura |          0 | 0.219971      |
+--------+-----------------------+--------------+-------------+------------+---------------+


texts: R-square or adjusted R-square for one variable model?

texts: I have model like y=mx. Since the adjusted R2 tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable and I have only one independent variable, do I need to consider my adjusted r-square value? Or, r-square is good for this type of model?


texts:
I'm quite shocked for encountering this error on PCA from sklearn

  ValueError: Complex data not supported

After trying to fit complex-valued data. Is this just unimplemented thing? Should I just go ahead and do it 'manually' with SVD or is their a catch for complex-values?


texts: can we make a word2vec NN of more than 3 layers using tensorflow?

texts:
To the best of my understanding , word2vec crated using gensim is of 3 layers only.
I was wondering can we customize word2vec NN and create word2vec NN of more than 3 layers to experiment with it using tensorflow ?


texts: How to create a model and make predictions with LightFM?

texts:
I've been researching on how to develop a hybrid recommender system for a simple book dataset, the main goal is to use both explicit data (purchases) and latent factors (features) to make the recommendations so I finally ended up choosing LightFM as the best option. I started with Surprise but then I realized that there's no way I can implement a Matrix Factorization model there that uses both item data and user data for the predictions.

So my dataframe looks something like this (simplified):

    number       type   username  product  price        model publishing_dt     author          genres
0        6     access   kerrigan     2345  12.99  printedbook    2020-02-01       john    fantasy,kids
1        4     access   kerrigan      897  14.95  printedbook    2019-03-05      alice         fantasy
2        1  orderline  45michael    86833   2.65        ebook    2020-02-04     joseph      action,war
3        1  orderline   kerrigan    86833   2.65        ebook    2020-02-04     joseph      action,war
1        1  orderline  45michael      897  14.95  printedbook    2019-03-05      alice         fantasy

Where type is a classification of the access made by the user: if it was an order (orderline) or if it was just a view (access), in the case type = access number indicates the amount of times the user accessed the book, and product is the unique id of the book (I think the other fields are self-explanatory).

I'm using purchased or not purchased as the interaction matrix as I don't have any kind of rating:

product        2345     897    86833
username
45michael      0.0      1.0      1.0
kerrigan       0.0      0.0      1.0

The main problem I've found with Surprise is that it would only use the values of that matrix to make the predictions, so as 45michael bought 86833 and 897 and Kerrigan also bought 86833, I'm assuming it will predict that Kerrigan will also buy 897 because of the similarity between both users (which is not a mistaken assumption).

But I would like the system to use the latent factors given by author and genres, that's how I ended up on LightFM.

So I would have an item-features matrix combining authors and genres:

genre        john    alice   joseph   fantasy   kids   action   war
product
2345          1.0      0.0      0.0    1.0      1.0     0.0     0.0
897           0.0      1.0      0.0    1.0      0.0     0.0     0.0
86833         0.0      0.0      1.0    0.0      0.0     1.0     1.0

And an user-features matrix:

genre        john    alice   joseph   fantasy   kids   action   war
username
45michael     0.0      1.0      1.0     1.0      0.0    1.0     1.0
kerrigan      0.0      0.0      1.0     0.0      0.0    1.0     1.0

I was trying to follow this article:
https://towardsdatascience.com/build-a-machine-learning-recommender-72be2a8f96ed
which seemed to cover my fundamental problems but as I have never used LightFM I'm having
a difficult time understanding how to define a model to use with this kind of data and the
creation of the item-features and user-features matrices seems pretty complicated,
can I use a sparse matrix created with scipy as input?

Another doubt that I have (it may seem basic knowledge but I'm considerably new to this
machine-learning/recommendation system world) is that I've been splitting the dataset into
train and test sets when testing Surprise library but I want to be able to make predictions
for all of the users in the dataset, even those used for training, should I make the whole
dataset a training set? But what happens when the dataset grows too big for this case?


texts: Why would GradientBoostClassifier do better than XGBoostClassifier?

texts:
I am working on the Kaggle home loan model and interestingly enough, the GradientBoostClassifier has a considerably better score than XGBClassifier. At the same time it seems to not overfit as much. (note, I am running both algos with default settings).
From what I've been reading XGBClassifier is the same as GradientBoostClassifier, just much faster and more robust. Therefore I am now confused on why would XGB overfit so much more than GradientBoostClassifier, when it should do the contrary? What would be a good reason why this is happening?


texts: How to get correlation between the categories of two categorical variable?

texts:
I have a categorical variable with 2 categories ("Health") ('healthy', 'not_healthy') and another categorical variable ("country") with 5 categories ("english", "eua", "Australia", "spain", "Germany"). I want to check if there is any relation between health and country. I can perform chi-squared and, having a p-value < 0.05, I can reject the null hypothesis and states that under a confidence interval of 95%, country is related to health.
However, what now I want to know is which country is more related to being healthy or not? That is, which country increases the likelihood that a person will be a healthy person if they are from that country?
Is there any statistical test that I can perform or is it determined only by data visualization (healthy graph x country)?


texts: Linear/Logistic Regression for unknown values or how to get a good prior for new coefficients

texts:
Suppose, we model the probability of making holidays by country and town. The input data are people and how many people actually made holiday in that particular town:

  country    city       people    holidays    ratio
  USA        New York   1000      100         0.1
  USA        Seattle    1000      10          0.01
  France     Paris      2000      300         0.15
  France     Metz       500       1           0.002

I translated this into a dataset where each person is a row with a response column telling you he did make holidays there or not. The data was one-hot-encoded and a logistic regression model has been fitted:

df = pd.DataFrame(data, columns=["country", "city", "response"])
y = df["response"]
x = df[["country", "city"]]

one_hot_encoder = OneHotEncoder(handle_unknown="ignore")
one_hot_encoder.fit(x)

x_encoded = one_hot_encoder.transform(x)

model = LogisticRegression(C=1000, max_iter=100, tol=1e-4, multi_class="auto", fit_intercept=False)
model.fit(x_encoded, y)

Obviously, the model fits the data perfectly. But I have problems getting sensible outputy for new cities:

test_data = [["USA", "New York"], ["USA", "Seattle"], ["France", "Paris"],
             ["France", "Metz"], ["USA", "LA"],  ["France", "Toulouse"]]

df_test = pd.DataFrame(test_data, columns=["country", "city"])

print(list(model.predict_proba(one_hot_encoder.transform(df_test))[:, 1]))

Giving me the ouput:

[0.09999993334172928, 0.010002334489653115, 0.14999954347142497,
 0.0020084516806597843, 0.09414556621759342, 0.06613327102757466]

As you can see, the probability of making holidays in LA is 0.094. Since it was 0.1 for NY and 0.01 for seattle, I expected something more in the middle, maybe around 0.05.

So lets have a look at the coefficients:

{'USA': -2.264036462775647,
 'New York': 0.06681114479175591,
 'Seattle': -2.3308476075654636
...
}

This is actually different from what I expected. I expected the mean of the coefficients of New York and Seattle to be zero. I can easily have this by shifting the coefficients of the two cities by -1.13201823139 and the coefficient of USA by 1.13201823139. The model would be the same in terms of loss and predictions.

Using different solvers, different regularization metrics or regularization parameters obviously gives you slightly different results, but none comes close to what I would like to have and what I expected.

It also makes the interpretation of the coefficients more complicated. By shifting the coefficients as above, I create completely different log odds. Its impossible to relate how much more likely a holiday in France is compared to a holiday in the USA.

Anyone has an idea how I can force the model to fit the data as I expected or any other ideas how to get good predictions for unknown data? Maybe someone has read something about this issue before?

Edit:
I figured it must somehow be the regualization that pushes the coefficients into this particular outcome...Any ideas how to prevent it or use a custom one that does the trick?


texts: General way of constructing adjacency matrix in Laplacian SVM semi-supervised technique

texts:
I am trying to implement a Laplacian SVM classifier (trained in primal) using algorithm from this paper.

I would like to know what is the most common way of constructing adjacency matrix and the most common weights to put in the matrix. By most common I mean something that give acceptable results on average.

In the paper the author talks about k-nearest neighbors or graph kernel as examples, I have also seen distance with threshold (what threshold value leads to acceptable average results?).
For the weights he talks about binary weights and heat kernel weights (I don't know others).

Thank you.


texts:
I've read plenty of online posts with clear explanations about the difference between accuracy and F1 score in a binary classification context. However, when I came across the concept of balanced accuracy, explained e.g. in the following image (source) or in this scikit-learn page, I was a bit puzzled as I was trying to compare it with F1 score.

I know that it is probably impossible to establish which is better between balanced accuracy and F1 score as it could be situation-dependent, but I would like to understand some pros/cons of the two performance metrics, as well as some applications in which one could be more suitable and useful than the other (especially in an imbalanced binary classification context).


texts:
I have some doubts on how to represent the relationships between words in texts.
Let’s suppose I have two sentences like these:
Angela Merkel is a German politician who has been Chancellor of Germany since 2005.

What I would expect is a connection between name Angela and Merkel (Angela is the name, Merkel the surname) who is German and politician and Chancellor.
I read about the use of word2vec to determine the semantic structure of a sentence.
My question is therefore if this model can allow me to determine such semantic structure or if another method would be better.


texts: Selecting most important features for multilinear regression

texts:
I have a set of 25 features. I would like to choose the best features for my model. Originally, I was looking at the correlation of features with respect to response, and only taking those which are highly correlated and run a regression model. Then, using that model I would predict the outcome based on test data, and compare to actual (metric RMSE) and this would be how I assess it.
I could then add each feature in order of decreasing correlation with response to the feature set and keep calculating above.
Is there any other way I could select features? Could I e.g. run a random forest and use feature importance report from that to also select most important features? Then run regression?
What is the best way to compare each regression model to the next? There are so many metrics: AIC, BIC, ADJ $R^2$ I am confused as to which one is most simplest way to compare... in fact MSE is not even given in the sm.OLS function (stats models in python) summary:


texts: How to get an intuitive value for regression module evaluation?

texts:
For regression module evaluation, I think only the MAE (Mean absolute error) value is not objective or practical.
 Consider following situations:

A
MAE=1 while target value follows the uniform distribution on the interval [1,100]
B
MAE=1 while target value follows the uniform distribution on the interval [1,10]

Obviously A model is better.

So how to get an intuitive value for regression module evaluation, regardless the data set's target value's scope?


texts:
Background
Using Python, I need to score the existence of a quote, containing around 2-7 words, a longer text. The quote doesn't have to match the text precisely, but similar words should have the same order.
For example, given the following long text:

The most beautiful things in the world cannot be seen or touched, they are felt with the heart

The following quotes should be scored high (say, above 80 / 100):

The beautiful thing in our world
World cannot see
They feel with the heart

Since they are not precise, but they preserve the order.
While, on the other hand, these quotes should be scored lower (say, below 50 / 100):

The beautiful heart cannot be felt or seen
They are the most seen in the world
These words don't even appear on this text

Because (the first 2) appear entirely in the text, but do not preserve the order.
The problem
This task cannot be accomplished by simply checking the existence of each word in the text. I don't know which algorithm fits best for this task.
What I have tried
Most of the functions in fuzzywuzzy (partial_token_sort_ratio, token_sort_ratio and etc) scored the later terms higher.
partial_ratio did score the earlier terms higher, but the quote

These words don't even appear on this text

Got 52 / 100 which is unreasonably high.
My question
How can I use python to score the existence of short quotes in longer texts as mentioned above?


texts: gradient descent for non convex function like $-x^2$

texts:
I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function.
What is really bothering me that for a non-convex function like $y = -x^2$, how gradient descent is actually calculated as here descent would go to minus infinity instead of directly converging to global maxima.
Thus, it contradicts the point of getting stuck in a saddle point for a function like $(x^2 - y^2)$.


texts: Understanding AC_errorRate loss function

texts:
I'm reading an article about Rolling Window Regression: a Simple Approach for Time Series Next value Predictions.

He explains about 5 different loss functions. I managed to understand the first four, but I don't understand the fifth one:

  Almost correct Predictions Error rate (AC_errorRate) — the percentage of predictions that is within %p percentage of the true
  value


texts: What is the largest public wearable accelerometer dataset?

texts:
I'm looking for public datasets of people wearing a device with an accelerometer (and potentially other sensors eg gyroscope or magnetometer).  What are some of the largest available datasets like that?

This has obvious applications to machine learning: a good dataset will help develop good models for activity recognition and event detection from sensor data, same as the effect ImageNet / COCO / YFCC100m datasets have had in the visual field. Sadly, I think the very large datasets in this field are all private.

For my purposes, I don't care what people are doing; a totally random sample of activity is okay (but a broader sample is better than a specific sample).  I also don't care where the device is (wrist, pocket etc) or whether it is a smartphone or some other device (watch, actigraph, IMU etc). Finally, I don't care if the data is labelled / annotated.

I do however want the largest possible size: as many different people as possible, and as many total hours of recording as possible.

What I've found so far...

A few collections of datasets: https://arxiv.org/pdf/1707.03502.pdf and http://mobilize.stanford.edu/data-sources/

Some specific datasets:

UK Biobank has recordings from 100k people x 24 hours each.  I believe these are actigraph (1-minute resolution) not raw accelerometer data (can anyone confirm?).  It is also not open.
NHANES 2003 7k people x 7 days each.  Definitely actigraph, no raw accelerometer data.
LTMM  71 people x 72 hours each
PAMAP2 9 people x ~1 hour each
MHEALTH 10 people x ~15 minutes each
UCI-Smartphone


texts: How does the meta Random Forest Classifier determine the final classification?

texts:
I am trying to understand exactly how the meta random forest classifier determines the final prediction, I understand that there is a voting system and an aggregation from the decision trees is used to find the final prediction, I have read from here: Classification Random Forests in Python:

"Random forest is an ensemble decision tree algorithm because the final prediction, in the case of a regression problem, is an average of the predictions of each individual decision tree; in classification, it's the average of the most frequent prediction"

I read in the RFC source code:
Ensemble/Forest:

"The predicted class of an input sample is a vote by the trees in the forest, weighted by their probability estimates. That is, the predicted class is the one with highest mean probability estimate across the trees"

Does this mean that for example:

We have 3 Classes (A, B and C) with 4 estimators, each tree has a certain amount of final leaf nodes with a class prediction;

Probabilities of Class A, B or C are a result of the number of the class prediction from the leaf nodes,
i.e Tree1: Class A is predicted 3 times out of 10 possible times, the other 7 times a different class was predicted.

Class A: [0.30, 0.25, 0.30, 0.25] | mean = 0.275
Class B: [0.10, 0.40, 0.10, 0.40] | mean = 0.250
Class C: [0.70, 0.65, 0.70, 0.65] | mean = 0.675

So the Meta Classifier would predict Class C as it has the highest mean probabilities. Is this correct? Or am I looking at this completely the wrong way?


texts: Does Tensorflow uses vectorization in its operators

texts:
I'm kinda new to tensorflow and just wanted to know if it already performs vectorization in its operators like multiplying matrices and so on.
And if not if its doable.

As an example let's say we have 2 matrices and we want to multiply them. Instead of for loops to do the operation I'm looking for a vectorized way to do this.

Thanks in advance.


texts: Learning with dirichlet prior - probabilistic graphical models exercise

texts:
I have the following problem:
Suppose we are interested in estimating the distribution over the English letters. We assume an alphabet that consists of 26 letters and the space symbol, and we ignore all other punctuation and the upper/lower case disctinction. We model the distribution over the 27 symbols as a multinomial parametrized by $\theta =(\theta_1,...,\theta_{27})$ where $\sum_i \theta_i = 1$ and all $\theta_i \geq 0$/

Now we go to Stanford's Green Library and repeat the following experiment: randomly pick up a book, open a page, pick a spot on the page, and write down the nearest symbol that is in our alphabet. We use $X[m]$ to denote the letter we obtain in the $mth$ experiment.

In the end we have collected a databse $D=\{x[1],...,x[2000] \}$ consisting of 2000 symbols, among which "a" appears 100times and "p" 87 times. We use a Dirichlet prior over $\theta$, i.e. $P(\theta)=Dirichlet(\alpha_1,...,\alpha_{27})$, where each $\alpha_i=10$.

Suppose we draw two more samples, X[2001] and X[2002]. If we use $\alpha_i=10$ for all $i$, what is the probability of $P(X[2001]="p",X[2002]="a"|D)$?

I thought we could compute: $P(x[2001]=p|D) \times P(x[2002]=a|D)=\frac{10+87}{270+2001} \times \frac{10+100}{270+2002}$ but it's wrong.

Recall formula : $P(x|u,D)=\frac{\alpha_{x,u}+M[x,u]}{\alpha_u+M[u]}$


texts: Given that two images are similar, how do I find and classify the difference between them?

texts:
We have used a derivative of the P-hash algorithm to determine similar images when the user selects an image.

This allows the customer to check out very similar looking photos together. It has also helped us cluster pictures.

When the user selects an image, we would like to show him/her how each of the similar images is different from the selected image, by highlighting the difference, and by classifying the difference.

(eg: this is the hard cover version of the book; The font is different on this cover; original dress in black: same dress in blue, green and yellow )

I'm unsure what kind of annotated training data would be useful for Tensorflow. Could someone please point us in the right direction?

My problem is that we have successfully been able to classify single pictures and even do a bit of object detection by drawing bounded boxes, but we are unable to do pairwise difference identification.


texts: How to extract the positions of employee from raw text

texts:
I have raw text like "Mr John Fullerton is Chief Executive Officer
and Managing Director of Australian Rail Track
Corporation Ltd, and was appointed to the
position in February 2011."

I easily identified name of PERSON(John Fullerton) by using Stanford NER, now i want to extract the positions(Chief Executive Officer). Stanford NER treat Chief Executive Officer as 'O', so which method should i use here?


texts: Number of features of the model must match the input. Model n_features is `N` and input n_features is `X`.

texts:
I am new to data science and trying get some results. I'm applying Decision Tree Classifier. When my train and test datasets' size are not equal I get an error `Number of features of the model must match the input. Model n_features is N (no. of entries in training datasets) and input n_features is X (no. of entries in test datasets).

If I have 100 entries in my dataset and parameter for split is test_size=0.30 as:

import pandas as pd
from pandas import Series, DataFrame
import numpy as np
from sklearn import tree
from sklearn.model_selection import train_test_split

data=pd.read_csv("ndata.csv")

X_train, X_test, y_train, y_test = train_test_split(data.dis, data.gen, test_size=0.30, random_state=42)

c = tree.DecisionTreeClassifier()

y_test_size = y_test.size
y_train_size = y_train.size
X_train = [X_train]
y_train = [y_train]
X_test = [X_test]
y_test = [y_test]

c.fit(X_train, y_train)

accu_train = np.sum(c.predict(X_train) == y_train)/y_train_size
accu_test = np.sum(c.predict(X_test) == y_test)/y_test_size

print("Accuracy on Train: ", accu_train)
print("Accuracy on Test: ", accu_test)

And the error occurs as follows:

ValueError                                Traceback (most recent call last)
<ipython-input-33-f6cc77390526> in <module>()
     24
     25 accu_train = np.sum(c.predict(X_train) == y_train)/y_train_size
---> 26 accu_test = np.sum(c.predict(X_test) == y_test)/y_test_size
     27
     28 print("Accuracy on Train: ", accu_train)

~/anaconda3/lib/python3.6/site-packages/sklearn/tree/tree.py in predict(self, X, check_input)
    410         """
    411         check_is_fitted(self, 'tree_')
--> 412         X = self._validate_X_predict(X, check_input)
    413         proba = self.tree_.predict(X)
    414         n_samples = X.shape[0]

~/anaconda3/lib/python3.6/site-packages/sklearn/tree/tree.py in _validate_X_predict(self, X, check_input)
    382                              "match the input. Model n_features is %s and "
    383                              "input n_features is %s "
--> 384                              % (self.n_features_, n_features))
    385
    386         return X

ValueError: Number of features of the model must match the input. Model n_features is 70 and input n_features is 30

Link to data file: https://gist.github.com/mutafaf/7715ad67bc3cf4e08985afefcc0ce08a#file-ndata-csv

Why do I'm getting this error. Is it necessary to have dataset size of both train and test equal?


texts:
Is there any implementation of the synthetic datasets from mlbench in python?
I am looking for these implementations in particular.
I've found sklearn.datasets for synthetic examples, but they do not contain the sets from mlbench.


texts: Neuroevolution library/framework with GPU CUDA support

texts: I'm looking for working library/framework allowing you to use neuroevolution algorithms like NEAT with GPU support (CUDA). Are there any working libraries? I know about AccNEAT library but I couldn't get it to work with CUDA according to README, probably it's not possible. Any others?


texts: Evaluating the result of topic modeling in a way that time matters

texts:
I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh disease).
before anything, I have divided my data into different 6-month data(from a starting point back every 6 months) and then run the topic modeling approach on every 6 months. I was going to see the difference between the derived topics of each 6 months.

For example, for the first six months there are 20 topics, then for the second six month there 20 topics and... till the tenth (5 years).
I was hopeful to see a different topic because of the use case I have in every six months or at least each 1 year. but sadly most of the words have been repeated in every 6 months. however, the number of the words has changed.

For example in the first six months word "sleeping" has been repeated 10 times in different topics but in the second 6 months, it has been repeated 4 times.

What I am going to say is that, if we look at this as a thing that times matters, I can not see any pattern visibly in my data unless I rely on the number of words changing in every six months.

Do you think analyzing my output and plotting the different words number in different 6 months makes sense at all? or its something unreliable.

Also, do you mind to let me know what other approach is there that I can apply to get insight out of the output of my topic modeling(please consider that the changing in each six months matters)?


texts: Implementing k-means algorithm for cluster analysis on multivariate and multidimensional data

texts:
I have a set of 1000 models. Each model is a (72 x 4) matrix, where 72 are the values associated to each of the 4 variables. The goal is to perform cluster analysis on these models, i.e. to group them based on a similar distribution of all the 4 variables.
It turns out that this is a multidimensional (72) and multivariate (4) problem.
I am trying to implement it in Python by using the k-means algorithm implemented in tensorflow module. First, I have transformed the input dataset to a ndarray (1000 x 288), where the rows are the models and the columns the 288 = 72 x 4 values (this should be fine, since I need to perform cluster analysis based on all the variables). In a sense, I have addressed the multivariate issue by converting the problem to a "univariate" problem.
As a second step, I have standardized the array, so to map the variable values to a common scale.
models = np.array(models)
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

models_scaled = scaler.fit_transform(models.transpose())
models_scaled = models_scaled.transpose()

import tensorflow as tf

#def input_fn():
    #return tf.constant(models_par, dtype=tf.float32)

def input_fn():
  return tf.compat.v1.train.limit_epochs(
      tf.convert_to_tensor(models_scaled, dtype=tf.float32), num_epochs=1)

num_clusters = 5
kmeans = tf.compat.v1.estimator.experimental.KMeans(num_clusters=num_clusters, relative_tolerance=0.001)

# train
num_iterations = 20
previous_centers = None
for _ in range(num_iterations):
  kmeans.train(input_fn)
  cluster_centers = kmeans.cluster_centers()
  if previous_centers is not None:
    print('delta: ', (cluster_centers - previous_centers))
  previous_centers = cluster_centers
  print('score: ' , (kmeans.score(input_fn)))
print('cluster centers: ' , (cluster_centers))

# map the input points to their clusters
cluster_indices = list(kmeans.predict_cluster_index(input_fn))
for i, point in enumerate(models_scaled):
  cluster_index = cluster_indices[i]
  center = cluster_centers[cluster_index]
  print('point: ' , (point) , 'is in cluster ' , (cluster_index) , 'centered at ' , (center))

Assuming this is all correct up to this point, I have troubles in visualizing the output returned by tensorflow. I am trying to get a parallel coordinate plot:
X_clustered = pd.DataFrame()
X_clustered['feature_vector'] = models_scaled.tolist()
X_clustered['cluster'] = cluster_indices

The code above is obviously wrong. Although I have solved (?) the multivariate aspect of the problem, the issue is it's still multidimensional. You may argue I could reduce dimensionality by implementing PCA, for instance, but for the parallel coordinate plot I think the original dataset is required.


texts: Is the percepetron algorithm's convergence dependent on the linearity of the data?

texts:
Does the fact that I have linearly separable data or not impact the convergence of the perceptron algorithm?
Is it always gonna converge if the data is linearly separable and not if it is not ? Is there a general rule ?


texts: Creating an LSTM NN for Fault Classification with Keras

texts:
Goal: Classify time series data from a wind turbine as being anomalous or
non-anomalus in real time and from there predict what the anomaly is in a later, more refined, model.

I have a CSV file with 6 columns corresponding to 6 sensors at uniform measurement. I know I need to separate these into separate vectors to vectors/tensors to run them through a 6 input NN.

My question is how do I create an LSTM that classifies data instead of predicting a value? Using a sigmoid?


texts:
I have the following dataframe containing training data that I have been using to perform a regression task using CNN + FC :

     fileName  var_t+15m  var_t+30m  var_t+45m  var_t+60m  var_t+90m  var_t+120m  var_t+180m  var_t+240
id
2016-10-15 15:00:00  201610151500.jpg     211.00     197.80     170.80      66.90    34.2000   10.120000    0.000867   0.001267
2016-10-15 15:15:00  201610151515.jpg     197.80     170.80      66.90      71.75    20.1600    2.120000    0.001534   0.000534
2016-10-15 15:30:00  201610151530.jpg     170.80      66.90      71.75      34.20    10.1200    0.206200    0.001000   0.001067
2016-10-15 15:45:00  201610151545.jpg      66.90      71.75      34.20      20.16     2.1200    0.012270    0.000400   0.000733
2016-10-15 16:00:00  201610151600.jpg      71.75      34.20      20.16      10.12     0.2062    0.000867    0.001267   0.000934

The task consists in predicting a certain variable at t+X where X goes from 15 minutes up to 240 minutes. So this is a regression task where my training input consists in timestamped picture.

In order to work with these data, I was until now using the .flow_from_dataframe method from Keras in order to perform data augmentation/pre-processing easily and to avoid loading the entire training set consisting of pictures inside the memory.

Up until now I did not leverage the time information and to do so I would like to try the convLSTM model available in Keras. Howevever I am very unfamilar with working with time series.

Has someone used the Keras convLSTM layer combined with the .flow_from_dataframe function ? I am unsure how to structure my data for this setup (convLSTM + .flow_from_dataframe) and I could not find an example on the internet.


texts: A question on data distribution probability

texts:
Let $D={D_k}_1^N$ denotes a set of i.i.d observations, and x denotes random variables, p() denotes probability, whether $p(D|x)$ equals to $p(D)$? Thank you for any help.

EDIT: To be more clear, I talk this problem in Bayesian inference context where we seek the posterior distribution $p(x|D)~p(x)p(D|x)$, so D is dependent on x.


texts: Do Clustering algorithms need feature scaling in the pre-processing stage?

texts: Is feature scaling useful for clustering algorithms? What type of features, I mean numeric, categorical etc., are most efficient for clustering?


texts: Preprocessing to-be-predicted data in ML with R - "learn" and "apply" features

texts:
I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem.

I apply the "usual" preparation for modeling (dummy variables, normalization, PCA etc., of course in the necessary cases) to the training data. So far, so good. But when I get the to-be-classified new data to make prediction the model constructed above, it's evident that I must apply these preparatory steps to this new data set as well.

And the problem arises here, because if I simply apply the preparatory steps for my new data in turn again, these doesn't take into consideration the characteristics of the training data. So, if I convert the new data factors into dummies, then it takes only the existing factor levels in the new data into account; if I min-max normalize the new data, it will be normalized according its own min-max values, disregarding the values in the training data; and if I use PCA, then the resulting components from the new data will be totally independent of the training data.

So essentially my point is that applying the same conversions separately to the train set and the new data set(s) (which could be only one observation as well), then the two resulting transformed sets will have nothing in common, so the prediction will be baseless.

I found some traces that in some cases there is some "learning" step in the training phase in these transformations as well and apply this "knowledge" to new data (caret and Sklearn, for instance, with "predict" could transform to the new data with characteristics learned from the training data), but generally speaking this inconsistency remains unmentioned otherwise.

What is the correct practice here?


texts: Generator loss not decreasing- text to image synthesis

texts:
I am implementing Scott Reed's paper on Generative Adversarial Text to Image Synthesis.(https://arxiv.org/pdf/1605.05396.pdf)

The dataset I am using is a simple one, consisting of images of circles, equilateral triangles and rectangles. The caption of for example a circle is "This is a circle of radius 10".

Instead of the GAN used in the paper, I am using a BiGAN(this shouldnt make that much of a difference in the losses) and training it for 200 epochs. The fake images generated by the generator look quite real in the end. However, the generator loss doesn't seem to decrease, while the discriminator loss decrease fairly rapidly. I can attach more details if needed. Here are the plots of both the generator and the discriminator loss: https://imgur.com/a/2NGGZ

Any ideas as to why this is happening?


texts:
I have been using the following model for image classification. I wonder why there is 512 Dense layer before the Dense3. Should I just use only Dense3? instead of Dense512-Dense3?
model = tf.keras.models.Sequential([
    # Note the input shape is the desired size of the image 150x150 with 3 bytes color
    # This is the first convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(150, 150, 3)),
    tf.keras.layers.MaxPooling2D(2, 2),
    # The second convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The third convolution
    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The fourth convolution
    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # Flatten the results to feed into a DNN
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dropout(0.5),
    # 512 neuron hidden layer
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax')
])


texts: what is the difference between multilabel and multilabel-multiclass classification?

texts:
I am trying to classify news articles into their required category. However I am confused by the above(multilabel and multilabel-multiclass) terminologies. My dataset consists of 2 csv files. The first file contains content of the news article. The second file in that same corresponding row number consists of labels associated with it. Here is the sample entries in second file i.e. labels file:

So for row number 1 the first csv file contains news article which is labelled as sport and football. Similarly for row number 2, the article in first csv has labels sport and media. My question here is, is this dataset multi-label or multilabel-multiclass? what is the difference between them?


texts: How to find out which features were used frequently by RandomForestClassiferer in scikit-learn?

texts: I have several random forest models that work well. Now, I would like to do feature selection based on this models. How do I find out which features were used frequently by a model e.g. RandomForestClassifier in scikit-learn?


texts: Universal cost function for process control?

texts:
I have a problem of process control which looks like this:

The outputs of the process are a number of outcome variables 0-100
where 0 is worst and 100 is best. There is no specific rule (already
known, or related to the nature of the problem) for how to combine
them into a single cost function.  There is some intuition about what results are good, however.
All the variables are equally important, and scaled the same way.
The cost function should be symmetric with respect to permutation of
the values.
If all the variables are 100 except one is 0, that is a bad
outcome.
If all the variables are 90, that is a very good outcome.

The last two statements imply that the right way to combine variables into a cost function is something similar to min(v1, v2, ...).  But that has a problem, in that the direction of optimization would always be determined by the lowest variable alone.

Intuitively, it is desirable to trade off a slight worsening in one variable for a large improvement in another, even if the variable that is getting worse is already pretty bad, for example: 30 and 50 is less preferred than 28 and 75 (or something like that).  This implies some relative elasticity of variables which in this case is close to 100:1 for variables near the opposite ends of the range, and 1:1 for variables which are equal valued.

How can a good cost function be constructed for this kind of problem in a general-purpose way?  (with some adjustable parameters for elasticity, etc)


texts: Why does applying PCA on targets causes underfitting?

texts:
The goal:

I am new to machine learning and experimenting with neural networks.
I would like to build a network that takes as an input a series of 5 images and predicts the next image. My data set is completely artificial, just for my experimentation. As an illustration, here are a couple of examples of input and expected output:

The images of the data points and of the targets are from the same source: the target image of a data point appears in other data points, and vice-versa.

What I have done:

For now I have built a perceptron with one hidden layer and the output layer gives the pixels of the prediction. The two layers are dense and made of sigmoid neurons and I used the mean squared error as the objective. As the images are fairly simple and do not vary much, this does the job well: with 200-300 examples and 50 hidden units, I get a good error value (0.06) and good predictions on test data. The network is trained with gradient descent (with learning rate scaling). Here are the kinds of learning curves I get, and the error evolution with number of epochs:

What I am trying to do:

This is all good, but now I would like to reduce the dimensionality of the data set so that it would scale to bigger images and more examples. So I applied PCA. However I did not apply it on the list of data points but on the list of images, for two reasons:

On the data set as a whole, the convariance matrix would be 24000x24000, which does not fit in the memory of my laptop;
By doing on the images, I can also compress the targets, since they are made of the same images.

As the images look all similar, I managed to reduce their size from 4800 (40x40x3) to 36 while loosing only 1e-6 of the variance.

What does not work:

When I feed my reduced data set and its reduced targets to the network, the gradient descent converges very fast to a high error (around 50 !). You can see the equivalent plots as the ones above:

I had not imagined that a learning curve could start at a high value, and then go down and back up... And what are the usual causes of gradient descent stopping so fast? Could it be linked to parameter initialization (I use GlorotUniform, the default of the lasagne library).

Then I noticed that if I feed the reduced data but the original (uncompressed) targets, I get back the initial performance. So it seems that applying PCA on the target images was not a good idea. Why is that? After all, I just multiplied the inputs and the targets by the same matrix, so the training input and target are still linked in a manner that the neural network should be able to figure out, no? What am I missing?

Even if I introduce and extra layer of 4800 units so that there is the same total number of sigmoid neurons, I get the same results. To sum up, I have tried:

24000 pixels => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
180 "pixels" => 50 sigmoids => 36 sigmoids (= 36 "pixels")
180 "pixels" => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 sigmoids (= 36 "pixels")
180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 linear (= 36 "pixels")

(1) and (3) work fine; but not (2), (4) and (5), and I don't understand why. In particular, since (3) works, (5) should be able to find the same parameters as (3) and the eigen vectors in the last linear layer. Is that not possible for a neural network?


texts:
I try to cluster a dataframe of 227 rows in 5 clusters using kmeans algorithm. Each time I run my code I got different labels and different clusters which make my analysis afterwards a bit tricky.

Someone told me to use the parameter: randomstate to have a reproductility in my results. I did.  I have the same clusters but still not the same label. Is it normal? Is there a way to get the same labels ?

below my code:

Test sur 5 clusters

# Data
X = df.iloc[:,1:]
myseed = 10

# Modèle kmeans à 5 clusters
km = KMeans(n_clusters=5, random_state=myseed, n_init=30)

# Fitting du modèle aux points
km = km.fit(X)
y_km = km.predict(X)


texts: Validation Curve Interpretations for Decision Tree

texts:
I'm working on a machine learning class, and we're using supervised learning right now, starting with decision trees. I'm using the UCI Credit Card dataset (whether or not certain people will default in their payments due to past history).

Using a decision tree classifier for this attempt. Running a validation curve using scikit-learn, I'm getting a plot I'm not quite sure how to interpret. As you can see from the axes, the parameter is min_samples_leaf, and I'm varying it from 1 to 30 (by 2).

Based on this plot and some Googling, I believe the correct way to interpret this is that this dataset has high bias with no variance and nothing is really being learned. Or, in other words, decision trees are not a good algorithm for this dataset, since there doesn't really seem to be a trade-off.

For max depth, I'm getting a validation curve that looks like this:

Based on what I see here, there is quite a bit of bias at the smaller set, and more variance as the depth increases. Given that GridSearchCV returns an ideal max_depth of 5 and min_samples_leaf of 19. (edit: corrected numbers). Those numbers seem to indicate a very high bias, and there really is nothing to be learned here using decision trees.

Overall, based on the min_samples_leaf, I would hesitate to recommend a decision tree for this data set. However, the learning curve and the max_depth validation curve both seem to show there might be some value.

Puzzling to me is that the accuracy score (using metrics.accuracy_score and the ideal parameters from GridSearchCV) is 82%, which doesn't seem that bad. How do I reconcile these crappy validation curves with the accuracy score?


texts: AttributeError: 'numpy.ndarray' object has no attribute 'img_to_array'

texts:
from keras.models import load_model
from keras.preprocessing import image
import matplotlib.pyplot as plt
import numpy as np
import math
import cv2
import numpy as np
import matplotlib.pyplot as plt
from keras.models import model_from_json
import json

img_width, img_height = 150, 150
with open('model_architecture.json', 'r') as f:
    model = model_from_json(f.read())

model.load_weights('second_try.h5')
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
image = cv2.imread('/home/hduser/Desktop/IITDPalmprint/training_data/class1/002_5.bmp', 0).astype(np.float32) / 255
plt.title('image')

kernel = cv2.getGaborKernel((121, 121), 4.0, 0, 10.0, 0.5, 0, cv2.CV_32F)
kernel /= math.sqrt((kernel * kernel).sum())
img1 = cv2.filter2D(image, -1, kernel)
x = image.img_to_array(img1)


texts: What is my training score the mean_train_score or mean_test_score?

texts:
I am using sklearn to train some models (random forest, decision tree). For the training I am using RandomsearchCV with Stratified k-fold as cross-validation. Then I make a predictions on the test set and calculate the test score.

However, I would like to compare the test score with the training score. I assumed I could use the mean_train_score of the cv_results_ report from the RandomseachCV as training score for the model, because I thought it would show the validation against the hould-out-fold from the k-folds.
However, I am not sure about this because there is also a mean_test_score.

I was looking for an explanation of the mean_train_score and mean_test_score. I know these scores exits also for the single folds. But how are these scores calculated? And is one of them my training score, which shows how my model during the training performed?

I found an approach of explanation, but it's too superficial for me: GridSearch mean_test_score vs mean_train_score


texts: What is the reason behind Keras choice of default (recurrent) activation functions in LSTM networks

texts:
Activation function between LSTM layers
In the above link, the answer to the question whether activation function are required for LSTM layers was answered as follows: as an LSTM unit already consists of multiple non-linear activation functions, it is not necessary to use a (recurrent) activation function.
My question:
Is there a specific reason why Keras by default uses a "tanh" activation and "sigmoid" recurrent_activation if those activations are not necessary? I mean, for a Dense layer the default activation is "none". Keras could just have used none as default for LSTM units as well, right? Could it be that Keras uses these activations for a reason? Also, a lot of tutorials or blogs use ReLu (without clarifying why), and I have not come across a one specifying "none" as (recurrent) activation. Why is ReLu used so much (while the outputs from the LSTM unit are already activated)?


texts: LSTM Multi-class classification for large number of classes

texts:
I want to build a model that classifies 473 classes -product categories-, but I'm facing a problem with loss not decreasing.

Data
I have almost 3,000 data points for each class -473 classes- (data size is almost 1.5 million)
The data is a sequence of 5 words [iPhone, Pro, Max, 0, 0]
and of course they're numbered [345, 344, 123, 0, 0]
Examples:
Input: [iPhone, Pro, Max, 0, 0]
Output: iPhone

Input: [Go, Pro, Camera, New, 0]
Output: GoPro

Input: [LG, TV, 50, Inches, Used]
Output: LG_TV

Input: [Apple, Watch, 42, mm, 0]
Output: Apple_Watch

Loss
Epoch: 1, Loss: 5.607430, Val Loss: 5.538741
Epoch: 2, Loss: 5.493465, Val Loss: 5.516405
Epoch: 3, Loss: 5.487641, Val Loss: 5.513667
Epoch: 4, Loss: 5.474956, Val Loss: 5.508683
Epoch: 5, Loss: 5.472722, Val Loss: 5.508304
Epoch: 6, Loss: 5.472691, Val Loss: 5.510557
Epoch: 7, Loss: 5.472782, Val Loss: 5.508627
Epoch: 8, Loss: 5.472320, Val Loss: 5.533378
Epoch: 9, Loss: 5.472340, Val Loss: 5.520573

I've tried to train it for 50 epochs, but the loss still not decreasing.

Model
I'm using PyTorch
LSTMClassifier(
  (embedding): Embedding(15278, 200)
  (lstm): LSTM(200, 256, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (dense): Linear(in_features=256, out_features=473, bias=True)
)

The loss function is: CrossEntropyLoss

Hyperparameters
Batch size: 512
Embedding Dim: 200
Vocabualry size: 15,278
LSTM Layers: 2
Hidden Dims: 256
Optimizer: Adam
Learning rate: 0.002

Can you please direct me to the problem? Is the model weak? Or is it the data having a problem?


texts: Predicting next action to take to reach a final state

texts:
Does anyone know of an algorithm that could be used to determine the next action to take to reach a desired state when trained on time-series data?

For example, a robot starts at a certain state, then takes an action to get to another state. This occurs continuously for many iterations (imagine the robot is randomly exploring a room). If the robot is at a specific starting state, and I desire the robot to end up in a different state, is there an algorithm that could recommend the best next action (or set of next actions) to take to reach that final desired state?

One approach I've tried is to use a neural network with the current state and the next state being the input and the action to get from the current state to the next state being the output. The network would know for a single state how to get to a next desired state that is one action away. The issue is, what if the desired state is many actions away?


texts: Visualising feature selection results for multiple classifiers and feature subset sizes

texts:
I am using information gain feature selection technique to get different features subset sizes for my dataset, like so:
fs1 = SelectKBest(score_func=mutual_info_classif, k=10)
fs1.fit(X_train, y_train)
X_train_fs1 = fs1.transform(X_train)
X_test_fs1 = fs1.transform(X_test)

fs2 = SelectKBest(score_func=mutual_info_classif, k=20)
fs2.fit(X_train, y_train)
X_train_fs2 = fs2.transform(X_train)
X_test_fs2 = fs2.transform(X_test)

fs3 = SelectKBest(score_func=mutual_info_classif, k=30)
fs3.fit(X_train, y_train)
X_train_fs3 = fs3.transform(X_train)
X_test_fs3 = fs3.transform(X_test)

I am then testing the performance of 4 different algorithms (Logistic Regression, SVM, AdaBoost and Decision Trees) using different subset sizes of feature selected features (subset 1 has k=10, so 10 features, subset 2 has 20 features, etc.). To evaluate the model's performance, I am calculating the Precision, Recall and AUC, like so:
def compareAlgorithms(X_train, y_train, score):
    # Compare Algorithms
    seed = 7

    # prepare models
    models = []
    models.append(('LR', LogisticRegression()))
    models.append(('SVM', SVC()))
    models.append(('Linear SVC', LinearSVC()))
    models.append(('ADABOOST', AdaBoostClassifier()))
    models.append(('DT', DecisionTreeClassifier()))

    # evaluate each model in turn
    results = []
    names = []
    scoring = score

    print(score, ":")

    for name, model in models:
        skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)
        #kfold = model_selection.KFold(n_splits=5, random_state=seed)
        cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=skf, scoring=scoring)
        results.append(cv_results)
        names.append(name)
        msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
        print(msg)

    return results, names

As I now have lots of results, I am trying to plot the results to better visualise which algorithm is performing better with which subset. I want to create plots similar to the ones I found in this article:

I have tried using matplotlib to do this, but have found it quite difficult seeing as I am trying to plot different classifiers on different features subset. I can (sort of) plot a line plot of the algorithms performance for one data subset using this function:
def plot(results,names, score):
    import matplotlib.pyplot as plt
    # plot for algorithm comparison
    fig = plt.figure()
    fig.suptitle(score)
    ax = fig.add_subplot(111)
    plt.plot(results)
    ax.set_xticklabels(names)
    plt.show()

Which results in this plot:

The problem with the above plot (beside the overlapping model names below which I am working on fixing) is that it is for one feature subset.
Can anyone help me do a plot like the one in the paper I have attached and maybe direct me to useful resources for someone just starting to learn about data visualisation?
Many thanks.


texts: How is the LSTM RNN forget gate calculated?

texts:
I understand that the LSTM recurrent neural network has a forget gate,  input gate, and output gate.

However, I do not understand how the equation below calculates 'forget' information (from Chris Olah's LSTM post)

How does this forget gate decide which information is discarded from the previous cell state? In other words, what is forgotten or not.

It appears to me that this equation just takes $x_t$ from the current state and $h_{t-1}$ from the previous state.

Can someone explain how this equation works as a forget gate to retain or discard cell state?


texts: How do I convert an L2 norm to a probability?

texts:
I am using the dot product as a way to measure the similarly of two facial-model vectors extracted by a ML algorithm (OpenFace in fact).

I would like to convert the L2 norm to a probability U[0,1] in order to compare with other solutions from other providers that map directly to probabilities but in a blackbox fashion.

By probability I mean the probability that the two vectors represent the same person. So an identical image would give a probability of 1; images of the same person would give a high probability; and orthogonal images (different people, or some suitable toy case) would in the limit have such a probability of zero.

I'd like a uniform distribution for comparison with other providers, but this boils down to knowing what the underlying probability distribution of the feature vectors is.

How do I do this?

Rayleigh distribution CDF
Cosine similarity
(1) is the same as (2)
Draw from a simulated distribution
Arbitrarily scale to U[0,1]
Train into the comparison probability space directly (skipping the feature-vector step)
None of the above
Something else?

Some useful links are:

Cosine similarity versus dot product as distance metrics

http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii

http://houseofjeff.com/2015/05/10/using-probability-distribution-and-cosine-similarity-to-automatically-detect-data-contents - this one is almost there?


texts: Seeking advice: How would you define "top performer" with a set of data?

texts:
still consider myself a noobie in ds here, so I may be asking a very easy and stupid question, but would really appreciate some guidance.

The hypothetical situation is that, you have a website where users write let's say movie reviews. So there are in general two types of users on your website, the ones who write normal amount like the rest of us, and the ones who write very often and are so called "top contributors".

Now the question is, how would you, using data, to define who are "top contributors"?

My idea was that, we'll take the last 24 weeks (~6 months) data, then we take the average daily contributions per user over each week, then we take average of these averages for each user. Then, with the average rate calculated, we can list out the top contributors overall or for each region. So basically:

for each user:

daily_avg = [# of contributions]/[# of days]   where [# of days]=7
weekly_avg = [sum of daily_avg]/[# of weeks]    where [# of weeks]=24

My questions are:

Does my approach make sense?
Would love to learn from your better approaches.

-- notes --

The reason why I picked only the past 6 months instead of all time, is because in case some users used to be extremely active but no longer is.

Again, here to learn from everyone. Appreciate any critique or suggestions!
Thanks.


texts: Formula to calculate size of Capsule output similar to the formula for CNN?

texts:
Is there any formula to find the output dimensions of a capsule network similar to that of a Convolutional Neural Network?

For Example: In CNN, we know that

(W - F + 2P/S)+1 = output_size where W = Dimensions of Input, F= Filter Size, p= Padding and S = Stride

Kindly note that the same formula does not work for capsule networks as well.