texts: How to collect data which exists centre or border of the cluster?

texts:
After applying clustering algorithm I need to extract those data which exists centre of the cluster and which exists border of the cluster. How could I do this by using python.

I use k-means clustering algorithm and divide those data into 19 cluster. I am also using scikit learn library.

Edited:

Here's my plot after clustering:


texts:
What loss-function / optimizer to use for fuzzy classification problems?
E.g: Four categories hot, mild, cold, freezing.

Edit:
I use one-hot encoding and have ~ 60 datapoints.


texts:
I am a newbie in the ML field. So please, neglect or better correct, if I am wrong somewhere.
Currently working on a model training for time series data.
My problem is a little more specific to bike-sharing. I have a count of bike-sharing for each area and for each bike type(gear, without gear...) for each day.

For example, Data:

Date    BikeType    Area    BikeCount
1/1/19  Gear        1        10
1/1/19  WithoutGear 1        15
1/1/19  Gear        2        8
1/1/19  WithoutGear 2        12
2/1/19  Gear        1        11
2/1/19  WithoutGear 1        17
2/1/19  Gear        2        9
2/1/19  WithoutGear 2        16

So, I will have a trend for each type of bike for each area.
How to use time series for this data.
I have to predict the bike required for each Type and each Area.
For example, for the given data I have to predict the count of Geared bike required and WithoutGeared bike required for both 1 and 2 areas on 3/1/19. (Considering two dates' data is enough for prediction, I have 2 years data for each specific area and type, they have a good trend)

And the second question is...
Currently, I have to only two dimension bike types and area, they may increase later(like their color and their condition) how to handle this.
Any contribution will be helpful.

I get a hint from the question:
https://stackoverflow.com/questions/55545501/how-to-perform-time-series-analysis-that-contains-multiple-groups-in-python-usin

But, is this the best any only way...

Thank you
(I need suggestion for the question title also)

Edit:

I get a similar problem in the following references:

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

Multivariate and multi-series LSTM

Now, I have two more doubts:

Is the LSTM is the only way?
Are my data columns (data types and area) dimensions or features.


texts: Feature construction widget on Orange 3.13

texts:
I got stucked with this widget... I am an absolute beginner and I am now working with Orange software for my thesis with logs and core data.
I need to combine different features to compare them.

What kind of information should I put in "Values" field with a categorical feature?

If you have some example with will be really appreciated. the ones from orange doesn't help me


texts: Regression ANN getting high root mean squared error values after re scaling dataset

texts:
I have a data set on predicting solar power generation, I am getting root mean squared loos of 0.3196 on training set on scaled values, but when I inverse transform them my loss rises to 298 on training and 488 on test set. but my r2scores are .883 and .69 on tests and training sets. R2 scores seems acceptable but why i am getting large value of error.
Here is my code:
    data_path = r'drive/My Drive/Proj/S.P.F./solarpowergeneration.csv'
    dts = pd.read_csv('solarpowergeneration.csv')
    dts.head()
    X = dts.iloc[:, :-1].values
    y = dts.iloc[:, -1].values
    print(X.shape, y.shape)
    y = np.reshape(y, (-1,1))
    y.shape
    from sklearn.model_selection import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

   from sklearn.preprocessing import StandardScaler
   sc_X = StandardScaler()
   X_train = sc_X.fit_transform(X_train)
   X_test = sc_X.transform(X_test)

   # outcome scaling:
   sc_y = StandardScaler()
   y_train = sc_y.fit_transform(y_train)
   y_test = sc_y.transform(y_test)

def get_spfnet():
  ann = tf.keras.models.Sequential()

  ann.add(Dense(X_train.shape[1], activation='relu'))
  ann.add(BatchNormalization())
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
  ann.add(BatchNormalization())
  ann.add(Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
  ann.add(BatchNormalization())

  ann.add(Dense(1))
  ann.compile(loss='mse',
              optimizer='adam',
              metrics=[tf.keras.metrics.RootMeanSquaredError()])
  return ann

spfnet = get_spfnet()
#spfnet.summary()
hist = spfnet.fit(X_train, y_train, batch_size=32, validation_data=(X_test, y_test),epochs=250, verbose=2)

the accuracy and loss graphs are
plt.plot(hist.history['root_mean_squared_error'])
plt.plot(hist.history['val_root_mean_squared_error'])
plt.title('Model error')
plt.xlabel('Epochs')
plt.ylabel('error')
plt.show()

what changes are needed to improve my model
dataset:

colab link of code


texts: How to display an Heat Map (using Tableau or other BI tool)?

texts:
I've a simple dataset:

Id
Longitude
Latitude
Value

What can I obtain at the moment are "one map point" in every place.. Sum of Value don't affect "density" color.
I would like to display a Geographic Heat Map like this (i copied from Google image). How ?


texts: How to find impactful words affecting classification?

texts:
So I know there are many methods to classify sentences into types. Like in sentiment analysis (positive, negative, neutral), spam emails (spam, not spam), etc. The thing I want to ask is how would I find the words most responsible for the categorisation. For example: sad-negative, happy-positive, the-no information, are-no information.

So how do I find the words which impact the classification?


texts: Is FPGrowth still considered "state of the art" in frequent pattern mining?

texts:
As far as I know the development of algorithms to solve the Frequent Pattern Mining (FPM) problem, the road of improvements have some main checkpoints. Firstly, the Apriori algorithm was proposed in 1993, by Agrawal et al., along with the formalization of the problem. The algorithm was able to strip-off some sets from the 2^n - 1 sets (powerset) by using a lattice to maintain the data. A drawback of the approach was the need to re-read the database to compute the frequency of each set expanded.

Later, on year 1997, Zaki et al. proposed the algorithm Eclat, which inserted the resulting frequency of each set inside the lattice. This was done by adding, at each node of the lattice, the set of transaction-ids that had the items from root to the referred node. The main contribution is that one does not have to re-read the entire dataset to know the frequency of each set, but the memory required to keep such data structure built may exceed the size of the dataset itself.

In 2000, Han et al. proposed an algorithm named FPGrowth, along with a prefix-tree data structure named FPTree. The algorithm was able to provide significant data compression, while also granting that only frequent itemsets would be yielded (without candidate itemset generation). This was done mainly by sorting the items of each transaction in decreasing order, so that the most frequent items are the ones with the least repetitions in the tree data structure. Since the frequency only descends while traversing the tree in-depth, the algorithm is able to strip-off non-frequent itemsets.

Edit:

As far as I know, this may be considered a state-of-the-art algorithm, but I'd like to know about other proposed solutions. What other algorithms for FPM are considered "state-of-the-art"? What is the intuition/main-contribution of such algorithms?

Is the FPGrowth algorithm still considered "state of the art" in frequent pattern mining? If not, what algorithm(s) may extract frequent itemsets from large datasets more efficiently?


texts: Are linear regression models with non linear basis functions used in practice?

texts:
I know that popular linear regression models such as Lasso or Logistic Regression are widely used in practice because they perform reasonably good, are efficient and interpretable.

As far as I know, the only way for these models to learn non-linear relationships between $X$ and $y$ is by applying non-linear transformations to $X$ through basis functions $\phi(X)$ (as it is summarized in the scikit-learn documentation).

I wonder if this approach is actually used in real world problems, where usually we don't have any prior information about how to design $\phi$.


texts: Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result?

texts: I am working with a large data set (~9M rows with 20+ features). Is it ok to tune via grid search on a fraction of the data (~100k rows) to determine optimal hyperparameters? This is mostly for choosing max_features, min_samples, max_depth. Trees and learning rate come later. Will I get different results tuning the fraction versus the whole data set?


texts: what is a tf.TensorSpec() and what is it used for?


texts: Choosing weights on random forest for imbalanced data with the aim to minimize false positives

texts:
I am currently dealing with a binary classification task on imbalanced data with the following distribution:

y_train: 4981 positive / 863894 negative samples
y_test:  128  positive / 128309 negative samples

The goal is to aim for a high precision (as little false negatives as possible).

How do I go on about choosing the weights for the random forest?

I tried to balance out the y_train ratio by assigning weight 1 to "negative" and 173 to "positive", but that still caused all the samples to be assigned to negative.

At this point, should I already consider this a problem with the features used, or should I try to assign higher weights with GridSearch (roc_auc as scoring parameter) and set the decision-threshold higher first?


texts: normalization of probabilities in predicting a poly-neuron output in neural nets

texts: When predicting a poly-neuron output in neural nets, say, predicting multiple handwritten digits and giving an output neuron vector (0.1,...,0.9,0.1,...), many use sth like softmax (or sth like the energy dependent probability exponential formula in statistical mechanics) to normalize the output vector such that all the components of the output vector sum up to 1, and that the normalized output vector becomes a probability vector. I doubt the necessity of this normalization, for without which I can equally well predict as per the biggest vector component. Is there anything I overlooked?


texts: Weighted Entropy (or) Conditional Entropy calculation error

texts:
I'm getting an assertion error for the test case: Y = np.array([0.,1.,0.,1.]), X = np.array([1.,1.,4.,4.])

The code executes when assert np.allclose(ce,0.,atol =1e-3)and doesn't pass when assert np.allclose(ce,1.,atol=1e-3)

Can you please correct the code to sort this out?

Code:

def conditional_entropy(Y,X):
    eps = np.finfo(float).eps
    attribute = X
    target_variables = np.unique(Y)
    variables = np.unique(X)
    entropy_attribute = 0

    for variable in variables:
        entropy_each_feature = 0
        for target_variable in target_variables:
            temp =0
            for i in range(len(X)):
                if(X[i]==variable and Y[i]==target_variable):
                    temp = temp+1
             num = temp
             print(temp)
             den = np.count_nonzero(Y==target_variable)
             print(den)
             fraction = num/(den+eps)
             entropy_each_feature += - fraction*(np.log(fraction+eps))
             print(entropy_each_feature)

    fraction2 = den/len(X)

    entropy_attribute += -fraction2*entropy_each_feature

    ce = entropy_attribute

    print(ce)
    return ce


texts: How to optimize hyperparameters in stacked model?

texts:
I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air!

Thanks,

Jack


texts: Getting inf% accuracy for Random Forest model with scikit learn

texts:
I have pandas dataframe with 8000 observations and with many different columns -some of them are dates and hours that were converted into dummies (get_dummies) and some are numerical,and y label which is what I want to predict that has two classes: 0 or 1:
>>>y   x1     x2     x3     x4   10   11   12   13
0  1  0.532  0.431  0.214  0.11  0    0    1    0
1  0  0.512  0.410  0.340  0.09  0    1    0    0
...

I have used random forest model in scikit learn in order to predict the y column (treatment) either is 1 or 0.
the results suprised me because when I print the accuracy I got -inf  and I did not excpect that.
this is how I calcualte and run the rf:
X = df.drop(['y',],axis=1)
y = df['y'].astype(int)

#split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=123)

###Random forest

#function for evaluate:
def evaluate(model, test_features, test_labels):
    predictions = model.predict(test_features)
    errors = abs(predictions - test_labels)
    mape = 100 * np.mean(errors / test_labels)
    accuracy = 100 - mape
    print('Model Performance')
    print('Average Error: {:0.4f} degrees.'.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%.'.format(accuracy))

    return accuracy

#random forest
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier()
rfc.fit(X_train,y_train)
predictions = rfc.predict(X_test)

base_model = RandomForestClassifier()
base_model.fit(X_train, y_train)
base_accuracy = evaluate(base_model, X_test, y_test)

>>>Model Performance
Average Error: 0.3290 degrees.
Accuracy = -inf%.

the confusion matrix looks like this:

so my question is what that -inf% accuracy mean? where is the problem in the model?


texts: Cross Validation with Artificial Neural Network in Keras

texts:
I needed to use cross-validation in the Neural network.
I used
 kf = KFold(n_splits=5, shuffle=True)

and
  for train_index, test_index in kf.split(train):
    train_X, train_y = train_XX[train_index], train_yy[train_index]

but I couldn't manage to run. One of the errors I see is
ValueError: Cannot have number of splits n_splits=5 greater than the number of samples: n_samples=2.

I would be glad to have your suggestions if you ever tried CV with in Neural network.


texts: Unsupervised learning if existing image captions match the images

texts:
I need to train a system on a large set of images and associated captions to determine which (image, caption) pairs are correct and which are not. I don't have any labeled pairs, but I can assume that more pairs are correct than are not. Furthermore, the images and captions have small subsets that are reasonably similar, but not exactly.

Current idea

My current idea is to build a sort of GAN architecture where I have a generator and a discriminator, except that the generator is simply sampling from the existing corpus of images and captions to generate false pairs and the discriminator is in charge of determining which pairs are correct or not.

images, captions = [], []
bpg = bad_pair_generator(images, captions)

clfr = classifier()

for epoch in range(epochs):
    good_pairs, bad_pairs = zip(images, captions), bpg.generate(N)
    X, y = good_pairs + bad_pairs, [1]*len(good_pairs) + [0]*len(bad_pairs)

    clfr.train(X, y)

    trickiness = [yi == yi_pred for yi, yi_pred in zip(y, clfr.predict(X))]
    bpg.fit(bad_pairs, trickiness[-len(bad_pairs):])

After this, I would hope that the classifier was stressed sufficiently to learn a good model for whether the pairs are good or not.

Questions

Are there better researched approaches?
If not, does this approach seem sane and reasonably safe?
What kind of model could I use to compared the image to the text? Clearly a CNN/RNN would be good for the image/text, but how do you combine them?


texts:
I have built a face recognition model based on Pyimagesearch Tutorial

It works great on images but the issue I am facing is with videos.

I am recognizing the faces frame by frame(5fps). It gives me people present in the particular video frame.

But the same person is identifed with different names in different frames.

How can I restrict it to identify same name with throughout the video?

Sometimes it also recognized two different people with same name.
Eg. Sam and Dean are present in the frame but it identifies both as sam.


texts: Adding and subtract inbetween row inputs and value equal to the first column next row using pandas

texts:
Assume I have a dataset with three inputs:

   x1   x2  x3
0  a    b   c
1  d    e   f
2  g    h   i
3  j    k   l
4  m    n   o
5  p    q   r
6  s    t   u
      :
      :

0,1,2,3 are times, x1, x2, x3 are inputs that are measured. So here x1 inputs are measured at every one hour. x2 and x3 will be measured at different time. What I need to do , I want write that what ever the measured in x1, x2, x3 it will add and subtract the values are equal to the x1 input next time value
So here what I want to do is:

    x1   x2   x3   y
 0  a    b    c   a+b-c==d
 1  d    e    f   d+e+f==g
 2  g    h    i   g+h-i==j
 3  j    k    l   j+k-l==m
 4  m    n    o   m+n-o==p
 5  p    q    r   p+q-r==s
 6  s    t    u   s+t-u==v
         :
         :

I tried shift method and it didn't work for me what I want exactly. I tried another method and it worked, but didn't came as I want. Here I upload the code.

Code :
 data = pd.read_csv('data6.csv')
 i=0
 j=1
 while j < len(data):
   j=data['x1'][i] - data['x2'][i] + data['x3'][i]
   i+=1
   j!=i
  print(j)

This is works , but it is just showing only one data

  63

In my csv file this is second input value of x1 input.
I want to write this code contonously happened and read the value as I shown above.
Can anyone help me to solve this problem?

My csv file


texts: Doc2Vec - How to label the paragraphs (gensim)

texts:
I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint.

Do you need to have each sentence / paragraph / document with its own unique label (e.g. "Sent_123")? This seems useful if you want to say "what words or sentences are most similar to a single specific sentence labeled "Sent_123".

Can you have the labels be repeated based on content? For example if each sentence / paragraph / document is about a certain product item (and there are multiple sentence / paragraph / document for a given product item) can you label the sentences based on the item and then compute the similarity between a word or a sentence and this label (which I guess would be like an average of all those sentences that had to do with the product item)?


texts: Use Machine Learning/Neural Network + Distance Measurements to Find the Position of Devices (Localization)

texts:
I want to find the position of several devices using at least distance measurements. These measurements are done using a radio, and it might be that not all devices are in radio range (no distance measurement can then be done). All devices are taking turns on being the "distance measurement master", while the rest of the devices are acting as "distance measurement slaves".
The (training) data generated looks something like this (all the numbers are totally random):

master
slave_0
slave_(...)
slave_n
master_x_coordinate
master_y_coordinate

0
5684
897
2378
500
3650

1
690
NaN
722
8450
894

2
9875
645
NaN
348
1580

3
1059
269
8952
5697
36

4
598
896
NaN
7836
365

The numbers under slave_x are distances in millimeter, and NaN means that there were no radio contact (no distance measurement done).
I am able to train a model using TensorFlow (Keras) that to my understanding uses regression (Mean Absolute Error):
n_inputs = training_features_normalized.shape[1]
n_outputs = training_labels.to_numpy().shape[1]

model = Sequential()
model.add(Dense(512, input_dim=n_inputs, kernel_initializer='he_uniform', activation='selu'))
model.add(Dense(512, activation='selu'))
model.add(Dense(512, activation='selu'))
model.add(Dense(512, activation='selu'))
model.add(Dense(512, activation='selu'))
model.add(Dense(512, activation='selu'))
model.add(Dense(n_outputs))

model.compile(loss='mae', optimizer='adam')

model.fit(training_features_normalized, training_labels.to_numpy(), verbose=1, epochs=20)

'training_features_normalized' consists of the columns: 'master', 'slave_0', 'slave_(...)' and slave_n (normalization has been done).
'training_labels' consists of the columns: 'master_x_coordinate' and 'master_y_coordinate'.
This trained model gets good at predicting master_x_coordinate and master_y_coordinate for nodes (positions and distances) that are included in the training data, but the model is bad at predicting the position of devices that are not included in the training data.
So my question is:
Is there a way to train a model using this kind of training data so that the model can predict the position of "new" nodes with "new" positions (positions and relating distance measurements that are not included in the training data)?


texts: Machine Learning: Why do the error in cost function need to be squared?

texts:
I have recently started Andrew NG's Machine Learning course on Coursera and I came across this cost function which is:

Why does the error in the cost function need to be squared? If its purpose is to eliminate the negative sign in the error then why don't we simply use the absolute function?


texts:
a common technique to get an image to a particular size is by either resizing it completely which can lead to losing the aspect ratio or e.g. resizing the bigger side and then 0-padding the other.
My question here is: if I resize the image e.g. from (120,120) to (240,240) with some form of interpolation, I'm also changing my resolution am I not; because lets say in the original picture 1px displays 1 inch, so in the resized image 1px displays 1/2 inch.
This means, in the end, I´m feeding my CNN images of the same SIZE but different RESOLUTION. So how can I create Images of same SIZE and RESOLUTION?


texts: What is stored in heap structure in the following example?

texts:
I am planning to use heap structure to find the minimum distance between a set of 2D points and form a cluster.. and after to spend a couple of hours surfing on the internet, I have not still gotten a clearly example.

Imagine you have a set of 2d points (x1,y1; x2,y2;....;xn,yn) and then you compute all possible distances between these pairs of points to finally get a matrix with all distances values.

After that, I want to use a min heap structure to sort data and match the pairs with the mininum distance... and here is my question...

Which value/data is really stored at every root or leaves in heap structure? I know that the root of min Heap is the shortest distance but, the data stored in every globe at Heap graph represent the index of every distances of the pairs of points or the pairs of points?

In the first case, if you store the index value located at the top of heap, Do you need to perform after a new search in the distance matrix to obtain which pairs of points form these shortest distance? Is it optimal to do that?

I hope my question was clear and thanks for your support.


texts: How to handle large number of features in machine learning?

texts:
I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case, it's clearly out of question to examine each features one by one to figure out what are they exactly and what optimization or feature engineering could be done with them.

Still, I have to do all the necessary preprocessing steps like imputation, standardization etc. But even such basic steps like categorical feature encoding or imputation are problematic because R/Python-pandas are sometimes wrongly recognized the numeric/categorical nature of some variables (and as a consequence, wrongly try to encode or mean-impute the NAs), not to mention other very problematic issues that could be handled if one could oversee the features one by one.

Of course, I could turn to models which are capable of handling non-standardized features with NAs but this limits the number of possible models on one hand and seems me very unprofessional on the other hand. What is the way to get over this issue?


texts: How to encode multiple inputs and multiple outputs

texts:
I have a piece of math more complicated than I can understand at the moment, a dynamic graph visualization. It uses a physics metaphor of springs and magnets, where the vertices act as magnets repelling each other, and the edges act as springs pulling the vertices back together. This simulation has six working parameters at the moment, which I graph from the running simulation at an interval.

I am trying to find the settings for the simulation necessary to produce a stable visualization.

My approach so far has been to make an array of settings, and an array of edge lengths for a fixed arrangement (a simple tetrahedron). I'm not entirely sure where I'm going with this since I'm still very new to machine learning, but I figured that by randomizing the settings during the data collection run, I would be able to predict the lengths of the edges for a given set of settings, then use regression to find the settings corresponding to stable lengths of edges.

To do this, I began adapting Google's Tensorflow.js tutorial, skipping the first data visualization part because I have 6 setting inputs, and 6 edge length labels.

The model looks like this:

  function createModel() {
    // Create a sequential model
    const model = tf.sequential();

    // Add a single hidden layer
    model.add(tf.layers.dense({inputShape: [6], units: 1, useBias: true}));

    // Add an output layer
    model.add(tf.layers.dense({ units: 6, useBias: true}));

    return model;
  }

I got the model to train after the graph visualization "blows up", that is the vertices disappear from the screen after edges are added, with the lengths of the edges recorded along the way. But I am stuck on testModel function:

function testModel(model, inputData, normalizationData) {
    console.log('inputData.shape', inputData.shape);

    const {inputMax, inputMin, labelMin, labelMax} = normalizationData;

    // Generate predictions for a uniform range of numbers between 0 and 1;
    // We un-normalize the data by doing the inverse of the min-max scaling
    // that we did earlier.
    const [xs, preds] = tf.tidy(() => {

      const xs = tf.linspace(0, 1, 100);
      console.log('xs', xs)
      const preds = model.predict(xs.reshape([null, 6]));

      const unNormXs = xs
        .mul(inputMax.sub(inputMin))
        .add(inputMin);

      const unNormPreds = preds
        .mul(labelMax.sub(labelMin))
        .add(labelMin);

      // Un-normalize the data
      return [unNormXs.dataSync(), unNormPreds.dataSync()];
    });

    const predictedPoints = Array.from(xs).map((val, i) => {
      return {x: val, y: preds[i]}
    });

    const originalPoints = inputData.map(d => ({
      x: d.settings, y: d.lengths,
    }));

    tfvis.render.scatterplot(
      {name: 'Model Predictions vs Original Data'},
      {values: [originalPoints, predictedPoints], series: ['original', 'predicted']},
      {
        xLabel: 'settings',
        yLabel: 'edge lengths',
        height: 300
      }
    );
  }

but specifically, it is these lines that throw an error:

const xs = tf.linspace(0, 1, 100);
console.log('xs', xs)
const preds = model.predict(xs.reshape([null, 6]));

(The reshape!)

So my question for now is this: How do I tell tensorflow to give me a list of 6 random values to predict the function with, instead of just a single value?

Thanks in advance!

The complete example currently lives at this github page. (click Construct Pyramid and wait for the visualization to blow up, and the tensorflow will take over.)

Maybe the entire approach is wrong, any feedback is welcome.


texts:
I have data as below:

Carpanter
Carepnter
Carpentor
Labourer
Labor
Labour
Housewife
House Wife
housewife.

I want to clean data and rectify the spelling mistakes but not manually because its a huge data. Due to spelling mistakes these 50/60 occupations have become around 2000.


texts:
I have the following model:

a table with events, each row is event, each event has a timestamp.

I want to try and learn how many event happen per day of week per hour.

I tried to work with the naive bayes learner (with binning), but got bad
results, I think because of the binning, the numbers does have meaning.

I'm assuming there is a relation between day and hour to the amount of events, some simple formula/tree.

What learner should i use that can work with numbers and not nominal values?

thanks,


texts: Tensorflow dataset batching for complex data

texts:
I tried to follow the example in this link:

https://www.tensorflow.org/programmers_guide/datasets

but I am totally lost about how to run the session. I understand the first argument is the operations to run, and feed_dict is the placeholders (my understanding is the batches of the training or test dataset),

So, here is my code:

batch_size = 100
handle_mix = tf.placeholder(tf.float64, shape=[])
handle_src0 = tf.placeholder(tf.float64, shape=[])
handle_src1 = tf.placeholder(tf.float64, shape=[])
handle_src2 = tf.placeholder(tf.float64, shape=[])
handle_src3 = tf.placeholder(tf.float64, shape=[])

I create the dataset from mp4 tracks and stems, reading mixture and sources magnitudes, and pad them to be suitable to batching

dataset = tf.data.Dataset.from_tensor_slices(
    {"x_mixed":padded_lbl, "y_src0": padded_src[0], "y_src1":   padded_src[1],"y_src2": padded_src[1], "y_src3": padded_src[1]})

dataset = dataset.shuffle(1000).repeat().batch(batch_size)

iterator = tf.data.Iterator.from_structure(dataset.output_types, dataset.output_shapes)

from the example I should do:

next_element = iterator.get_next()

training_init_op = iterator.make_initializer(dataset)
for _ in range(20):
    # Initialize an iterator over the training dataset.
    sess.run(training_init_op)
    for _ in range(100):
        sess.run(next_element)

However, I have a loss, summaries, and optimiser operations and need to feed the data as batches, following another example as:

l, _, summary = sess.run([loss_fn, optimizer, summary_op], feed_dict={handle_mix: batch_mix, handle_src0: batch_src0,
                             handle_src1: batch_src1, handle_src2: batch_src2, handle_src3: batch_src3})

So I thought something like:

batch_mix, batch_src0, batch_src1, batch_src2, batch_src3 = data.train.next_batch(batch_size)

or maybe a separate run to fetch the batches first, then run the optimisation as above, such as:

batch_mix, batch_src0, batch_src1, batch_src2, batch_src3 = sess.run(next_element)
l, _, summary = sess.run([loss_fn, optimizer, summary_op], feed_dict={handle_mix: batch_mix, handle_src0: batch_src0, handle_src1: batch_src1, handle_src2: batch_src2, handle_src3: batch_src3})

This last attempt, returned string names of the batches as created in the tf.data.Dataset.from_tensor_slices ("x_mixed", "y_src0", ... etc) and failed to cast to tf.float64 placeholders in the session.

Can you please let me know how to create this dataset, there might be an error in the structure from tensor slices in the first place, then how to batch them,

thank you very much,

Manal


texts: Feature selection caused more overfitting

texts:
What does it mean if I used sklearn.feature_selection.SelectFromModel to select features and cross-validation score went up but the testset score went down. This was even more significant compared to when I did not use feature selection.

Does this mean it's causing more overfitting?

I thought feature selection is used to reduce overfitting.


texts:
I am looking for a way to extract sequences/patterns from a dataset such as this one:

dataset = ['sample1', 'sample2', 'sample3', 'sample1', 'sample2', 'sample3', 'sample3', 'sample2'...]

And my goal is to know that the sequence ['sample1', 'sample2', 'sample3'] occurs 2 times in this dataset.
Ideally, I would also like to know all sequences that occur more than once in my dataset.

Is there a library (sklearn...) that could help me do that or do I just have to iterate over my dataset and test each and every possible combination? I assume there must be a more intelligent way to do that.

Thanks for your help!


texts: Text Localization using Convolutional NN

texts:
I am new on Tensorflow. I am trying to build a neural network that is able to recognize letters and words from documents.

I have divided the task into 3 phases as what is mentioned in ICDAR2017:

Text Localization
Cropped Word Recognition
End-to-End Recognition

I am having some problems in the 1st phase which is Text Localization. I used an architecture that is called EAST.

Architecture stages:

Fully Convolutional Network
NMS merging stage

This is the model I started with. It works fine but I am facing some problems in detecting some characters and letters as in:

Model layers:

def model(images, weight_decay=1e-5, is_training=True):
        '''
        define the model, we use slim's implemention of resnet
        '''
        images = mean_image_subtraction(images)

        with slim.arg_scope(resnet_v1.resnet_arg_scope(weight_decay=weight_decay)):
            logits, end_points = resnet_v1.resnet_v1_50(images, is_training=is_training, scope='resnet_v1_50')

        with tf.variable_scope('feature_fusion', values=[end_points.values]):
            batch_norm_params = {
            'decay': 0.997,
            'epsilon': 1e-5,
            'scale': True,
            'is_training': is_training
            }
            with slim.arg_scope([slim.conv2d],
                                activation_fn=tf.nn.relu,
                                normalizer_fn=slim.batch_norm,
                                normalizer_params=batch_norm_params,
                                weights_regularizer=slim.l2_regularizer(weight_decay)):
                f = [end_points['pool5'], end_points['pool4'],
                     end_points['pool3'], end_points['pool2']]
                for i in range(4):
                    print('Shape of f_{} {}'.format(i, f[i].shape))
                g = [None, None, None, None]
                h = [None, None, None, None]
                num_outputs = [None, 128, 64, 32]
                for i in range(4):
                    if i == 0:
                        h[i] = f[i]
                    else:
                        c1_1 = slim.conv2d(tf.concat([g[i-1], f[i]], axis=-1), num_outputs[i], 1)
                        h[i] = slim.conv2d(c1_1, num_outputs[i], 3)
                    if i <= 2:
                        g[i] = unpool(h[i])
                    else:
                        g[i] = slim.conv2d(h[i], num_outputs[i], 3)
                    print('Shape of h_{} {}, g_{} {}'.format(i, h[i].shape, i, g[i].shape))

                # here we use a slightly different way for regression part,
                # we first use a sigmoid to limit the regression range, and also
                # this is do with the angle map
                F_score = slim.conv2d(g[3], 1, 1, activation_fn=tf.nn.sigmoid, normalizer_fn=None)
                # 4 channel of axis aligned bbox and 1 channel rotation angle
                geo_map = slim.conv2d(g[3], 4, 1, activation_fn=tf.nn.sigmoid, normalizer_fn=None) * FLAGS.text_scale
                angle_map = (slim.conv2d(g[3], 1, 1, activation_fn=tf.nn.sigmoid, normalizer_fn=None) - 0.5) * np.pi/2 # angle is between [-45, 45]
                F_geometry = tf.concat([geo_map, angle_map], axis=-1)

        return F_score, F_geometry

I have been advised to use Pyramid Networks, I don't know who to integrate this Pyramid Networks with the EAST model in Tensorflow and it is worth it or not.

I would like also to know my options to increase the Text Localization accuracy in the table based data


texts: Forcing distinct, independent sets of outputs in a neural network architecture

texts:
I'm building a neural network architecture, which starts with an autoencoder, which reduces the dimensionality of the data and performs something akin to a PCA - finds a lower-dimensional representation of the data.
On the basis of this representation, I'm trying to perform reinforcement learning, but I'm trying to find distinct, "independent", strategies.
How do I prevent the reinforcement learners from converging to the same strategy and enforce "independence"?
Thank you very much in advance.
(Alternatively, this could be implemented as multiple sets of feedforward neural network layers. But the question remains, how do I penalize "dependence" / "similarity" across those sets of layers?)


texts: Where do Kohonen and counterpropagation networks fall in the scheme of neural network architectures?

texts:
This site sums up various neural network architectures, which fall into broader categories listed below:

feed-forward networks (don't form cycles)
recurrent networks (have directed cycles)
symmetrically connected networks

Then, specific types of neural networks are listed:

perceptron
convolutional
recurrent
long/short term memory
Hopfield
Boltzmann machine
deep belief
autoencoder

I have not found where Kohonen (now more commonly referred to as self-organizing maps) and counterpropagation networks (multi-layer Kohonen networks) fit into this scheme in this or other literature.  They used to be more commonly used in the 1980s and 1990s but I am not sure where they fall in the categorization of neural nets.


texts:
I've started reading some literature about reinforcement learning and I can't understand what is the result of the application of RL. I'll be more specific: let's have a time series problem in continuous state space, finite numbers of actions and a linear approximator of the policy function. So I follow an algorithm to find the best policy, that is, in this specific case, the optimal values of the weights of the linear function I've considered. Now my doubt is here: the so-called best policy is the one found in the process of applying the algorithm or I have to take the final optimal values and, for each period, I have to use them to find which action maximise the action-value function? In other words, the result of RL is a classic function to (re)apply at each time step, as if it was a regression? I think the answer of this question is No, but I would appreciate if someone can confirm this.

(to better explain what I meant with "policy found in the process of applying the algorithm" let's consider this stupid consideration: the best policy also include those time steps of exploration)


texts: What are the differences between MLP and DNN?

texts:
So I have been reading about the topic for a while, but i did not find a clear answer why MLP and DNN are being used interchangeably even though there are some differences between them.
So far I have filtered some informations:
"The field of artificial neural networks is often just called neural networks or multi-layer perceptrons after perhaps the most useful type of neural network.
A perceptron is a single neuron(input, output, weights, activation) model that was a precursor to larger neural networks.
MLP is a subset of DNN.
While DNN can have loops and MLP are always feed-forward(a type of Neural Network architecture where the connections are "fed forward", do not form cycles (like in recurrent nets).
Multilayer Perceptron is a finite acyclic graph, not like RNN and it's subsets which are cyclic in nature.
MLP uses back propagation for training the network."
So what makes MLP different from DNN ?


texts: Understanding one of the assumptions of linear regression: Multicollinearity

texts:
I've read that multicollinearity is one of the main assumptions of multivariate linear regression - Multicollinearity occurs when the independent variables are too highly correlated with each other.
However, when learning linear regression, one of the key topics is the idea of introducing interaction terms into the model to model the interaction effect which is when the effect of an independent variable on a dependent variable changes, depending on the value(s) of one or more other independent variables.
Aren't these two statements contradictory? If there were really interactions between $X_1$ and $X_2$ in the model $y = \beta_0 + \beta_1*X_1 + \beta_2*X_2$ surely we should remove either $X_1$ or $X_2$ so that the independent variables in the regression model are no longer correlated thus the multicollinearity assumption holds. Putting in the interaction terms seems to ignore this assumption and rather introduce a further term to further complicate it.
From a modeling standpoint this makes sense, but doesn't the mathematics breakdown if we do this?


texts: How to speedup message passing between computing nodes

texts:
I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire process as possible, I'm wondering if there any standard solutions to improve the performance of the message passing layer of my application.

What changes/improvements could I apply to my code to reduce the time spent sending messages? For what it's worth, I'm communicating up to 10GB between 9 computing nodes, and the framework I'm using is implemented with OpenMPI.


texts: Classify sentence into predefined category

texts:
I am working on a project which takes text data (Social Media , reviews etc) and classify into a predefined category.

Now I already have a Sentiment Analyzer in place. So basically I am only working with Negative sentiment text.

Now I have a predefined category ex. 'Data speed is slow', 'No Network Coverage' etc. The total number of category is around 30.

My goal is to classify those text into those categories.

I have already tried few methods with limited success:
1. Defining Anagrams (bi and tri) for each category and then filtering.
2. Converting sentences to Sentence Vector Embedding and finding cosine similarity between sentence embedding and  an embedding for a category template.
3. KMeans Clustering and LDA.

Note : I don't have an option for labeling now so cannot do supervised learning. But I wanted to know whether Supervised learning would be the best option here.

I wanted to know the best strategy here using unsupervised technique. How to accurately capture the semantics and classify. Any traditional NLP approach which can be used here?.  Any ideas would be appreciated.


texts: How to use ontologies for text classification?

texts:
I am new to machine learning and want to classify sentences using ontologies (taxonomies/ knowledge graphs) and supervised learning methods (I have an annotated training dataset).

My question is how to use the ontology for this task? Is the following method correct?

I will first perform the tokenization, stemming and stop word removal (pre-processing). Then, I will search for each term in the ontology and after finding them, I will add their related hierarchy to an array or vector for each document. Then I will train the classifiers (supervised) on those vectors.

Please let me know if this method is correct or if there are steps that I am missing here.

Thanks! :)


texts: A/B test results contradictory with offline machine learning model performance

texts:
This seems to be a common problem when bringing machine learning models to production.
Let's say we have an optimized machine learning model which gives decent performance metric in the unseen testing dataset. We are quite satisfied with that, and decided to bring the model online. Then we use A/B test to compare our website performance (i.e., revenue, customer engagement etc) with and without the new model. Somehow, our new model is not a clear winner or even a clear loser in the A/B test. How do we deal with such situation?
Here the model I mentioned is a machine learning model, for example ranking algorithm or a recommendation algorithm, but can be any algorithm in reality. Thanks for any help!


texts: Why might several types of models give almost identical results?

texts:
I've been analyzing a data set of ~400k records and 9 variables The dependent variable is binary. I've fitted a logistic regression, a regression tree, a random forest, and a gradient boosted tree. All of them give virtual identical goodness of fit numbers when I validate them on another data set.

Why is this so? I'm guessing that it's because my observations to variable ratio is so high. If this is correct, at what observation to variable ratio will different models start to give different results?


texts:
I have a binary classification problem.
The accuracy score is 52%
The precision for 0 is 53% and the precision for 1 is 49%
When using predict_proba() does this mean my model more accurately predicts when the outcome should be classified zero as opposed to one?
I'm not sure if this is telling me that I should be using the the first value (ynew[0][0]) returned from predict_proba() as opposed to the second (ynew[0][1]).
Here is the entire confusion matrix:


texts: Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set?

texts:
As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my training or testing set. I have about 10 features, most of which are binary.

Some of the parameters that I was tuning were:

the number of trees/iterations;
learning depth;
and learning rate.

The percentage of negative values seemed to max at ~2%. The learning depth of 1(stumps) seemed to have the largest % of negative values. This percentage also seemed to increase with more trees and a smaller learning rate. The dataset is from one of the kaggle playground competitions.

My code is something like:

from sklearn.ensemble import GradientBoostingRegressor

X_train, X_test, y_train, y_test = train_test_split(X, y)

reg = GradientBoostingRegressor(n_estimators=8000, max_depth=1, loss = 'ls', learning_rate = .01)

reg.fit(X_train, y_train)

ypred = reg.predict(X_test)


texts: How to plot using facet_wrap, over multiple pages as a .pdf files in r cran

texts:
I am using ggplot, to compare 114 unique studies for a particular variable I'm interested in.
This is what I have used.
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

Whilst this works, all 114 studies are plotted on one page and the formatting is all squashed. How do I split this over 4x4 pages ?

Many thanks

S

edit ****

As there are 114 unique studies, I have 5 pages in total
1)
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 1)

2)
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 2)

3)
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 3)

4)
 ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 4)

5)

ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,))+ facet_wrap_paginate(~id_study, nrow = 5, ncol = 5, scales
 = "fixed", shrink = TRUE, labeller = "label_value", as.table = TRUE, switch = NULL, drop = TRUE, dir = "h", strip.position = "top", page = 5)

all of these into 1 page?
thank you


texts: Is it correct to say that a high variance feature will be important to my model?

texts:
Analysing 3 features I was faced with the following situation:
Two of them have almost no variance and the last one have a greater variance than the other (almost two times). Is it correct to assume using only this information that the feature with more variance will be more important to my model ? Also, if a feature has 0 variance the model simply won't learn anything from it, am I right?

Lastly, if you have some nice references I would love to read them.


texts: Is the result of feature extraction a feature representation?

texts:
If a use a feature extraction method on images, do I then get a feature representation or is there a different meaning behind feature representation?
To my understanding, when I use a CNN on an image the convolutional layers perform feature extraction. Can I then also say that I get a new feature representation after each of these layers?
Thanks in advance! I am relatively new to machine learning...


texts:
I'm wondering how to interpret a recurrent architecture in an EEG context. Specifically I'm thinking of this as a Recurrent CNN (as opposed to architectures like LSTM), but maybe it applies to other types of recurrent networks as well

When I read about R-CNNs, they're usually explained in image classification contexts. Theyre typically described as "learning over time" or "including the effect of time-1 on the current input"

This interpretation/explanation gets really confusing when working with EEG data. An example of an R-CNN being used on EEG data can be found here

Imagine I have training examples each consisting of a 1x512 array. This array captures a voltage reading for 1 electrode at 512 consecutive time points. If I use this as input to a Recurrent CNN (using 1D convolutions), the recurrent part of the model isn't actually capturing "time", right? (as would be implied by the descriptions/explanations discussed earlier) Because in this context time is already captured by the second dimension of the array

So with a setup like this, what does the recurrent part of the network actually allow us to model that a regular CNN can't (if not time)?

It seems to me that recurrent just means doing a convolution, adding the result to the original input, and convolving again. This gets repeated for x number of recurrent steps. What advantage does this process actually give?


texts: Why don't means of color channels average to equal flattened array mean

texts:
I am experimenting with different types of mean subtraction on RGB images for a convolutional neural network. I tried calculating a single grand mean of all the training images. I also tried calculating a mean per color channel. The grand mean I got was 122.84 which sounds like a reasonable number. However, when I tried calculating the means per color channel I got  [4.86049, 4.86049, 4.86049].
First off, I find it strange that they are all the same number. Secondly, they seem ridiculously low. And thirdly and perhaps most confusingly, shouldn't the mean of the three color channels equal the grand mean?

Maybe it is a problem with my code? This is what I'm using. The images are in a numpy array of the shape (17611, 224, 224, 3)

trainImgsArr = np.random.rand(17611, 224, 224, 3)
grandMean = np.mean(trainImgsArr)
meanByChannel = np.mean(trainImgsArr, axis = (0,1,2))

Any ideas?

EDIT: Ah, ha! I think I have found the true channel means with the code

 ch1, ch2, ch3 = np.split(trainImgsArr, 3, axis = 3)
 np.mean(ch1)
 np.mean(ch2)
 np.mean(ch3)

Sorry if this was super basic I am very new to python. Could someone perhaps explain where I went wrong using a single np.mean call as in the first code bit?


texts: Can we automatically chose k value in k-means algorithm?

texts:
Can we choose automatically the K value, trying every possible values (k=1,.., n) where n is the number of instances to be clustered. We then keep the value of K for which we obtained the minimum value by the method of sum of least squares.
Can this strategy work?


texts:
I am experimenting with xgboost.

I ran GridSearchCV with score='roc_auc' on xgboost. The best classificator scored ~0.935 (this is what I read from GS output). But now when I run best classificator on the same data:

roc_auc_score(Y, clf_best_xgb.predict(X))

it gives me score ~0.878

Could you tell me how the score is evaluated in both cases? Thanks


texts: Some suggestion for career in data science or predictive modeling

texts: I am looking to choose my career in the area of decision science or predictive modeling and I am aware that this is kind of opinion based but I would like to have some suggestion from experts that I can use it to build my career in correct path. What are the tools should I know like R, SAS or any other. What are the thinks I should know to work in a data science or machine learning or predictive modeling. For me I am having problem in identifying steps that I should follow. Please suggest me some steps to follow.


texts: loss is neither increasing nor decreasing in siamese network

texts:
I am making a model which differentiate between two fingerprint(dataset) using simple siamese network but even after 400 epochs loss doesn't decrease. Loss is stuck at 6000 and accuracy is also not increasing at all. I am using triplet loss to train the model and the code of the loss function is:
def triplet_loss(y_true, y_pred, alpha = 0.2):
    anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]

    pos_dist = tf.reduce_sum((anchor - positive)**2, axis=-1)
    neg_dist = tf.reduce_sum((anchor - negative)**2, axis=-1)
    basic_loss = pos_dist - neg_dist + tf.constant(alpha)
    loss = tf.reduce_sum(tf.maximum(basic_loss, tf.constant(0.0)))

    return loss

The model is as follows:
def model(input_shape):
  anc_inp = Input(input_shape, name='anchor_input')
  pos_inp = Input(input_shape, name='positive_input')
  neg_inp = Input(input_shape, name='negative_input')

  network = Sequential()
  network.add(Conv2D(128, (7,7), activation='relu', input_shape=input_shape))
  network.add(MaxPooling2D())
  network.add(Conv2D(128, (3,3), activation='relu'))
  network.add(MaxPooling2D())
  network.add(Conv2D(256, (3,3), activation='relu'))
  network.add(Flatten())
  network.add(Dense(4096, activation='relu'))
  network.add(Dense(128))
  network.add(Lambda(lambda x: K.l2_normalize(x,axis=-1)))

  anc_emb = network(anc_inp)
  pos_emb = network(pos_inp)
  neg_emb = network(neg_inp)

  model = Model(inputs=[anc_inp, pos_inp, neg_inp], outputs=[anc_emb, pos_emb, neg_emb])
  return model

and i used different types of optimizer to train the model but loss is not decreasing.
model_a = model((3, 96, 96))
adam_o = Adam(0.01)
sgd_o = SGD(0.1, momentum=0.1, nesterov=True)
ada = Adagrad(0.01)
model_a.compile(optimizer = adam_o, loss = triplet_loss, metrics = ['accuracy'])

I am using generator to train the model. The genrator is:
def get_triple(real_id, data_ids, dic_data, dic_real):
  while True:
    anc_id = np.random.choice(real_id)
    new_anc_id = [i for i in data_ids if i != anc_id]
    neg_id = np.random.choice(new_anc_id)

    anc_img = dic_real[anc_id][0]
    pos_img = np.random.choice(dic_data[anc_id])
    neg_img = np.random.choice(dic_data[neg_id])

    anc_img = np.around(np.transpose(cv2.resize(cv2.imread(anc_img), (96, 96)), (2, 0, 1))/255.0, decimals=6)
    pos_img = np.around(np.transpose(cv2.resize(cv2.imread(pos_img), (96, 96)), (2, 0, 1))/255.0, decimals=6)
    neg_img = np.around(np.transpose(cv2.resize(cv2.imread(neg_img), (96, 96)), (2, 0, 1))/255.0, decimals=6)

    yield [anc_img, pos_img, neg_img]
def batch_generator_RN(batch_size, real_id, ids, dic_data, dic_real):
    triplet_generator = get_triple(real_id, ids, dic_data, dic_real)
    y_val = np.zeros((batch_size, 2, 1))
    anchors = np.zeros((batch_size, 3, 96, 96))
    positives = np.zeros((batch_size, 3, 96, 96))
    negatives = np.zeros((batch_size, 3, 96, 96))

    while True:
        for i in range(batch_size):
            anchors[i], positives[i], negatives[i] = next(triplet_generator)

        x_data = {'anchor_input': anchors,
                  'positive_input': positives,
                  'negative_input': negatives
                  }

        yield (x_data, [y_val, y_val, y_val])


texts: How can the forecast accuracy of three models for a categorical time series be compared?

texts:
This is a general question. Can anybody please point me in the direction of how I can compare the forecasts of a 3 level categorical time series by three competing models? I would like to compare the forecasts to the ground truth of what actually occurred. The forecasts are 1 to 6 days ahead. Is it correct to analyse each day ahead separately? We have forecasts for 4 years. Should they be analysed separately to allow for variation in each year or can the years be used more wholistically? The prediction accuracy for each each model for each day is assessed with the Gerrity score. I am an R user.
I am happy to provide more info if required. Thanking you in advance.


texts:
I have a multicategorial classification problem for images. There are 5 (imbalanced) classes for which i use different class weights. In general there are only a few training images per class: ~56-238

To classify them, I use a neural network with much data augmentation. I have a validation set with the same distribution as the train set (but it only has about 30% of the images per class).

The resulting loss / accuracy graphs look a bit weird (edit: the second graph contains the term "Test Loss", but it is the "Validation Loss"):

I'm not sure how I can interpret these two images: The validation accuracy cleary increases, but the validation loss doesn't change too much. Can anyone help me with the interpretetion of these graphs?

Thank you very much


texts: Example of Logistic Regression using a proportion as the dependent variable?

texts:
I am trying to run a logistic regression on a data set where my dependent variable is a proportion of a binary variable, rather than the binary variable itself.  I have seen a ton of documentation that says this is possible, but I am having trouble finding an example of how to actually do it.  I am open to using scikit learn, statsmodels, or any other library that will do it.

I have added a photo showing a simplified version of my data.

successes here is just a count of a binary (1/0) outcome.  instead of having the individual observations, I only have them rolled up, but my understanding is that it is still a logistic regression problem.  I want to predict the dependent variable 'proportion' based on the features.  I understand this conceptually, but am just trying to find an example of this in python.  all of the examples I have seen assume a binary dependent variable.

your help is appreciated!


texts:
I have a graph for a model on train accuracy and validation accuracy:

But I'm having trouble interpreting it. By the way i interpreted, I would say it is of poor performance. But I would like to know how this graph is interpreted in a more broad manner.

As in if we were to look at the point where the validation acc begins to bisect the train acc line, and went all the way down, what does this indicate? Would appreciate some help on this as I'm horrible at interpreting graphs.


texts:
I code a lot for web, games and some basic ML scripts. Now I would like to learn about data science.

This post is a good starting point but I would like some readings. I would like advices on books for a beginner (maths, tools, whatever).

I've found these ones from O'Reilly :

Machine Learning for Hackers, By Drew Conway, John Myles White
Agile Data Science, By Russell Jurney
R Cookbook, By Paul Teetor

(Sorry, my reputation is too low to let me post more than 1 link haha)

Do you recommand these readings ? Are there other must-reading books ?

Thanks.


texts: What exactly is a dummy trap? Is dropping one dummy feature really a good practice?

texts:
So I'm going through a Machine Learning course, and this course explains that to avoid the dummy trap, a common practice is to drop one column. It also explains that since the info on the dropped column can be inferred from the other columns, we don't really lose anything by doing that.
This course does not explain what the dummy trap exactly is, however. Neither it gives any examples on how the trap manifests itself. At first I assumed that dummy trap simply makes the model performance less accurate due to multicollinearity. But then I read this article. It does not mention dummy trap explicitly, but it does discuss how an attempt to use OHE with OLS results in an error (since the model attempts to invert a singular matrix). Then it shows how the practice of dropping one dummy feature fixes this. But then it goes on to demonstrate that this measure is unnecessary in practical cases, as apparently regularization fixes this issue just as well, and algorithms that are iterative (as opposed to closed-form solution) don't have this issue in the first place.
So I'm confused right now in regards to what exactly stands behind the term "dummy trap". Does it refer specifically to this matrix inversion error? Or is it just an effect that allows the model to get trained but makes its performance worse, and the issue described in that article is totally unrelated? I tried training an sklearn LinearRegression model on a OHE-encoded dataset (I used pd.get_dummies() with the drop_first=False parameter) to try to reproduce the dummy trap, and the latter seems to be the case: the model got trained successfully, but its performance was noticeably worse compared to the identical model trained on the set with drop_first=True. But I'm still confused about why my model got successfully trained at all, since if the article is to be believed, the inversion error should have prevented it from being successfully trained.


texts: Time Series Forecasting with RNN/LSTM/NARX

texts: I have some experimental datasets (like 4 or 5), and each dataset has three time series data, say $u1(t)$, $u2(t)$, and $x(t)$. The three time series of each experiment are similar but not the same. How can I train my model to learn the relation between the series, such that when a new $u1(t)$ and $u2(t)$ come in, I can predict the $x(t)$ in real-time?


texts: Desicision tree classification with a "false" attribute

texts:
This is a pretty specific problem but I think it can help me understand better the whole concept of the subject.

A doctor in the hospital is in charge of 20 medical students. For every patient, the doctor consults with the students rather the patient has a problem. Each student can either say "yes", "no" or "Don't know".

The doctor builds a decision tree based on the answers of the students in period of two months. It is known that one of the students that answered the questions is an imposter, and all of his answers are based and opposite of another student the he picked. When the "real" student answered "yes", the imposter answered "no" and vice versa. When the "real" one answered "I don't know" so did the imposter.
Also it is known that every real student is giving a decisive answer - "yes" or "no" for at least 60% of the cases.

Tasks:

Give an algorithm to find the imposter.
A tree was built again without the imposter's answers. Prove it is the same as the old tree.

I understand that the information gain of the imposter and the real student he based his answers on is the same. So the algorithm needs to find groups of students with the same information gain. Within those groups, find two students with answered all reversed. The one who is wrong more than 60% of the time will be the imposter.

Regarding the second task I'm not sure why it is true. Thinking of a case of a patient where all the students answered "I don't know" except for one student who answered "yes" and the imposter who answered "no". Isn't it possible that the new tree will classify this patient as "yes" instead of "no", now when the imposter is gone?


texts: vgg16 needs less epochs than resnet ,why?

texts:
Recently i Have been comparing the vgg16 with resnetv1 with 20 layers.I have found out that although each epoch on vgg takes more time to complete,it generally needs less epoch to reach a certain training accuracy than resnet20.Why vgg learns faster ? is my experiments correct ? I have tried it on Cifar100 and a proportion of imagenet(tiny image net from stanford cv course)

the vgg has nearly 14m parameters but resnet has only 0.3m.
here is my implementation of the resent:

def resnet_layer1(inputs,initializer,
             num_filters=16,
             kernel_size=3,
             strides=1,
             activation='relu',
             conv_first=True,batch_normalization=True):

conv = Conv2D(num_filters,
              kernel_size=kernel_size,
              strides=strides,
              padding='same',
              kernel_initializer=initializer
             )

x = inputs
if conv_first:
    x = conv(x)
    if batch_normalization:
        x = BatchNormalization()(x)
    x = Activation(activation)(x)
else:
    if batch_normalization:
        x = BatchNormalization()(x)
    x = Activation(activation)(x)
    x = conv(x)
return x
def resnet_1(model_number,x_train,y_train,x_test,y_test,datagen,initializer,epochs=20,bs=512,output_nodes=10,optim='adam',padding='same',dout=True,callbacks=None):

depth=20
# Start model definition.
num_filters = 16
num_res_blocks = int((depth - 2) / 6)

if model_number == 1:
    resnet_layer=resnet_layer1
elif model_number == 2:
    resnet_layer=resnet_layer2
elif model_number == 3:
    resnet_layer=resnet_layer3
elif model_number == 4:
    resnet_layer=resnet_layer4

inputs = Input(shape=x_train.shape[1:])
x = resnet_layer(inputs=inputs,initializer=initializer)
# Instantiate the stack of residual units
for stack in range(3):
    for res_block in range(num_res_blocks):
        strides = 1
        if stack > 0 and res_block == 0:  # first layer but not first stack
            strides = 2  # downsample
        y = resnet_layer(inputs=x,initializer=initializer,
                         num_filters=num_filters,
                         strides=strides)
        y = resnet_layer(inputs=y,initializer=initializer,
                         num_filters=num_filters,
                         activation=None)
        if stack > 0 and res_block == 0:  # first layer but not first stack
            # linear projection residual shortcut connection to match
            # changed dims
            x = resnet_layer(inputs=x,initializer=initializer,
                             num_filters=num_filters,
                             kernel_size=1,
                             strides=strides,
                             activation=None,
                             batch_normalization=False)
        x = layers.add([x, y])
        x = Activation('relu')(x)
    num_filters *= 2

# Add classifier on top.
# v1 does not use BN after last shortcut connection-ReLU
x = AveragePooling2D(pool_size=8)(x)
y = Flatten()(x)
outputs = Dense(output_nodes,
                activation='softmax',
                kernel_initializer=initializer)(y)

# Instantiate model.
model = Model(inputs=inputs, outputs=outputs)
model.summary()
# Compile the model
model.compile(loss=['categorical_crossentropy'], optimizer=optim, metrics=["accuracy"])

checkpointer = ModelCheckpoint(filepath=str(model_number)+'_weights.hdf5', verbose=1,save_weights_only=True,save_freq=2000000)

callbacks.append(checkpointer)

if x_test.all() == None:
    history=model.fit_generator(datagen.flow(x_train, y_train,batch_size=bs),callbacks=callbacks,epochs=epochs, steps_per_epoch=x_train.shape[0]//bs)
else:
    history=model.fit_generator(datagen.flow(x_train, y_train,batch_size=bs),callbacks=callbacks,epochs=epochs,validation_data=(x_test, y_test), steps_per_epoch=x_train.shape[0]//bs)

return history,model


texts: Clustering the data if some values in each vector is not applicable

texts:
I have four different categories which the number of variables in each category is 7, 3, 8, and 9 (in total 27). The value of each variable changes from 1 to 5 but some of the variables were not applicable for some observations. In total, there are 20 observations which their data is available for 27 variables in the mentioned context (from 1 to 5 plus not applicable). How can I cluster these four categories for 20 observations to get three clusters for each category?

edit: There are 4 categories (A, B, C, and D). Each category has a different vector including 7, 3, 8, and 9 values (from 1 to 5 plus NA). Now, it's 18 observations for each category of A, B, C, and D. I need to make three groups for each category. How can I find the centroid of each cluster for each category? It means I need three centroid in each category, totally 12(=4 category* 3 groups). Any feedback would be appreciated. Thank you.


texts:
I've built hotel embeddings which gives very satisfactory results in returning similar hotels for each hotel.

Now the problem I'm trying to solve is to rank the hotels in order of relevancy to the user when they search for a location.

My current approach is to get the recent n bookings made by the user, average their embeddings and find the most similar hotels to this vector.

But this result is not satisfactory. Averaging hotel embeddings is not working as expected.

Any suggestions here?


texts:
I want to create a CNN in tensorflow that does the following:
Classify a recipe headline and find out the topic. For instance super yummy cheesy cake should result in cheese cake and so on.

I thought for going with tensorflow, but need some help in getting things started.

My strategy is like that:

Normalize the headlines so cheesy becomes cheese and cheesecake becomes cheese cake for instance and so on.
Having a dataset like:

super yummy cheesecake | cheese cake
summer strawberry cake | strawberry cake

Train the model to learn what matters for the topic and what is just additional information.

The way, the dataset is modeled, I have no static lables, as I understand. This makes things complicated, right?

As this is my first AI experiment with tensorflow, I don't really know if this will work out, or if I should go with another strategy, therefore I need your help.


texts:
Hello I am trying to understand LSTMs but have a few problems:

What is the input? Since LSTM is seq2seq I would think it is a sequence of words, but in a Codecademy lesson is mentioned that each sentence is represented as a matrix with a bunch of vectors containing 1 or 0 for the timestep -> sentence "I like Bobo" like = [0, 1, 0], so what is now the input? The matrix or the sequence of words?

What is passed to the next LSTM cell after a prediction before was false? Since the false prediction is noted in the hidden state, how does the network know whether previous predictions were false? Or does it even know when predicting the next step?

I am excited for the answers,
love Phiona.


texts:
what are the default kernels used in convolution done in cnn

for example in this code of keras there are 32 filters of size 5x5 each.

my question is 1) What will be the different default kernels in those 32 filters.
for example in an tutorial I read that the
\begin{bmatrix}
    1 & 1 \\
    -1       & -1\\
\end{bmatrix} could be taken as horizontal edge detector and
\begin{bmatrix}
1& -1\\
1 & -1
\end{bmatrix}
could be taken as vertical edge detector .

def baseline_model():
    # create model
    model = Sequential()
    model.add(Conv2D(32, (5, 5), input_shape=(1, 28, 28), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Conv2D(32, (5, 5), activation='relu'))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


texts:
I am trying to train the model, I keep ending up with this ValueError:

  ValueError: Error when checking input: expected conv2d_13_input to have shape (3, 150, 150) but got array with shape (150, 150, 3)

How can I fix this? Should I use numpy.resize or cv2.resize to change the dimensions to (3,150,150). If so, how would I resize it in the generator?

Here is my code:

train_datagen = ImageDataGenerator(
    rescale=1./255,
    shear_range=0.2,
    zoom_range=0.2)

test_datagen = ImageDataGenerator(rescale=1./255)

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(3, 150, 150),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

batch_size = 64

# this is a generator that will read pictures found in
# subfolers of 'data/train', and indefinitely generate
# batches of augmented image data
train_generator = train_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\train',  # this is the target directory
    target_size=(150, 150),  # all images will be resized to 150x150
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')  # since we use binary_crossentropy loss, we need binary labels

# this is a similar generator, for validation data
validation_generator = test_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\val',
    target_size=(150, 150),
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')

model.fit_generator(
    train_generator,
    steps_per_epoch=2000 // batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=800 // batch_size)
model.save_weights('first_try.h5')


texts: To extract the skills required for the job given the job description

texts:
Any recommendation on the libraries/methods to extract the skill set required for the job from the job description (raw text) ?
And also to extract the skillset from the resume ? The resume could be of any format raw text, pdf, docx.

I have attached the job description format.


texts: Any insights on gradient boosting algorithms for classification task with large class number?

texts: Recently I have been learning Gradient Boosting (GB) on multi-class classification. I searched Google Scholar for a while and found one paper on multi-class classification using GB 1. However, little literature on GB for classification task works with large class number, e.g., 1000 for ImageNet dataset. I am curious what is the technical difficulty in this scenario.


texts:
I'm struggling with a seemingly simple problem and could really use your help! I'm starting to learn about neural networks. I thought I would create a simple example to get started, using one dimensional inputs X and one dimensional outputs Y. When I first started with Y = 2X, the neural network was able to learn it very well over 1000 epochs. However, when I increased the complexity of the formula to below, it completely breaks apart (the MSE is very high).

X = random numbers between 0 and 100 (one-dimensional)

Y = (8(x^3)) + 5

I have tried adding more hidden layers and epochs without much success. Could anyone provide me with some guidance on where I'm going wrong in my method?

Much appreciated!

Code:

from keras.models import Sequential
from keras.layers import Dense
import numpy as np
# fix random seed for reproducibility
np.random.seed(7)

def generate_Y(X):
    return np.add(np.power(np.multiply(X, 2.0), 3.0), 5.0).T[0]

X = np.random.rand(100,1)
X = np.multiply(X, 100.0)
Y = generate_Y(X)

# create model
model = Sequential()
model.add(Dense(12, input_dim=1))
model.add(Dense(12))
model.add(Dense(12))
model.add(Dense(12))
model.add(Dense(1))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mse'])
# Fit the model
model.fit(X, Y, epochs=10000, batch_size=10)
# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

test_X = np.random.rand(10,1)
test_X = np.multiply(test_X, 500.0)
test_Y = generate_Y(test_X)
preds = model.predict(test_X)

print preds
diffs = np.subtract(preds.T[0], test_Y)
print diffs

MSE: mean_squared_error: 73360074604544.00%

Prediction vs actual diffs:
[ 4.97601229e+05 -1.60322447e+06 -7.85835700e+08 -7.00977203e+08
 -7.52094155e+08 -1.88780991e+08 -1.64643340e+08 -9.38229901e+08
 -2.02173897e+08  4.45327360e+05]


texts: How to see/change learning rate in Keras LSTM?

texts: I see in some question/answers that ask to decrease the learning rate. But I don't know how can I see and change the learning rate of LSTM model in Keras library?


texts:
There is a few dense layers, 2-layer LSTM network and to Linear Layers in the end.
Even after I made a network very small, all GPU memory (8 GB) got consumed in a few epochs.
I understand that Apollo optimizer is quasi-Newton and attempts to approximate second derivative, but still - why memory consumption grows with every epoch?


texts:
I have a set of curves in 2D space each expressed as a set of (sampled) data points.
Each set has more or less the same number of items - eventually I guess I’ll use binning to make sure the number of points is the same (say 50) if that can help.
I would like to cluster the curves in N groups. Computing N should be part of the solution.
Possible translations on the first dimension are irrelevant.
I have a k-means implementation available.
I was thinking to transform the problem into a 100-dimensional space (50x2) where the samples of each curve become features.
Could this approach work?
Is there a better one, either using k-means or a different algorithm?


texts: Is Python a viable language to do statistical analysis in?

texts:
I originally came from R, but Python seems to be the more common language these days.  Ideally, I would do all my coding in Python as the syntax is easier and I've had more real life experience using it - and switching back and forth is a pain.
Out side of ML type stuff, all of the statistical analysis I've done have been in R - like regressions, time series, ANOVA, logistic regression etc.  I have never really done that type of stuff in Python.  However, I am trying to create a bunch of code templates for myself, and before I start, I would like to know if Python is deep enough to completely replace R as my language of choice.  I eventually do plan on moving more towards ML, and I know Python can do that, and eventually I would imagine I have to go to a more base language like C++.
Anyone know what are the limitations of Python when it comes to statistical analysis or has as link to the pros and cons of using R vs. Python as the main language for statistical analysis?


texts:
If you have a differentiable function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ that is $\beta$-smooth (for all $v$ and all $w$, you have $\|\nabla f(v)-\nabla f(w)\| \leq \beta \|v-w\|$), how can you show the equation below?

$f(v) \leq f(w) + \langle\nabla f(w), v-w \rangle + \frac{\beta}{2}\|v-w\|^2$


texts: Are there deduplication algorithms that do not work on a metric space?

texts:
Recently I got interested in the process of data cleansing and specifically in record linkage.

Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some lesser degree also about machine learning methods. It struck me that the key part of all algorithm basically introduce a metric space. Through the metric space every two data points can be assigned a distance. The distance is then basically a measure of how close these two data points are related to another.

However I do wonder, if there were not also different kinds of algorithms that do not work on this principle?


texts:
I'm reviewing the Rainbow paper and I'm not sure I understand how they can use DQN with multi-step learning, without doing any correction to account for off-policiness.

So. I understand how you can use 1-step update off-policy: the reward for a single transition doesn’t depend on the current policy, so you can reuse this experience in the future.

I also understand the point of n-step updates: it’s a trade-off between having high biais with 1-step update (as you get only limited information from a single transition) and having high variance with n-step update (as in that case both the policy and the environment can be stochastic, so you end up adding n random variables together).

What I do not get is how you can use n-step return off-policy, which is what the Rainbow DQN seems to do. With n-step returns you are considering trajectories, and you can’t assume that these trajectories would have been taken if the agent was using the current policy.

If I understood correctly, in the case of policy gradient this is dealt with using importance sampling, which will reduce the impact of policies which are further away from the current one.

But I don’t see the equivalent of this for multi-step DQN?


texts:
I am building a classification model of heart disease from 12-lead ECG data.
I would like to implement Grad-CAM (Paper) to visualize the results.
Any way to implement Grad-CAM on time series data?


texts:
I am currently working on an object detection project in which I amtrying to detect very small objects 50x50 object in a 2k image.
EfficientDet produced a very low result if I just put the raw annotations and start training.
So my question is: Is there any pre process technique, apart from augmentation, that would help improving the model?


texts: Extracting sections from document based on list of keywords - Python

texts:
I am new to NLP and I would like to ask how can I extract sentences from the text based on keywords that I have using Python. I created a list of keywords which will be used to extract sentences from the document.

If this will be a simple tokenization problem in which you will loop the list through the tokens, how can I capture synonyms or related words?

For example:

Keyword: Internal business

Sentence: You can only use this software for your business only.

Keyword: Confidentiality

Sentence: Information will be kept as secure as possible.

I actually implemented text categorization using TF-IDF, but with small dataset and large number of keywords. I don't think this will work to.
Thanks in advance.

Is it possible to apply pre-trained models like word2vec?

Is it also possible to create a custom model that will fit my concerns?


texts: Can Micro-Average Roc Auc Score be larger than Class Roc Auc Scores

texts:
I'm working with an imbalanced data set. There are 11567 negative and 3737 positive samples in train data. There are 2892 negative and 935 positive samples in validation data. It is a binary classification problem and I used Micro and Macro averaged ROC for evaluation of my model. However, I noticed that Micro averaged Roc-Auc score is higher than Class specific Roc-Auc scores. That doesn't make sense to me.

As you can see on the plot, micro averaged roc-auc score is higher for all points. If it is possible can you explain the reason behind that ? I used that sklearn-link and I converted it for binary classification (y-true -> one hot representation). I also added my code on below.
xgboost_model = XGBClassifier(n_estimators= 450,max_depth= 5,min_child_weight=2)
xgboost_model.fit(X_train,y_train)

yy_true,yy_pred = yy_val, xgboost_model.predict_proba(Xx_val)# .predict_proba gives probability for each class

# Compute ROC curve and ROC area for each class
y_test = flat(yy_true) # Convert labels to one hot encoded version
y_score = yy_pred

n_classes=2
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

# Compute micro-average ROC curve and ROC area
fpr["micro"], tpr["micro"], _ = roc_curve(y_test.ravel(), y_score.ravel())
roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])

# First aggregate all false positive rates
all_fpr = np.unique(np.concatenate([fpr[i] for i in range(n_classes)]))

# Then interpolate all ROC curves at this points
mean_tpr = np.zeros_like(all_fpr)
for i in range(n_classes):
    mean_tpr += interp(all_fpr, fpr[i], tpr[i])

# Finally average it and compute AUC
mean_tpr /= n_classes

fpr["macro"] = all_fpr
tpr["macro"] = mean_tpr
roc_auc["macro"] = auc(fpr["macro"], tpr["macro"])

# Plot all ROC curves
plt.figure()
plt.plot(fpr["micro"], tpr["micro"],
         label='micro-average ROC curve (area = {0:0.2f})'
               ''.format(roc_auc["micro"]),
         color='deeppink', linestyle=':', linewidth=2)

plt.plot(fpr["macro"], tpr["macro"],
         label='macro-average ROC curve (area = {0:0.2f})'
               ''.format(roc_auc["macro"]),
         color='navy', linestyle=':', linewidth=2)

colors = cycle(['aqua', 'darkorange', 'cornflowerblue'])
for i, color in zip(range(n_classes), colors):
    plt.plot(fpr[i], tpr[i], color=color, lw=lw,
             label='ROC curve of class {0} (area = {1:0.2f})'
             ''.format(i, roc_auc[i]))

plt.plot([0, 1], [0, 1], 'k--', lw=lw)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve for nonSampling Training Data')
plt.legend(loc="lower right")
plt.savefig('nonsample.png', format='png', dpi=600)

plt.show()


texts: How to check quality of latent space like in β-VAE article?

texts:
There is a nice plot in the β-VAE article that shows quality of latent space code:

Is there a general way to visualize or analyze latent space code dimensions so that is would be clear if they are too entangled or a mess?

UPD:

  The dataset consists of Gaussian blobs presented in various locations on a black canvas. Top row: original images. Second row: the corresponding reconstructions. Remaining rows: latent traversals ordered by their average KL divergence with the prior (high to low). To generate the traversals, we initialise the latent representation by inferring it from a seed image (left data sample), then traverse a single latent dimension (in [−3, 3]), whilst holding the remaining latent dimensions fixed, and plot the resulting reconstruction. Heatmaps show the 2D position tuning of each latent unit, corresponding to the inferred mean values for each latent for given each possible 2D location of the blob (with peak blue, -3; white, 0; peak red, 3).

This means that to generate such heatmaps one should be able to smoothly move the blob while knowing it position. So one should actually know what is the best latent encoding. And the plot is actually a comparison with this best latent encoding.

I'm curious if there is a way to measure or plot "something" that would help to understand the quality of the latent code when I don't know what is the best latent code.


texts: How do I tell my model about the sample size of input statistics?

texts:
Let's say I am trying to predict whether a cat will be adopted, and I have found the ratio purred_when_petted / total_times_petted aka p / n to have predictive power. However, for some cats I have much more data than others. For example:

Cat A:
    p: 1
    n: 1

Cat B:
    p: 9
    n: 10

I would like my neural network to consider that there is more data for Cat B and possible consider that it is more likely to be adopted. How should I tell it about both the ratio and the sample size?

Here are some ideas I have so far:

Use ratio as input feature and don't worry about sample size
Use ratio and sample size as separate input features
Use $p$ and $n$ as separate input features, and let the NN do what it wants with them.
Use an average of the cat statistic and the average population statistic over all $M$ cats weighted by number of samples:
$$
x_i = \beta  * (\frac{p_i}{n_i}) + (1-\beta) * (\frac{1}{M}\sum_{j=1}^{M}\frac{p_j}{n_j})
$$
where $\beta \in [0,1]$ is an increasing function of sample size $n_i$.
Use a Binomial proportion confidence interval

Is there a standard way to do this? I am most interested in neural networks but also open to advice relevant to any other type of model.

Note that I also have other categorical and numeric features such as color and age that also need to be fed in to my model.


texts: On regression to minimize log distance rather than distance

texts:
Suppose I have a lot of points $ x_i \in \mathbb{R}^N $ with corresponding non-negative labels $ y_i \in \mathbb{R} $ and I want to do regression and make a prediction on some new datapoint $ x^* \in \mathbb{R}^N $ for which I don't have a label.   Is there a name for the procedure of choosing a parametric model $ f_\theta : \mathbb{R}^N \rightarrow \mathbb{R} $ so as to minimize the cost function $ \sum_i {|\log(f_\theta(x_i)) - \log(y_i)|^2 } $ rather than $ \sum_i{|f_\theta(x_i) - y_i|^2} $?  It seems that minimizing the difference between logs has some nice properties, and I'm surprised I didn't see this discussed in Bishop's machine learning book, for example.

I thought of this when I was considering a house pricing problem, where I figured I cared more about the percentage by which I was wrong than the pure difference.  After all, in my application (and I'm sure many others like it), being wrong by \$50,000 is terrible for a \$60,000 home, but it's okay for a $2.5M home.

Any data science veterans reading this who have used a cost function like the one I suggested above with logs, or who can tell me what it's called (if it has a formal name)?


texts: Getting sentence embeddings with sentence_transformers

texts:
I have a text column in my data frame which contains paragraph(s) having multiple and variable sentences in each instance/example/row of the dataframe. Then, I created the sentence tokens of that paragraph using sent_tokenizer of nltk and put it into another column.
So my data frame looks like this:
index       text                                              class

0           ["Hello i live in berlin", 'I'm xxx']                                                          1
1           ["My name is xx", "I have a cat", "Love is life"]                                              0

now when I'm using:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
sentences = df['text']
sentences = sentences.tolist()
embeddings = model.encode(sentences)

I'm getting:
TypeError: expected string or bytes-like object

The encode  method is not taking a list of list of sentences as an argument.


texts: In datasets, why don't we represent nominal values that are part of a scale with numbers?

texts:
I am trying to pre-process a small dataset. I don't understand why I am not supposed to do the thing I explained below:

For example, say we have an attribute that describes the temperature of the weather in a set of 3 nominal values: Hot, mild and cold. I understand these definitions may have derived from numerical values while summarising.

But why would we summarise such values that are on a scale, and lose the scale in the process?

Would it not help to have the algorithm(any classification algorithm) realise that the difference between hot and cold is twofold of the difference between hot and mild by representing hot, mild and cold as integers 1, 2 and 3 respectively?


texts: Gaussian Process Classifier and specifying kernel

texts:
I am using scikitlearn's gaussian process classifier and either I don't think I understand how the kernel is used (more likely), or there is an error in the module (less likely). In short, the behavior is different when I specify the kernel and when I don't, even in cases where I specify the default kernel. Furthermore, when I fit a model to data not specifying the kernel, I cannot access the fit results, but "scores" for the fit appear reasonable. Here is an example of the problem using the commonly available learning iris data:

import numpy as np
from sklearn import datasets
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF

# import some data to play with
iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
y = np.array(iris.target, dtype=int)

gpc_default =  GaussianProcessClassifier().fit(X, y)
gpc_default.score(X,y) #result is 0.82

kernel = 1.0 * RBF([1.0])
#my goal here is to specify the kernel explicitly, but use what I thought was the default kernel.
gpc_specifickernel =  GaussianProcessClassifier(kernel=kernel).fit(X, y)
gpc_specifickernel.score(X,y) #result is 0.8266666667

Two questions:

1) Why are these scores different? The inputs are the same.... right?

2) gpc_specifickernel.kernel_ has the best-fitting kernel values. In this case those are: CompoundKernel(9.18, 0.913, 2.5, -0.26, 2.98, 1). However, gpc_default.kernel_ holds nothing interesting. More precisely, its value is CompoundKernel(). Why the difference? Also, when I use the default kernel, how to I access the best fitting parameters?


texts: help finding research discussion on HTS classification

texts:
My question is about the theory of this problem, and not necessarily syntax.

I'm wondering if anyone here has experience with automating HTS (Harmonized Tax Schedule) classifications, specifically training a classifier to predict the HTS code that should be assigned to a product? There are rules for these designations that are stored as descriptive text data that 'explain' the product. This seems (to me) to make predicting how a part will be classified very difficult.

For context, it would seem that NLP is heavily involved in this process, but with the high cardinality involved in these descriptions and the high amount of overlap from one description to another, are there any best practices for success in making such predictions? Similarly, I'm having a hard time finding any public research on the matter. It appears that almost all advancement in this focus was developed behind the closed doors of companies. I understand why they'd make that decision, but I am a bit surprised how little public discussion exists on the topic considering how much business it affects across the globe.

things I am considering/researching:

Word vectorizing (Word2Vec): I got the idea from another user here, but given that it is typical to train a vectorizer with a large public corpus, I am not sure how that would affect the vectoring process, as the products I am classifying are domain specific.

n-gram: I suppose using 2, 3, or 4 would be sufficient for classification, given that most descriptions that I can see are succinct. But as I said, a lot of these descriptions will have overlap, for example:

    HTS CODE             HTS Description
------------------     ------------------
11111 = 'football'     An inflated ball (used for kicking)
11112 = 'basketball'   An inflated ball (used for dribbling)
11113 = 'ball'         An inflated ball

In this case (if my understanding is correct), n-gram 6 would be needed for the first two products but only n-gram 3 is needed for the third product. Is the n-gram value something we can 'soft code' to adapt to each record or will I need to somehow find the appropriate number for n?

Thanks


texts: Extracting list of locations from text using R

texts:
I have a string containing many words [not sentences], I want to know how I can extract all the words that correspond to a location in that string for example:

text<-c("China","Japan","perspective","United Kingdom","formatting","clear","India","Sudan","United States of America","Bagel","Mongolian",...)

The output should be:

 > China, Japan, United Kingdom, Mongolian

something of the type. Basically I am looking at extracting locative information from random text.
This is a very general problem I am looking for guidance on how to model my solution, is there any dataset or something I can use to compare or extract information from. I don't want to carry out word by word comparison.

I have looked up OpenNLP but I am not sure how to use it's location-models for carrying out Named Entity Recognition in R. In the above example there are only countries but I would like to identify other places, such as provinces, states, counties, cities, etc. as well.
I am new to machine learning and R-programming, any guidance is greatly appreciated.


texts:
Today I read an interesting post to drop RNNs for sequential models.

However, the post, unfortunately, didn't go into much detail on how one would study the attention based models and start experimenting with them.

The only usefull link was to this paper which uses convolutional networks.

The paper is very packed and dense, making it hard for me to understand, and I couldn't find any books on implementation of these attention models.

Does anyone have any suggestions?


texts: What does it mean for an activation function to be "saturated/non-saturated"?

texts:
For context, in this paper

  Several RNN variants such as the long short-term memory (LSTM) [10,
  18] and the gated recurrent unit (GRU) [5] have been proposed to
  address the gradient problems. However, the use of the hyperbolic tan-
  gent and the sigmoid functions as the activation function in these
  variants results in gradient decay over layers. Conse- quently,
  construction and training of a deep LSTM or GRU based RNN network is
  practically difficult. By contrast, ex- isting CNNs using
  non-saturated activation function such as relu can be stacked into a very deep network (e.g. over 20 layers using the basic convolutional
  layers and over 100 lay- ers with residual connection


texts: FP16, FP32 - what is it all about? or is it just Bitsize for Float-Values (Python)

texts:
What is it all about FP16, FP32 in Python? My potential Business Partner and I are building a Deep Learning Setup for working with time series. He came up with "FP16 and FP32" while finding a GPU. It looks like he's talking about Floating Point values in 16 vs 32bit. (Our data points look like this: "5989.12345", so I'm pretty sure 16bit ain't enough.)

Is FP16 a special technique GPUs use to improve performance or is it just a fancy term for using 16-bit float values instead of 32 standard floats?