texts: Variational Autoencoder (VAE) latent features

texts:
I'm new to DL and I'm working on VAE for biomedical images. I need to extract relevant features from ct scan. So I created first an autoencoder and after a VAE. My doubt is that I don't know from which layer I can extract feautures. My personal idea is to use features extracted by layers that compute the mean and variance (before reparameterization trick), but I think that also the layer before these is suitable for the purpose. I left here code of the encoder part:

class Sampling(tf.keras.layers.Layer):
    """Uses (z_mean, z_log_var) to sample z, the vector encoding a digit."""
    def call(self, inputs):
        z_mean, z_log_var = inputs
        batch = tf.shape(z_mean)[0]
        dim = tf.shape(z_mean)[1]
        epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
        return z_mean + tf.exp(0.5 * z_log_var) * epsilon

def Encoder():
    inp = tf.keras.Input(shape=(32,256,256,1)) # prima era 64

    #enc = tf.keras.layers.Conv3D(8, (2,2,2), activation = 'relu', padding = 'same')(inp)
    #enc = tf.keras.layers.MaxPooling3D((2,2,2), padding = 'same')(enc)

    enc = tf.keras.layers.Conv3D(16, (2,2,2), activation = 'relu', padding = 'same')(inp)
    enc = tf.keras.layers.MaxPooling3D((2,2,2), padding = 'same')(enc)

    enc = tf.keras.layers.Conv3D(32, (2,2,2), activation = 'relu', padding = 'same')(enc)
    enc = tf.keras.layers.MaxPooling3D((2,2,2), padding = 'same')(enc)

    enc = tf.keras.layers.Conv3D(64, (2,2,2), activation = 'relu', padding = 'same')(enc)
    enc = tf.keras.layers.MaxPooling3D((2,2,2), padding = 'same') (enc)

    enc = tf.keras.layers.Conv3D(32, (2,2,2), activation = 'relu', padding = 'same')(enc)
    enc = tf.keras.layers.MaxPooling3D((2,2,2), padding = 'same') (enc)
    #enc = tf.keras.layers.Flatten()(enc)
    enc = tf.keras.layers.Conv3D(16, (2,2,2), activation = 'relu', padding = 'same')(enc)
    enc = tf.keras.layers.MaxPooling3D((2,2,2), padding = 'same') (enc)
    '''
    # conv 2D
    code = tf.keras.layers.Reshape((8,8,96)) (enc)
    code = tf.keras.layers.Conv2D(96,(2,2), activation = 'relu', padding = 'same')(code)
    code = tf.keras.layers.MaxPooling2D((2,2), padding = 'same') (code)
    '''

    # latentent code vae
    latent_code = tf.keras.layers.Flatten()(enc)
    latent_code = tf.keras.layers.Dense(256, activation='relu')(latent_code)
    latent_mu = tf.keras.layers.Dense(32, activation='relu')(latent_code) # èprima era 10
    latent_sigma = tf.keras.layers.Dense(32, activation='relu')(latent_code) # prima era 10
    # Reparameterization trick
    #z = tf.keras.layers.Lambda(sample_z, output_shape=(128,), name='z')([latent_mu, latent_sigma])
    z = Sampling()([latent_mu, latent_sigma])
    encoder = tf.keras.Model(inp, [latent_mu, latent_sigma, z ], name = 'encoder')

    #encoder = tf.keras.Model(inp, enc)#[latent_mu, latent_sigma, z ], name = 'encoder')
    return encoder

```


texts:
I'm new to ML world and been reading about ML and TensorFlow.
My goal is to read the following example in real time with Android phone:

So I tried firebase ML OCR and it works really good, it reads the complete value but it does not read the decimal point and also reads a lot of the surrounding text.
So my idea is that I should first detect black and red bounding boxes and then detect individual numbers inside

is this the right way to go? How would I accomplish this?
Also how do you use two kinds of a model, one to extract a part of the image (black and red bounding areas) and then pass them to OCR model?
What about last digit which can always be in between two numbers (example: 1 and 2)?


texts: What open-source books (or other materials) provide a relatively thorough overview of data science?

texts: As a researcher and instructor, I'm looking for open-source books (or similar materials) that provide a relatively thorough overview of data science from an applied perspective. To be clear, I'm especially interested in a thorough overview that provides material suitable for a college-level course, not particular pieces or papers.


texts: What do the symbols in this image mean in relation to autoencoding? (which concepts do they represent and why?)

texts: I'm completely new to neural nets, and am trying to get a grasp of autoencoding. I understand the function of the decoder and encoder, but what does the arg min section and beyond mean? Why is it equal to both phi and psi? Feel free to be abundant in your explanation. Thanks!


texts:
How does feature selection impact outlier detection and also, removing outliers impact feature selection?
It could be a basic question. However, just to know the boundaries, I asked. Thanks in advance.
I have gone through the following:Feature selection and outlier order


texts: What is the impact of changing image sources on an image recognition?

texts:
I have a fairly general question pertaining to an image recognition ML model. I’ve recently developed an image recognition model using a single camera collecting more than 5000 images and then trained/developed the model.
My question is if I change the camera source (also changing resolution) would this impact the accuracy of the model? The scripts we’ve developed would still resize the image to the same size as the training set.
There is some discussion within our team about this and I’m hoping someone with a little more expertise in image recognition can add some insight.
Thanks!


texts:
I am a beginner in KNIME and I need to predict attribute I have some questions :

1- How can I choose the most related attribute to predict the target attribute?

2- can I choose the attributes randomly then compare the accuracy?

3- can I use nominal attributes or only numerical attributes?


texts: grid search result max_features = 'sqrt' in random forest - how to understand

texts:
I did a grid search at random forest params. the result of
print(randomforestreg.best_params_)

The result is =
{'max_depth': 28, 'n_estimators': 500 ',max_features': 'sqrt', 'min_samples_split': 2,  'min_samples_leaf': 1'}
The Random Forest documentation:
If “auto”, then max_features=sqrt(n_features).
If “sqrt”, then max_features=sqrt(n_features) (same as “auto”).
So 'sqrt' is like max_features=sqrt(n_features) --> same as 'auto'.
--> the number of features with which tree is build. I dont understand this. When I have 19 columns(features) and 100.000 rows, what is now the answer of the question; "what is the best value for max_features?". Is it the squared value of n features = 19^2 = 361?


texts: What could be some Classification techniques to classify a tree of webpages given the category of each webpage

texts:
I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need guidance on how can I combine those node categories to get the overall category of the tree. What classification techniques could be used ?
As of now I know about usage of markov chains for this task from this research paper, Web Site Mining. I would be grateful if i could get some more ideas about how to perform the task.


texts: Why does a simpler model performs better than a complicated one?

texts:
This has happened to me, a complicated model couldn't solve the problem when a simpler one solved it in a few epochs. How is that? I believed that a more complicated model means more number of parameters and more number of parameters means a higher capability to solve a problem.
I have heard that people say, A simpler Model can perform well because it has a low variance. But, what does that even mean?


texts: How to get a output of a hidden layer of a single-layer LSTM

texts:
How can get the hidden layer outputs in a simple one-layer lstm?

cat("Building Model\n")

model <- keras_model_sequential() %>%
  layer_lstm(units = 64, dropout = 0.2, input_shape = c(seqlength, length(chars))) %>%
  layer_dense(units = length(chars), activation = "softmax") %>%

  compile(loss = 'categorical_crossentropy',
          optimizer = optimizer_sgd(lr = 0.001,
                                    decay = 1e-6,
                                    momentum = 0.9,
                                    nesterov = T),
          metrics = c('accuracy'))

summary(model)

cat("Training \n")

history <- model %>%
  fit(train,
      trainLabels,
      epochs = 6,
      batch_size = 16,
      validation_split = 0.2)

I found this guid but I don't know how to fit it to this simple model and what is data.

model <- ...  # create the original model

layer_name <- 'my_layer' intermediate_layer_model <- keras_model(inputs =
                  model$input,outputs = get_layer(model, layer_name)$output)
intermediate_output <- predict(intermediate_layer_model, data)

Can anybody give a sample of this?


texts: Skip gram Word2Vec model, neural network implementation

texts:
I have referred and used materials from these - cs224n stanford.

Following is the algorithm provided in the stanford course notes for training a skip gram model. Here |V| is the vocab size, matrix V is hidden layer matrix that would select a particular input word embedding for center word.

I am assuming that hidden layer matrix V is of dimensions - |V| x d. (d - feature lengths), and input one hot vector of length |V| x 1. The result of hidden layer then would be of (1 x d) dimensions.

Problem - In step 4, it is said to generate 2m score vectors. To me it seems the dimension of the score vectors would be 1 x 1 each, as they would be a product of result from the hidden layer of dimension (1 x d), and output word vector matrix U, of dimension |V| x d. The overall result should be of |V| x 1, And each word score will be of 1 x 1 dimension (column vector).

Then how can we compare each of them to one hot vectors as asked in step 6 (in doing so, won't it make probability of a particular word ideally 1, and hence all other word probabilities should be zero)? Also, what are the actual output one hot vectors? And is the probability of each word in the 2m window equal to all other words there?


texts:
I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications:

A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is made to run.

This means that every task of a Yarn Application are run is the same datanode?


texts:
I'm working on below reinforcement learning problem:
I have bottle of fix capacity (say 5 liters). At the bottom of bottle there is cock to remove
water. The distribution of removal of water is not fixed. we can remove any amount of water from bottle, i.e. any continuous value between [0, 5].

At the top of the bottle one tap is mounted to fill water in the bottle.  RL agent can fill [0, 1, 2, 3, 4] liters in the bottle. Initial bottle level is any value between [0, 5].

I want to train the agent in this environment to get optimal sequence of
actions such that bottle will not get empty and overflow which implies continuous supply of water demand.

Action space = [0, 1, 2, 3, 4] Discrete Space

Observation Space = [0, Capacity of Bottle] i.e. [0, 5] Continuous Space

Reward logic = if bottle empty due to action give negative rewards; if bottle overflow due to action give negative rewards

I have decided to use python to create an environment.

I'm new to RL. I'm not aware about for which condition I have to give the rewards. Is my reward logic is correct or I have to change it?


texts: decision trees on mix of categorical and real value parameters

texts:
I have about 3,000,000 samples and each sample is described by a list of size about 20. Some elements in this list are categorical, for example name of cities, day of week, etc. (some categories have a large number of options, for example one category is url with more than 700,000 unique elements in my dataset!). Also some elements have real values, for example for time of day.

My data is labelled (2 categories,) and I need to train a classifier for test data. I am inclined towards decision tree or random forest since they seem to be a good choice for this type of problem.

Now my questions are:

1) How do I pre-process categorical data? one-hot-encoding seem to be the right choice but given that some of my categories have huge number of possible values, one hot encoder will produce very long words! am I correct?

2) How do I combine data from different categories? For example data from category 'cities' with data from 'urls', since they have different lengths. Do I simply concatenate them?

3) How can I combine categorical data with real valued data, for example 'name of cities' with 'time of day' to produce one matrix that can then be passed to decision tree classifier?

4) Are there any special normalisation, etc. that I have to do before passing data to classifier?

I plan to use python and Scikit for this task.

Many thanks for your help.


texts:
I have a data frame consisting of some continuous data features. I did a kde plot of the features using seaborn kdeplot functionality which gave me a plot as shown below :

How do I interpret this visualization in order to check for things like skew in the data points, etc.?


texts: Zero Padding in Convolutional neural network

texts: We use Convolutional neural network because it by design learns features that generalize over spatial location , so when using conv operation it reduce image size and that what we hope to have so we can have more simplicity in the following layers , but when doing zero padding the image return to its original size , so what is the benefit of doing this ? the same problem return back since image size will be still large . Also, CNN works well for classify images; so how it can be used in games to detect best next move, I mean how it decide the Q values of the actions from the image itself even though understanding image doesn't give any indication of the next move  ?


texts: Keras: Misunderstanding the Conv2D's param "filters"?

texts:
From https://keras.io/layers/convolutional/ (Conv2D):

keras.layers.Conv2D(filters, kernel_size, strides=(1, 1), padding='valid',
                    data_format=None, dilation_rate=(1, 1), activation=None,
                    use_bias=True, kernel_initializer='glorot_uniform',
                    bias_initializer='zeros', kernel_regularizer=None,
                    bias_regularizer=None, activity_regularizer=None,
                    kernel_constraint=None, bias_constraint=None)

  filters: Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

Why is it possible to indicate the number of features maps (results of the scalar product between the kernel and the input), since we specify the kernel's size and the input?

Indeed, if we know the kernel's size (width and height) and the input's size (width and height), thus we know the number of features maps.


texts: What algorithms can be used to derive matching rules between known matches in datasets?

texts:
Lets say I have two datasets with different column names except for a unique ID key

Table 1 CSV

first_name,middle_name,last_name,uno,id
John,D,Smith,1,1
John,C,Smith,1,2
John,B,Doe,1,3
Suzy,C,Q,1,4

Table 2 CSV

fname,mname,lname,one,id
John,D,Smith,1,1
John,C,Smith,1,2
John,B,Doe,1,3
Suzy,C,Q,1,4

John D Smith is user ID #1 and is in both tables.

Is there a pre-built algorithm, package or tool that can do the following.

Join across tables where id is the same
For known matches, try to identify what rules could have been used to match the two records together.
Test hypothesis, like "fname and first_name are the same, is that enough to produce the target 'id' variable? Let me check other data. No. What about fname + lname?
Test if assertions hold true against other known matches.

End output would be

table 1 (first_name, middle_name, lastname) are the best join against

table 2 (fname, mname, lname)


texts: ARIMA d Parameter and Explicit Differencing

texts: Can I use only d parameter for ARIMA instead of applying differencing to data before training and applying inverse transform to forecasts in order to get them into original scale? Do libraries like statsmodels do this implicitly?


texts: How to add additional information in a neural network other than putting as input data?

texts: I am trying to use a neural network to predict final concrete strength. I have used material properties of concrete as input data. Now if I want to add some additional information like early age strength, is there any way to put them in the hidden layer? I mean other than using as input data?


texts: Loss Function for Probability Regression

texts:
I'm trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in the context of a binary classification problem where the labels are $\{0, 1\}$, but in my case I have an actual probability as the target. Is one of these options clearly best, or maybe are they all valid with just minor differences around the extreme 0/1 regions?

Assuming $x$ is the output of the final layer of my model.

Cross Entropy:
$target * -log(sigmoid(x)) + (1 - target) * -log(1 - sigmoid(x))$

Mean Squared Error with Sigmoid:
$(sigmoid(x) - target)^2$

Mean Squared Error with Clamp:
$(x - target)^2$

When I use the output I clamp the values between $[0, 1]$.


texts: what's the SOTA for medical imagery classification for diagnostic purposes?

texts:
Many medical image datasets - Ultrasound for example - have a time component that is quite powerful. But some of the papers I have read use ConvNets and seem to ignore temporality.

Have there been any advances in using LSTMs and/or 3D ConvNets for medical imagery classification to leverage temporality? Can anyone shed some light on recent developments?

[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5116541/


texts: High correlation between the independent and dependent variables but low performance of regression model

texts:
I have a dataset of 4900 rows and 2060 feature. I calculated the correlation using kendall method between the dependent and independent features, and found out that 5 of these features are having a correlation with the output(dependent) variable very high correlation, the highest independent feature has a correlation of .836, and the fifth feature has correlation of  .736. So I had high hopes that my regressor model will fit well.
I split the data to 80% training data and 20% testing data. However, I got overfitting and the training abs error itself is still not so good.
Is there any reason to have a bad fitting despite the very high correlation values? or is it because they are only 5 features out of 2066 features? or is it because the small number of the data rows?


texts:
I am not able to distinguish the outliers - When to go with std. dev and When do we need to go with Median.
My understanding on std. dev. is - if the data is away from mean by more than 2 std dev. we consider that as outlier.
Similarly for Median, we say that any data that is not in-between q1 and q3, we say again that as outlier.

So am confused which one to choose.

Can you guys help me understand.


texts: Using a KMeans to classify URLs: validate the number of cluster and visualise

texts:
I'm relatively new to the NLP and DataScience, so apologies for omission or things like this.

I've been trying to use the KMeans to classify a list of 1000 of unique URLs containing several keywords that span across the taxonomy of a site.
The intent is to understand how the algorithm would see these URLs fitting the best and - up to a degree - validate what the taxonomy exercise completed by the specialist.

The first step was to vectorise the URLs, obtaining an array of 1416 features.

I could run the KMeans without no problem, but here the first challenge: how can I verify the number of clusters. As far as I understand could be done with the Elbow algorithm, and the right number should be where the curve happens.
In my case, it seems there is not a curve but a progressive degradation of the number of items falling in each group. See image for a range 1 to 40. If I have to see a curve, I'd say this is 17.

Assuming for a moment the number is correct, my next challenge is, how can I visualise the clusters and how can I understand about any outliers?

I thought that using a scatterPlot would have been a good idea, but with my data set this doesn't seem an option. I still have an array of 1464 feature each row vs the 2 data points needed.

Any idea on how I can proceed?


texts: is final fit with X,y or X_train , y_train?

texts:
I split the dataset with

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

and the fit

from sklearn.metrics import log_loss
clf.fit(X_train, y_train)
clf_probs = clf.predict_proba(X_test)
score = log_loss(y_test, clf_probs)
print(score)

is final submission with

clf.fit(X,y) or clf.fit(X_train,y_train)???


texts: What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?


texts: Inference of a function f(x,y) where inputs have different dimensionality

texts:
I am interested in inferring the value of a high dimensional unknown function f which accepts two inputs, let's say x and y, with very different dimensionality (for example x could be a vector with hundreds of entries, whereas y could be a vector with less than 10 components). Of course, my training dataset is composed of many measurements of x, y and the corresponding values of the function. By the way, x, y, and f take real values, so this is a regression problem.

My first idea to tackle this problem would be to train a multi-layer perception in which the input of the network is a vector where x and y are stacked over each other and the output is the corresponding value of the function f. However, such an approach seems sub-optimal to me, because intuitively the low dimensional input (i.e. y) would end up playing a negligible role in the approximation of f. In other words, I already know from the nature of my problem that every single value in y is more relevant to the approximation of f than any of the many components in x. My question then is: what would be a better algorithm to tackle this regression problem with multiple inputs with different dimensionality?

I apologize if the question is trivial, but I am not a data scientist and I could not find any good answers online.


texts: How can I handle missing categorical data that has significance?

texts:
I have a data set that is highly categorical and has a lot of missing values. For instance:

i | A_foo | A_bar | A_baz | outcome
--+-------+-------+-------+--------
0 |  nan  |  nan  |  nan  |   1
1 |   0   |   1   |   0   |   1
2 |  nan  |  nan  |  nan  |   0
3 |   1   |   0   |   0   |   0

The problem is that 1 and 0 have a different meaning that nan. I don't want to impute the data because assigning values of 0 or 1 will bias my data, but many machine learning algorithms will not work on a dataset with missing values. How can I handle this?


texts:
I have a dataset consisting of a bunch of food reviews. I'm trying to build a model that determines if a given review talks about a food safety issue. The dataset has a lot of features, but relevant to my goal there are only two features of importance --  food reviews and their corresponding numerical ratings.
I'd love a pointer or two to figure out how to approach this problem.
As a first pass attempt, I'm thinking of pipelining all reviews into a sentiment analsyis filter and then look at the ones have keyword related to food safety (eg: "high cholesterol", "unsafe", etc)


texts: Same model/same data different results in Keras/TF

texts:
So first to mention I used for all models the identical dataset (just in different shapes).
I started with a binary classification based on text following the tutorial from https://keras.io/examples/nlp/text_classification_from_scratch/. The results are quite promising on my dataset at ~80%. Dataset were .txt files generated from the CSV.

Further I wanted to add additional structured data to increase the accuracy following this tutorial (https://www.tensorflow.org/tutorials/load_data/csv#mixed_data_types). But the results were quite bad at ~50%. I then adapted the layers from the first tutorial but the results are still bad. Dataset was CSV directly.

Now I removed all structured data from the CSV and just left the text in it but the results are not similar. Why? Dataset was CSV directly.

import pandas as pd
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
import numpy as np
from typing import Union
from tensorflow.python.keras.engine.keras_tensor import KerasTensor

titanic = pd.read_csv("Dataset.csv")
titanic.head()

titanic_features = titanic.copy()
titanic_labels = titanic_features.pop('survived')

inputs = {}

for name, column in titanic_features.items():
  dtype = column.dtype
  if dtype == object:
    dtype = tf.string
  else:
    dtype = tf.float32

  inputs[name] = tf.keras.Input(shape=(1,), name=name, dtype=dtype)

numeric_inputs = {name:input for name,input in inputs.items()
                  if input.dtype==tf.float32}

x = layers.Concatenate(name='ConcatNumeric')(list(numeric_inputs.values()))

norm = preprocessing.Normalization(name='PrepNormalization')

norm.adapt(np.array(titanic[numeric_inputs.keys()]))
all_numeric_inputs = norm(x)

# Not adding numeric values to dataset
preprocessed_inputs = []

# Model constants.
max_features = 20000
embedding_dim = 128
sequence_length = 500

# Now that we have our custom standardization, we can instantiate our text
# vectorization layer. We are using this layer to normalize, split, and map
# strings to integers, so we set our 'output_mode' to 'int'.
# Note that we're using the default split function,
# and the custom standardization defined above.
# We also set an explicit maximum sequence length, since the CNNs later in our
# model won't support ragged sequences.
vectorize_layer = TextVectorization(
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

for name, input in inputs.items():
  if input.dtype == tf.float32:
    continue

  x = vectorize_layer(input)
  x = layers.Embedding(max_features + 1, embedding_dim)(x)
  x = layers.Dropout(0.5)(x)
  x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
  x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
  x = layers.GlobalMaxPooling1D(name='GlobalMaxPooling')(x)
  x = layers.Dense(128, activation="relu")(x)
  x = layers.Dropout(0.5)(x)
  x = layers.Dense(1, activation="sigmoid", name="predictionasds")(x)
  preprocessed_inputs.append(x)

titanic_preprocessing = tf.keras.Model(inputs, preprocessed_inputs)

titanic_preprocessing.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])

titanic_features_dict = {name: np.array(value)
                         for name, value in titanic_features.items()}

titanic_preprocessing.fit(x=titanic_features_dict, y=titanic_labels, epochs=3)

tf.keras.utils.plot_model(model = titanic_preprocessing , rankdir="LR", dpi=72, show_shapes=True)


texts: Distance measure calculation addresses for record linking

texts:
At the moment we use different methods for record linking locations in different datasets.

Theoretically given two locations we can give a prediction on how well they match (are the same). This is not just based on address data (street, house number, zip, city, country, latitude, longitude) but also based on the name, type of establishment and other properties like phonenumber.
Since most features are prone to fuzzy errors (different spellings, writing styles, formatting, human entry error, null-values (absent)) this means we cannot do strict comparisons.

Using many business rules, basic distance calculations, etc.  we can clean & reformat the data so we can have a pretty good comparison and high accuracy on matches.

eg.

  Datlinq BV,
  Roer 266,
  2908MC, Capelle aan den IJssel

is the same as

  datlinq,
  266 Roer,
  capelle a/d yssel

but not as

  Ferro BV,
  Roer 266,
  2908MC, Capelle aan den IJssel

Our problem is many a scaling issue:

We need smart ways to reduce the dataset so we don't compare every location in dataset A to every location dataset B.

At this moment we do some pre-filtering (boxing) on latitude / longitude (if present)

However I was wondering if we could somehow generate a vector for each data location and us a basic distance calculation to find matches (or match candidates) in other datasets.

Or maybe another way to find very similar locations across huge datasets.

I'd love to use a machine learning model for this, but I am stumped how to generate numerical features out of a string address, location type & telephone number, without applying to many business rules.

So the actual question is:

Are there smart ways to do (parallel) record linking across big datasets (using Spark or so)
Is the approach of converting locations to vectors using a ml model and then using a distance calculation to find nearby points a good approach?
Is there terminology I should use to find more information about this problem? Are there papers / experiments?

-- Update 20170829 151213

So we are working on prefiltering datasets based on nearby locations, but unfortunately for many datasets we don't have reliable lat/longs. So I'm looking for a more general approach.

So based on the fact that we cannot positively resolve a nearby location without comparing textual addresses. A smart heuristic is something I'm looking for, but I want to incorporate weights to certain (fuzzy) features, eg. zipcodes should match more closely towards the beginning then towards the end, phonenumbers other way around. Streets and cities maybe spelled very differently including abbreviations.

Sure I can extract all these features and create business rules with arbitrary weights, but I'd rather have an unsupervised ML model learn the most important features based on positive and negative test/trainingsets of address matches.
I would also guess that weights will vary differently by country/language so maintaining these by hands seems a suboptimal solution.


texts: System Requirements to train a Image Recognition Neural Network

texts:
I have 6000 Images to be trained on a Neural Network.

My current PC Specs :-

  32GB RAM, i5 2 core Processor, Standard GPU (No work going on GPU),
  1TB Hard Disk

My Neural Network Specs :-

  3000 Epoch Size, 6 Batch Size

Process is getting killed with these specifications.

  How much Hardware is required to trained it well? What would be the
  ideal Batch Size and Epoch Size?


texts:
I'm trying to model a random threshold as a weight,the threshold should help the error to decrease, the weights are not random, they are 1. It's possible to change the threshold so that the error will be 0?

import numpy as np

# input dataset
X = np.array([[0, 0],
              [0, 1],
              [1, 0],
              [1, 1]])

# output dataset
y = np.array([[0, 0, 0, 1]]).T

syn0 = np.zeros((2, 1)) + 1

threshold = np.random.random_integers(-5, 5)

# forward propagation
l0 = X
l1 = np.dot(l0, syn0)

# how much did we miss?
l1_error = y - l1
print(l1_error)

This is what I have so far..!


texts: How do you visualize neural network architectures?

texts:
When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.

What are good / simple ways to visualize common architectures automatically?


texts: How do I efficiently load data from disk during training of deep learning models in pytorch?

texts:
I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this?
It seems like HDF5 is a common method that people accomplish this, and is what I tried first. However when using pytorch's dataloader class, this ran extremely slowly. I created my own iterator which ran faster, however the data is not randomized every batch. I'm trying to understand why the pytorch dataloader is running slowly and if there is something I can do about it.
Below is my code
First I defined a dataset class that takes in a filepath to an HDF5 dataset. My understanding of this code is that it reads from disk whenever getitem is called.
class My_H5Dataset(torch.utils.data.Dataset):

def __init__(self, file_path):
    super(My_H5Dataset, self).__init__()
    h5_file = h5py.File(file_path , 'r')
    self.features = h5_file['features']
    self.labels = h5_file['labels']
    self.index = h5_file['index']
    self.labels_values = h5_file['labels_values']
    self.index_values = h5_file['index_values']

def __getitem__(self, index):

    return (torch.from_numpy(self.features[index,:]).float(),
            torch.from_numpy(self.labels[index,:]).float(),
           torch.from_numpy(self.index[index,:]).float(),
           torch.from_numpy(np.array(self.labels_values[index])),
           torch.from_numpy(np.array(self.index_values[index])))

def __len__(self):
    return self.features.shape[0]

Then I simply pass this into a pytorch dataloader as follows
train_dataset = My_H5Dataset(hdf5_data_folder_train)
train_ms = MySampler(train_dataset)
trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size,
sampler=train_ms,num_workers=2)

My other method was to manually define an iterator. And this does run much faster. However I have not noticed a speed difference between the data stored in an HDD versus an SSD which makes me worried that there is a bottleneck somewhere that I am missing.
def hdf5_loader_generator(dataset, batch_size, as_tensor=True, n_samples = 10000):
    """Given an h5 path to a file that holds the arrays, returns a generator
    that can get certain data at a time."""
    stop = n_samples
    curr_index = start = 0
    while 1:
        stop_index = min([curr_index + batch_size, stop])
        ft, sp, index, sp_values, index_values = dataset[curr_index:stop_index]
        curr_index += batch_size
        if curr_index >= stop:
            curr_index = start
            continue

        yield ft, sp, index, sp_values, index_values

def hdf5_data_iterator(dataset, batch_size, as_tensor=True):
    return iter(hdf5_loader_generator(dataset, batch_size, as_tensor))

I timed pytorch's iterator as follows:

start = time.time()
for i, data in enumerate(trainloader, 0):
    ft, sp, index, sp_values, index_values = data
    ft, sp, index = ft.to(device), sp.to(device), index.to(device)
    if i > 500:
        break

end = time.time()

print(end-start)

I timed my iterator as follows:

train_dataset = My_H5Dataset(hdf5_data_folder_train)
samples = hdf5_data_iterator(train_dataset, batch_sizes)
start = time.time()
for i, data in enumerate(samples):
    ft, sp, index, sp_values, index_values = data
    ft, sp, index = ft.to(device), sp.to(device), index.to(device)
    if i > 500:
        break

end = time.time()

print(end-start)

Is there a better approach than what I am currently trying?
Thank you for the help!


texts: CNN Design for Counting on Simple Images

texts:
This is the first CNN I'm designing following college examples and assignments. I'm working on a CNN that I'd like to use to classify images by the number of shapes on them. My basic problem is that I can't seem, to get the CNN to respond (accuracy and val_accuracy are flat) after n EPOCHS (I have varied n these along with Steps and Batch Size) The images are 98 x 150 pixels and look like this:

This is 10 data squares on a by background. The image set has been built by varying the position of the shapes on the image, changing the background color (4 colors), changing the shape color (3 colors), changing the shape type (square, circle, hexagon), flipping the images (vert/horiz). The images are stored in folders (1-26) with 1 having 1 shape per image, 26 having 26 shapes per image. There are 3300 images per folder totaling ~86k images (I have experimented with a lower number of shapes). Here's an images of the training/validation file counts:

The problem I'm seeing is the model never seems to "initialize" in other words the accuracy and validation accuracy do not change much while training the model:

The accuracy and val_accuracy are very different to any models I've worked with - normally I see a fast ramp rate which tapers off as the model trains. In this case both are flat and never increase. Even after larger EPOCHs.
Here's one of the models I've tried:
# Define the Model
model = models.Sequential([
  layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.flatten(),
  layers.Dense(512, activation='relu'),
  layers.Dense(num_classes)
])

I've tried several variants of this model along with an AlexNET all yield the same results.
I'm thinking that maybe the images are too simple/varied for the model to train? Maybe putting the shapes on a pixelated background would help make them more complex/varied? I'm also wondering whether initializing the model with random weights/biasses might help? I'm using Tensorflow 2.x and don't know whether this is done automatically.
Does anyone have any thoughts/advice on how I may improve the model and at least see accuracy/validation increase rather than it being flat? I could deal with overfitting later (if it occurs).
I've be very grateful for any advice - thanks in advance!


texts: Adding learning episodes to epochs with Keras

texts:
I want to implement a prototypical network (https://arxiv.org/pdf/1703.05175.pdf) with keras (there are some implementations in pytorch and tensorflow, but those are not ideal for me).

This NN is used for few-shot learning and the algorithm works in the way that it chooses several observations of various classes which are embedded through a certain function, centroids of these classes are calculated and a loss function based on the distance of some other embedded observations from the centroids is calculated. The embedding function is then optimizied  (= 1 episode) and the process repeats. After all the train data have been used, a new epoch starts.

It is not clear to me whether this can be reasonably implemented with keras and, if so, what would be the best way to do it. Could anybody give me a hand on how to insert such episodes into an epoch, or recommend me a good source?

For a reference, see the tensorflow implementation https://github.com/kvpratama/prototypical-networks-tensorflow

Thank you very much in advance.


texts: Are there better solutions on visualizing of two variables having very different range and mean on same graph?

texts:
I'm now working on an analysis comparing time consuming of a pairwise experiment. AI and human processed same subject in experimental group and processing time recorded. The time consuming of AI are significantly shorter than human and two variables(time of AI route and time of human route) have different distribution which makes it is weird to show them on a same graph.

The head of record table is like this.

The graph visualization will be used in an essay focusing on showing the difference of their means and their ranges rather than the distributions. Because of that, the data should be presented objectively without transformation.

What below are my solutions and all of them seems ugly.

Is there any trick you guys using to visualize such data? Please teach me. Thank you


texts:
I use random forest to train on my data (My data had imbalance in the target class, i.e. rare 1 and abundant 0). I face 3 issues about the stability of estimator and its prediction power. I think these problems could be common on many machine learning algorithms.

I found the ROC_AUC_Score highly changeable when I resample the training set (the rest was test set). It can vary from $0.85$ to $0.45$ while changing the training set.
The parameter tuning can also cause the move of the estimator and ROC_AUC_Score but the effect was weaker than the first case above.
When running some iteration of model fitting, the results were also different from each other but the effect was the weakest.

For 2, 3, I think we can get the best parameter setting and fitting results by recording each iteration and parameter tuning. (Maybe can do it more efficiently, please advice)

Please also advice how to deal with the first problem to make the fitting convinced and reliable? Cross Validated?

Many thanks


texts: CNN, which layer to choose for a similarity measure

texts:
I built a model (InceptionResnet v2) to classify images and I would like to use it to measure similarity between objects.

One way to measure that similarity is to catch an intermediate layer's results from two vector and look their distance.

I don't know how to choose the layer, my last layers are :

Conv -> Flatten(98304,) -> Dense(256,) -> Dense (softmax)

I hesitate between Flatten and first Dense, how to choose ?


texts:
I came across this statement from CNTK 103A:
gzfname, h = urlretrieve(src, './delete.me')

I understand all the rest but that gzfname, h = ......
What exactly is the purpose of h, is it a filehead thing?
Code:
def load_or_download_mnist_files(filename, num_samples, local_data_dir):
    gzfname, h = urlretrieve(src, './delete.me')
    if (local_data_dir):
        local_path = os.path.join(local_data_dir, filename)
    else:
        local_path = os.path.join(os.getcwd(), filename)

    if os.path.exists(local_path):
        gzfname = local_path
    else:
        local_data_dir = os.path.dirname(local_path)
        if not os.path.exists(local_data_dir):
            os.makedirs(local_data_dir)
        filename = "http://yann.lecun.com/exdb/mnist/" + filename
        print ("Downloading from" + filename, end=" ")
        gzfname, h = urlretrieve(filename, local_path)
        print ("[Done]")

    return gzfname


texts: Is there a way to have the following occur: When a job is finished on the SAC (SAP Analytics Cloud) side, a job on the SAP BW side should be triggered. Is that at all possible? I have found this which seems to do what I want, but in reverse order. I am unsure if it is possible to do the opposite using their method


texts: Can we Create Neural network(Simple one such as Multi Layer perceptron) that only contains positive weights only?

texts:
I was wondering if there is a specific method to create a well performing neural network with only positive weights (I already tried clipping the weight before training or so and initializing the weights with only positive value ) but still doesn't give a good results so is there any other method to do so ?

Thank You


texts: key parameter in max function in Pyspark

texts:
In the example given for the max function for PySpark:

Pyspark

>>> rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])
>>> rdd.max()
43.0
>>> rdd.max(key=str)
5.0

Q1. How does it get evaluated to 5.0 when key=str ? Is it based on conversion to character/string type ?

Q2. Is there any value the parameter "key" can take ?

I also found the function definition of "max" at this location

https://github.com/adobe-research/spark-gpu/blob/master/src/rdd.py

def max(self, key=None):
        """
        Find the maximum item in this RDD.
        :param key: A function used to generate key for comparing
        >>> rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])
        >>> rdd.max()
        43.0
        >>> rdd.max(key=str)
        5.0
        """
        if key is None:
            return self.reduce(max)
        return self.reduce(lambda a, b: max(a, b, key=key))


texts: Custom layer in keras with multiple input and multiple output

texts:
I have implemented a custom layer in keras which takes in multiple input and also results to multiple output shape. My code goes as below:

class Attention(Layer):
def __init__(self, max_input_left=MAX_SEQUENCE_LENGTH,max_input_right=MAX_SEQUENCE_LENGTH,
             filter_sizes=[1,2,3], num_filters=400, **kwargs):
    self.init = initializers.get('normal')
    self.max_input_left = max_input_left
    self.max_input_right = max_input_right
    self.filter_sizes = filter_sizes
    self.num_filters = num_filters
    if self.num_filters!=0:
        self.num_filters_total=self.num_filters * len(self.filter_sizes)
    super(Attention, self).__init__(** kwargs)

def build(self, input_shape):
    input_dim=input_shape[1][-1]*len(self.filter_sizes)
    self.U = self.init((input_dim,input_dim))
    super(Attention, self).build(input_shape)

def compute_mask(self, inputs, mask=None):
    mask = super(Attention, self).compute_mask(inputs, mask)
    return mask

def call(self,inputs, mask=None):
    input_left = inputs[0]
    input_right = inputs[1]
    Q = tf.reshape(input_left,[-1,self.max_input_left,len(self.filter_sizes) * self.num_filters],name = 'Q')
    A = tf.reshape(input_right,[-1,self.max_input_right,len(self.filter_sizes) * self.num_filters],name = 'A')
        # G = tf.tanh(tf.matmul(tf.matmul(Q,self.U),\
        # A,transpose_b = True),name = 'G')
    print Q
    print tf.reshape(Q,[-1,len(self.filter_sizes) * self.num_filters])
    first = tf.matmul(tf.reshape(Q,[-1,len(self.filter_sizes) * self.num_filters]),self.U)

    second_step = tf.reshape(first,[-1,self.max_input_left,len(self.filter_sizes) * self.num_filters])
    result = tf.matmul(second_step,tf.transpose(A,perm = [0,2,1]))
        # print 'result',result
    G = tf.tanh(result)
        # G = result
        # column-wise pooling ,row-wise pooling
    row_pooling = tf.reduce_max(G,1,True,name = 'row_pooling')
    col_pooling = tf.reduce_max(G,2,True,name = 'col_pooling')

    self.attention_q = tf.nn.softmax(col_pooling,1,name = 'attention_q')
    print self.attention_q
    self.attention_a = tf.nn.softmax(row_pooling,name = 'attention_a')
    R_q = tf.reshape(tf.matmul(Q,self.attention_q,transpose_a = 1),[-1,self.num_filters * len(self.filter_sizes)],name = 'R_q')
    R_a = tf.reshape(tf.matmul(self.attention_a,A),[-1,self.num_filters * len(self.filter_sizes)],name = 'R_a')
    self.output_dim1  = R_q
    self.output_dim2 = R_a
    return R_q,R_a

def compute_output_shape(self, input_shapes):

    print (tuple(K.int_shape(self.output_dim1)),tuple(K.int_shape(self.output_dim2)))
    return (tuple(K.int_shape(self.output_dim1)),tuple(K.int_shape(self.output_dim2)))

I am calling this layer as:

  attn1, attn2 = Attention()([qns_cnn, ans_cnn])

This gives me the following error:

Traceback (most recent call last):
File "workspace-2/SemEval/test2QA.py", line 253, in <module>
attn1 = Attention()([qns_cnn, ans_cnn])
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 619, in __call__
arguments=user_kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 681, in _add_inbound_node
output_tensors[i]._keras_shape = output_shapes[i]
AttributeError: 'tuple' object has no attribute '_keras_shape'

Can someone suggest where is the issue?


texts:
Having a bit of a mind-blank at the moment and am looking for some advice.

I am extracting features from time series data for input into a classification algorithm, for example I'm extracting average and variance from inputX.

For input Y,  I have graphed the data and have seen that for class A, it can be seen that there is an upwards slope, and for class B, it can be seen that there is a downward slope, for class C there is no slope, the line is more or less straight.
For Feature Extraction, how can I best describe this? Would a calculation to get positive/negative slope be best?


texts: TypeError: Cannot cast array data from dtype('float64') to dtype('S32') according to the rule 'safe'

texts: I'm getting the above error while I'm extracting the sensor data from serial port and giving it for prediction.Tried reshape and string functions but every trial throws back me same error.i'm using mlp neuralnetwork in my model.Thanks in advance


texts: Why is IoU said to be non-differentiable?

texts: I have been trying to find an answer online but I couldn't really find one. If anyone could help me I would appreciate it


texts:
In terms of application, what happens after we train a classifier? What can we learn from it?

For example, if I trained a classifier to predict the success of a Kickstarter campaign with 80% accuracy, how can I apply this information to benefit my own Kickstarter campaign? Is there a way to know which attributes influence the outcome 'success' or 'failed' the most?


texts:
I have training data that is a set of transaction descriptions and the category of that transaction.
I am trying to build a spend analyzer using this information to help me classify my transactions into categories. I built a model by training a custom word2vec model on my data and got good accuracy with it.
The problem is that every few weeks, as I get newer data with newer words, the quality of my predictions starts dropping. I could retrain my model but I'll have to manually sit and label these transactions to get an updated training set. Labeling the transactions manually seems the opposite of what I am trying to achieve and takes more effort than building the model.
So my question is how do I retrain my model with newer descriptions of transactions without actually having the true labels?


texts: Vectorised Neural Network Backprapogation

texts:
I've started the CS231n course and have been going through the proofs and derivations for backpropagation. These two papers by the lecturers are what I've been reading:

Paper 1: - http://cs231n.stanford.edu/handouts/linear-backprop.pdf
Paper 2: - http://cs231n.stanford.edu/vecDerivs.pdf

Reading both papers individually they both make sense, but when trying to link the concepts explained in both together, something doesn't make sense to me.

Paper 2 lists the loss w.r.t the inputs as: dL/dX = dL/dY*W_transpose where W is the weights matrix
This follows on from the use of the chain rule where dL/dX = dL/dY * dY/dX

Paper 1 however, mentions in the first sections that dY/dX = W which is the weights matrix, but not transposed.

My question is, am I missing something here as to why one paper mentions W transposed and the other doesn't?

Thanks everyone for your help!


texts:
What happends in this numpy function:
https://numpy.org/doc/stable/reference/generated/numpy.unique.html
a = np.array([1, 2, 5, 3, 2])
u, indices = np.unique(a, return_inverse=True)

The results are:
u

array([1, 2, 3, 5)]
indices

array([0,1,3,2,1), dtype=int64)
u[indices]

array([1, 2, 5, 3, 1])
u are the single values inside this array. What is indices? why is it a 3 and not a 5? and what is going on in u[indices]?


texts: How to plot logistic regression decision boundary?

texts:
I am running logistic regression on a small dataset which looks like this:

After implementing gradient descent and the cost function, I am getting a 100% accuracy in the prediction stage, However I want to be sure that everything is in order so I am trying to plot the decision boundary line which separates the two datasets.

Below I present plots showing the cost function and theta parameters. As can be seen, currently I am printing the decision boundary line incorrectly.

Extracting data

clear all; close all; clc;

alpha = 0.01;
num_iters = 1000;

%% Plotting data
x1 = linspace(0,3,50);
mqtrue = 5;
cqtrue = 30;
dat1 = mqtrue*x1+5*randn(1,50);

x2 = linspace(7,10,50);
dat2 = mqtrue*x2 + (cqtrue + 5*randn(1,50));

x = [x1 x2]'; % X

subplot(2,2,1);
dat = [dat1 dat2]'; % Y

scatter(x1, dat1); hold on;
scatter(x2, dat2, '*'); hold on;
classdata = (dat>40);

Computing Cost, Gradient and plotting

%  Setup the data matrix appropriately, and add ones for the intercept term
[m, n] = size(x);

% Add intercept term to x and X_test
x = [ones(m, 1) x];

% Initialize fitting parameters
theta = zeros(n + 1, 1);
%initial_theta = [0.2; 0.2];

J_history = zeros(num_iters, 1);

plot_x = [min(x(:,2))-2,  max(x(:,2))+2]

for iter = 1:num_iters
% Compute and display initial cost and gradient
    [cost, grad] = logistic_costFunction(theta, x, classdata);
    theta = theta - alpha * grad;
    J_history(iter) = cost;

    fprintf('Iteration #%d - Cost = %d... \r\n',iter, cost);

    subplot(2,2,2);
    hold on; grid on;
    plot(iter, J_history(iter), '.r');  title(sprintf('Plot of cost against number of iterations. Cost is %g',J_history(iter)));
    xlabel('Iterations')
    ylabel('MSE')
    drawnow

    subplot(2,2,3);
    grid on;
    plot3(theta(1), theta(2), J_history(iter),'o')
    title(sprintf('Tita0 = %g, Tita1=%g', theta(1), theta(2)))
    xlabel('Tita0')
    ylabel('Tita1')
    zlabel('Cost')
    hold on;
    drawnow

    subplot(2,2,1);
    grid on;
    % Calculate the decision boundary line
    plot_y = theta(2).*plot_x + theta(1);  % <--- Boundary line
    % Plot, and adjust axes for better viewing
    plot(plot_x, plot_y)
    hold on;
    drawnow

end

fprintf('Cost at initial theta (zeros): %f\n', cost);
fprintf('Gradient at initial theta (zeros): \n');
fprintf(' %f \n', grad);

The above code is implementing gradient descent correctly (I think) but I am still unable to show the boundary line plot. Any suggestions would be appreciated.

logistic_costFunction.m

function [J, grad] = logistic_costFunction(theta, X, y)

    % Initialize some useful values
    m = length(y); % number of training examples

    grad = zeros(size(theta));

    h = sigmoid(X * theta);
    J = -(1 / m) * sum( (y .* log(h)) + ((1 - y) .* log(1 - h)) );

    for i = 1 : size(theta, 1)
        grad(i) = (1 / m) * sum( (h - y) .* X(:, i) );
    end

end

EDIT:

As per the below answer by @Esmailian, now I have something like this:

[m, n] = size(x);

x1_class = [ones(m, 1) x1' dat1'];
x2_class = [ones(m, 1) x2' dat2'];

x = [x1_class ; x2_class]


texts: Learning a and predicting workflow for a certain goal

texts: I am trying to come up with a strategy which could learn a workflow for various goals and then based on some input goal, predict the workflow which could fulfil the goal. Goals are achieved by workflows. E.g., Reach Office:= Tasks('Leave Home', 'Travel to Location A', 'Board Bus', 'Drop at Location B', 'Walk to Location C'). I wish to understand how a learning mechanism would enable learning such multiple goals and their corresponding workflows. And once the system has learnt the goals and workflows, the system should predict workflows for new goals like "Reach Theater", "Make Coffee" etc. The training data will include lots of existing goal and task recipes. Any pointers in this area would be very helpful. Can anyone suggest any existing approach?


texts: How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

texts: Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is expected to be low. How to choose a good decision boundary for such cases.


texts: Retraining EfficientNet on only 2 classes out of 4

texts:
EfficientNet model was trained on ~3500 images for a 4-class classification: A, B, C and Neither – with accuracy of 0.985 – by someone else, not me. I'm quite new to ML.  So we have this model, and it works pretty well.

As more real-world images are coming in from the users, we see more errors. Is it possible to improve the performance a bit by adding more training examples?

Collected and manually sorted 35 images of class A and 425 of class "Neither". Does it even make any sense with so small dataset?

I have put images into a folder with 4 subfolders, one per each class. Two are empty, two have images. Using Keras ImageDataGenerator flow_from_directory() method with validation/training = 0.2/0.8 ratio.

model.fit() training shows low loss and accuracy close to 1, while validation loss is high and accuracy is around 0.21 — 0.25 only.

Model performance after my "training" is totaly ruined.

Is this approach to retraining with a little set of images wrong?

How can I "add", not "replace" the model "skills" with a new added little set of images?

Maybe I should have trained a fresh model initialised with the ImageNet weights as in the initial training, and just add the new images to the set of images used, again, in the initial training?


texts: What methods to create singular content classification from inconsistent inbound info?

texts:
I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with inconsistent or absent in-bound taxonomy terms.

Primary source of profile info is biography pages on people's employer websites. Some of those sites choose to state employees' multiple specialist topics, some make only narrative biographies available, some both. I have collected all available info, using Python's Scrapy, in to CSV files - one per company, people are rows  - where available, topics at my end now themselves reside in a comma-separated field/string.

Example: in one sheet, cell S7 is: "Analytics Applications,Big Data,Cognitive Computing,Competitive Intelligence,eDiscovery,Enterprise Content Management (ECM),Information Architecture,Market Research,Product Information Management (PIM)"

The problem is severalfold:

Taxonomy terms across companies are inconsistent (eg. "Cognitive Computing" in the above example may, to another company, be "AI").
Some companies use far too many terms in total (eg. one company alone uses approx 450 tags in total).
Often, none are available at all.
As biography narratives describe more than just employees' specialist topics (eg. education and upbringing background), their usefulness in automation may be questionable.

My goal is to create a taxonomy that categorises all the collected person bios in a much more harmonious, consistent and briefer fashion.

System setup is PHP/MySQL/WordPress. Profile CSVs are imported in to WordPress, and the system has the ability to perform PHP functions on imported content (not just on the info in WordPress after import, but during import via PHP).

Total profiles count is approx 4,500, so manual taxonomisation is unappealing. So I have examined AI/machine learning techniques. I am not strictly a developer and certainly not a data scientist or mathematician.

So far, I have found text classification tests carried out using Aylien and Monkey Learn to yield poor results. In each case, output results are not granular enough, ie. turning in-bound terms of biogs about granular topics like cloud computing infrastructure and data centres in to overly basic terms like "Computers & Internet". Aylien uses the off-the-shelf IPTC NewsCodes taxonomy, and I understand I can use Monkey Learn to train. I like the idea of using a standardised off-the-shelf taxonomy like NewsCodes, but a) the results are questionable, and b) it may not be granular enough for my needs.

At this point, I have decided to draw up my preferred hierarchy of taxonomy terms, approx 230, which should each speak roughly to the swathe of inconsistent in-bound terms and profiles (in other words, correlate to the people's topics). That seemed like an important step, assuming I need to steer this manually. But I'm struggling to grasp how to actually implement that correlation.

So, I am looking for some guidance on best methods.

One idea I am toying with is to put my own preferred taxonomy in to WordPress as taxonomy terms, and, alongside each, put a cluster of terms from the actual source material so that, if one of the related terms is found in a user's inbound data, the term from my preferred taxonomy should be assigned. But I'm not sure whether this is particularly efficient, or even wise.

This is my first time on the Data Science group at StackExchange. I apologise if I have shot wide of the mark here at all.


texts: How to train models using Google ML ( keras & tenserflow)

texts:
Do I need to change my existing code I used to train neural network from my PC.
here is my current code.

Model = Sequential()
Model.add(Convolution2D(32,3,3, input_shape=(128,128,3),activation="relu"))

Model.add(MaxPooling2D(pool_size=(2,2)))

Model.add(Convolution2D(32,3,3,activation="relu"))
Model.add(MaxPooling2D(pool_size=(2,2)))

Model.add(Flatten())

Model.add(Dense(output_dim =128, activation= 'relu'))
Model.add(Dense(output_dim =1, activation= 'sigmoid'))
Model.compile(optimizer="adam", loss = 'binary_crossentropy', metrics= ["accuracy"])

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

traning_set = train_datagen.flow_from_directory(
        'data/train_set',
        target_size=(64, 64),
        batch_size=32,
        class_mode='binary')

test_set = test_datagen.flow_from_directory(
        'data/test_set',
        target_size=(64, 64),
        batch_size=32,
        class_mode='binary')

classifier.fit_generator(
        traning_set,
        steps_per_epoch=204,
        epochs=20,
        validation_data=test_set,
        validation_steps=101)

classifier.save('mymodel.h5')


texts: how to compare between kmeans and hierarchical clustering results

texts:
I am using 2 types of clustering algorithm
I apply hierarchical clustering the K-means clustering using python sklearn library
Now the results are a little bit different so how can I compare the results and which algorithm to use? because I want to write a conclusion for a set of unlabeled data.
Is there any benefit to use multiple algorithms and compare between them?


texts: need explanation on how an equation is being converted to cvxopt logic in solver.lq

texts:
This is the equation that is given in the example:

and the code to replicate it in python is

>>> from cvxopt import matrix, solvers
>>> A = matrix([ [-1.0, -1.0, 0.0, 1.0], [1.0, -1.0, -1.0, -2.0] ])
>>> b = matrix([ 1.0, -2.0, 0.0, 4.0 ])
>>> c = matrix([ 2.0, 1.0 ])
>>> sol=solvers.lp(c,A,b)

Can anyone please explain how are the the data given to variables a, b, c?


texts: tflearn - how to update model and save without completely retraining model

texts:
I am using tflearn to text classification.  I am able to create DNN and fit it and save it and also retrieve it for predictions.

I would like to update trained model with new test data. I checked documentation and also methods but the only way I could figure out is to call

model.fit() which would completely retrain the model and takes a lot of time since it is from scratch.

On other hand model.save() would just save the loaded model without updating new data.

Is there a function or way to update the trained model with just one data without going through whole retraining process?

Code used to load and save model

model.fit(train_x, train_y, n_epoch=1000, batch_size=8, show_metric=True)
model.save('model.tflearn')

Code used to retrieve and load existing model

tf.reset_default_graph()
# Build neural network
# train_x[0] and train_y[0] had to be regenerated from scratch
net = tflearn.input_data(shape=[None, len(train_x[0])])
net = tflearn.fully_connected(net, 8)
net = tflearn.fully_connected(net, 8)
net = tflearn.fully_connected(net, len(train_y[0]), activation='softmax')
net = tflearn.regression(net)

# Define model and setup tensorboard
dnn = tflearn.DNN(net, tensorboard_dir='tflearn_logs')
dnn.load('./model.tflearn')


texts: Existing pre-trained NLP models to detect if a text input is a question

texts: I would like to quickly filter text data into question and non-questions. Using the presence of question mark in the text is too crude. Are there any existing models I can use to aid me with my task?


texts: non-deep learning alternatives to mixture density network

texts:
I am currently using a Mixture density network to fit a distribution to a large dataset.
The input to my network contains many features, while the output is a simple one-dimensional probability distribution. The input parameters are a mix of continuous and discrete values.

I am using this approach because I am looking for conditional probabilities, i.e. I wish to know how the parameters of my distribution change with my input.

I am looking for alternative non-neural network algorithms that perform the same task and do not require too many additional assumptions.
I am looking for those alternative in order to better gauge the performance of my MDN.

Any help would be appreciated.


texts: How do I interpret loss and accuracy per epoch while training a CNN?

texts:
I am really new to Neural Networks, and I am training a CNN for image classification, and while training, I get the following:

which tells me the training loss and accuracy and validation loss and accuracy, soemone please correct me if i am wrong.
So, my question is:
What are pecisely these quantities? And is there a way to understand if I a, doing well with my problem looking at them?


texts: What does the activation of a neuron mean?

texts:
In a neural network, each neuron will have it's activation.
But what the activation mean?

Does it just mean nothing but a temporal value to produce the final result or it's has something to with our understanding to the problem?

For example, considering a neural network to recognize a handwritten character from a picture, I wonder if it's possible that a neuron's activation represents the how the pixel matches the specific value in a small part of the picture?


texts: Issue with backpropagation using a 2 layer network and softmax


texts: k-means classifies one data point as a group

texts:
I have 1000 sets of one dimensional data (360 each in length), and I want k means to classify what is a small/medium/large value (n_clusters=3) for each set of data, but I'm getting a lot of instances where the large group only has 1 data point because that value is so far away from the rest, but the rest look like they can clearly create 3 clusters.

In some other cases, it does seem to make sense to use 1 data point as the large group since the rest are so close together. It's not clear if there can be 3 distinctive clusters.

What would be an efficient way to deal with this?


texts: whether to use normalization while performing PCA?

texts:
I have an excel file containing a table where I registered the frequency of three linguistic phenomena in 72 poems. Since the poems have different lengths I normalized the results dividing each value by the number of tokens of each poem and multiplying per 100. After that, I opened orange, added a new file, connected data table and selected the table. At this point, I connected PCA and with two components I have 97% explained variance but only if I deselect the "normalize variables" option and then I have a nice scatter plot.
My question is the following: when do I need to use the "normalize variables" option? Because in this case, I deselected it because I had already normalized the data. However, I am not sure if I made any mistakes.


texts:
I am making an document parser which extracts data fields from the documents and store them in a structured way. Each field in my dataset is horizontal which is easy to extract.

But the model fails on following type of example -

Is there any way to extract invoice number and date from such images.


texts: Should Feature Selection processes be apply on training data or on all data?

texts:
I've realized that on examples and guides, sometimes feature selection processes (correlation elimination, backward/stepwise) are applied on the train data after splitting all data but on the other hand, sometimes its applied on all data.

So is there any clear answer for that? Which is more logical?


texts: PCA on conditional heteroscedastic timeseries

texts: What is the correct method of application of PCA on time series data. Since the time series may exhibit conditional heteroscedasticity, application of normal PCA might be wrong as the variance changes with time.


texts: Can 2 different OOV words get the same vector in FastText?

texts: Since FastText sums up the vectors(order is not considered) of an OOV word's subwords, is it possible for two different OOV words to get the same vector ? If so, then can you give an example?


texts: tflite_convert a Keras h5 model which has a custom loss function results in a ValueError, even if I add it in the Keras losses import

texts:
I have written a SRGAN implementation. In the entry point class of the Python program, I declare a function which returns a mean square using the VGG19 model:

# <!--- COST FUNCTION --->
def build_vgg19_loss_network(ground_truth_image, predicted_image):
    loss_model = Vgg19Loss.define_loss_model(high_resolution_shape)
    return mean(square(loss_model(ground_truth_image) - loss_model(predicted_image)))

import keras.losses
keras.losses.build_vgg19_loss_network = build_vgg19_loss_network
# <!--- /COST FUNCTION --->

(Vgg19Loss class shown further below)

As you can see, I have added this custom loss function in the import keras.losses. Why? Because I thought it could solve the following problem...: When I execute the command tflite_convert --output_file=srgan.tflite --keras_model_file=srgan.h5, the Python interpreter raises this error:

  raise ValueError('Unknown ' + printable_module_name + ':' + object_name)
  ValueError: Unknown loss function:build_vgg19_loss_network

However, it didn't solve the problem. Any other solution which could work?

Here is the Vgg19Loss class:

from keras import Model
from keras.applications import VGG19

class Vgg19Loss:
    def __init__(self):
        pass

    @staticmethod
    def define_loss_model(high_resolution_shape):
        model_vgg19 = VGG19(False, 'imagenet', input_shape=high_resolution_shape)
        model_vgg19.trainable = False
        for l in model_vgg19.layers:
            l.trainable = False
        loss_model = Model(model_vgg19.input, model_vgg19.get_layer('block5_conv4').output)
        loss_model.trainable = False
        return loss_model


texts: Extra output layer in a neural network (Decimal to binary)

texts:
I'm working through a question from the online book.
I can understand that if the additional output layer is of 5 output neurons, I could probably set bias at 0.5 and weight of 0.5 each for the previous layer. But the question now ask for a new layer of four output neurons - which is more than enough to represent 10 possible outputs at $2^{4}$.
Can someone walk me through the steps involved in understanding and solving this problem?
The exercise question:
There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.


texts:
I am using arulesCBA on dataset of words with class attribute which is polarity to be positive or negative. First, I am converting the words to numeric values by using as.numeric function. After that, I am discretizing the columns using this code:
trans.disc <- as.data.frame(lapply(df[2:75], function(x) discretize(x, categories=9)))

In this step, I have warnings that say: parameter categories is deprecated. Use breaks instead! Also, the default method is now frequency! The next step that I am applying is adding the polarity column:
trans.disc$polarity <- df$polarity

The last step, I am trying to build the classifier:
classifier <- CBA(trans.disc, "polarity", supp = 0.05, conf=0.9)

In this phase, there is an error message that says: (Error in discretizeDF.supervised(formula, data, method = disc.method) :data needs to be a data.frame).


texts: Conditional attributes in a prediction problem

texts:
I am trying to implement an algorithm to predict fantasy points of a player regarding past events. However, I am having some doubts about some attributes.
Imagine that a player is going to play away in the next match, I believe it is useful to have the points/goals he did in the past away matches to evaluate how well he performs away from his stadium. However, that same attribute would not be useful if the next match is at home. The same with the % of occupation of the stadium - it is important when a player plays at home but not relevant when he plays away.
What is the best approach to this kind of scenario? Do I put the away goals anyway even when the match is at home?

Regards


texts:
Consider the following context:

You have a website that is offering a search function, which produces a list of matching articles.

The search has prefix functionality: if you are looking for "Samsung Galaxy", the user can also input "Sams Galax" and he will still get a list of results. Now what you try to find out, how many users are using shortened (prefix-) and how many are using full terms.

As your dataset you have a list of customer-executed queries which also contains a number how many articles were in the produced results. But you have no data which of the resulted articles the users finally clicked on. So you can't easily indicate whether the search was successful or not and what exactly the users were looking for.

How would you analyze the data to split it in the two groups, e.g. prefix- and full-term?


texts: Rstudio using 2.5% of 250GB RAM. how to Increase it

texts: I am working on Rstudio on a server which has 250GB ram. But its taking too much time to handle a 2GB data file. how should i speed up my work?


texts: Plotting ROC curve after re-sampling the data set

texts:
I am a beginner in Machine Learning. I am working on a project to compare the performance of three ML algorithms which are: LR, SVM and ANN on a dataset containing medical records.

After exploring the data I have found that the dataset is imbalanced, and hence I have applied SMOTE. Then I applied PCA on the resampled data to reduce the dimensionality of the dataset and to improve the performance of the Algorithms.

After plotting a heatmap of the Confusion Matrices of each Algorithms I want to plot the ROC curves showing the AUC scores. Following is the screenshot of the heatmap of the Confusion Matrix of SVM.

And this is the code that I have used to plot the ROC Curve

    from sklearn.metrics import roc_curve, auc
    ### Fit a sklearn classifier on train dataset and output probabilities
    pred_val = svc.predict_proba(self.X_test)[:,1]
    ### Compute ROC curve and ROC area for predictions on validation set
    fpr, tpr, _ = roc_curve(self.y_test, pred_val)
    roc_auc = auc(fpr, tpr)

    ### Plot
    plt.figure()
    lw = 2
    plt.plot(fpr, tpr, color='darkorange',
             lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
    plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic example')
    plt.legend(loc="lower right")
    plt.show()

This is the result:

Please tell me wheter the result shown by the ROC curve is correct or not.


texts: Data Science Conferences w/ Poster Sessions?

texts:
I am in search of some data science conferences (for 2017) that have a mix of big name speakers and poster sessions.  Ideally, I would like to be able to present a poster that doesn't necessarily have to be a research paper, but simply the development of a tool and its use.

Please provide the name and link.  Thank you!


texts:
I am trying to write an optimal control agent for a simple game that looks like this:

The agent can only move along the x-axis, and has three actions available to it: left, right, and do nothing. A random number of falling rocks are spawned at arbitrary positions along the top row. The goal is to survive as long as possible by avoiding collision on each time step.

Here's what I've done so far:

1) I use a feature vector $Φ$ with $Φ_0(s)$ being the current x-coordinate of the agent and $Φ_1(s)...Φ_n(s)$ taking on a 0 or 1 (1 indicating the presence of a rock).

2) The corresponding weight vector $θ$ is initialized to 0 for all weights. So I have the linear function approximation $$\hat v(s,θ)=\sum_{i=0}^nΦ_i(s)θ_i$$

3) The reward on each time step is 1, and 0 upon collision.

I'm actually trying to implement the algorithm below from Sutton and Barto's 2017 draft.

Semi-gradient TD(0) for estimating $\hat{v} \approx v_{\pi}$

Input: the policy ${\pi}$ to be evaluated
Input: a differentiable function $\hat{v} : \mathbf{S^+} \times \mathbb{R}^n \rightarrow \mathbb{R}$ such that $\hat{v}(terminal,·) = 0$
Initialize value-function weights $\theta$ arbitrarily (e.g., $\theta = 0$)
Repeat (for each episode):
$\qquad$Initialize $S$
$\qquad$Repeat (for each step of episode):
$\qquad\qquad$Choose $A \sim \pi(·|S)\qquad$ # Not sure how to choose action here
$\qquad\qquad$Take action $A$, observe $R, S'$
$\qquad\qquad\theta \leftarrow \theta + \alpha[R + \gamma \hat{v}(S',\theta) − \hat{v}(S,\theta)]\nabla\hat{v}(S,\theta)$
$\qquad\qquad S \leftarrow S'$
$\qquad$until $S'$ is terminal

My problem is this:

1) Is this even an appropriate algorithm to apply to this sort of task? It feels like a policy gradient approach would be more suitable.

2) If yes to 1), how do I choose an action? In the algorithm above, this appears as "Choose $A \sim \pi(·|S)$" Since the policy is implicit,  I compute the approximate value of the state as shown in 2) above and then greedify over the three actions (no ε-greedy, though), but the only thing that changes is $Φ_0(s)$. I'm definitely missing something.


texts:
Given a sample $ X_1,X_2 \dots X_{100}$ and the density function   $ f(x;\theta) = \frac{1}{\pi \cdot \left(1+\left(x-\theta \right)^2\right)}$ , find an approximate solution for  $\hat{\theta}_{MLE.}$
My attempt:
I have found the joint likelihood $L(\theta;x_1,x_2\dots x_{100}) = \prod _{i=1}^{100}\left(\frac{1}{\pi \cdot \left(1+\left(x_i-\theta \right)^2\right)}\right)\:$
$l$ = $\log(L) = -100*\ln(\pi)-\sum^{100}_{i=1}(\ln(1+(x-\theta)^2)$.
I'm not sure of this step
$\frac{\partial }{\partial \theta}\left(\log(L)\right) = \sum_{i=1}^{100}(\frac{2(x_i-\theta)}{1+(x_i-\theta)^2}$
then I used Newton's method to find the maxima.
this is the script I used to calculate the maxima
#deravitive of log(L).
fun1 <- function(theta){
  y1 <- 0
  for(i in 1:length(x)){
    y1 <- y1 + (2*(theta-x[i]))/(1+(x[i]-theta)^2)
  }
  return(y1)
}

#derivative of fun1.
fun1.tag <- function(theta){
  y <- 0
  for(i in 1:length(x)){
    y <- 2*(theta^2+(x[i]^2)-20*x[i]-1)/((1+(x[i]-theta)^2)^2)
  }
  return(y)
}

# The Newton's method.

guess <- function(theta_guess){
  theta2 <- theta_guess - fun1(theta_guess)/fun1.tag(theta_guess)
  return(theta2)
}
theta1 <- median(data$x)
epsilon <- 1

theta_before <- 0

while(epsilon >0.0001){
  theta1 <- guess(theta1)
  epsilon <- (theta_before- theta1)^2
  theta_before <- theta1
}

What I got was $\hat{\theta}_{MLE} = 5.166$
I'm now trying to plot the data(in my case x) and check if $\hat{\theta}_{MLE} = 5.166$ is actually a maxima.


texts:
Given sequences of points that define similar lines amongst each others, e.g:

Where the blue lines are similar to each other, given the red point sequence and assuming that it will follow the same behaviour as the blue ones, I want to be able to determine what are the next red points.

Therefore my question is three fold: i) Is it possible to use machine learning approaches to address this problem? ii) Should I use machine learning approaches to address this problem? iii) Which known machine learning approaches are most suitable to address this problem?

Thank you for your time.


texts:
I would like to be able to resample a histograms bins without having access tot he raw data.  And just to be clear, by resample, I mean to change the number of bins and still provide a good estimate of the original probabilities of those bins.

I can think of many ways to do this, but having trouble figuring out which is the best method which maintains the same probability in the resulting histogram.  The easy one would be if the input histogram X had x bins and the desired result histogram Y has y bins where x = y.  This is a simple 1 - 1 sampling of the original bins.  The problem forms to me as I decrease y lower than x or increase it above x.

For example: If x = 10 bins and y = 20 bins, it seems like you could simply double each of x's bins so you have Y = { x1, x1, x2, x2, x3, x3, ..., x10, x10 } but this seems like a naive approach as it seems like the 2nd copy of the previous bin should be influenced also by the next bin value  y2 = (x1 + x2)/2 for example.

If x = 20 bins and y = 7 bins I can see it isn't fair to simply sample a value based upon a linear interpolation between data point as there might be 3 or 4 points on either side of the sample that should be a part of the probability for the resampled data.

I would also like to consider the possibility that the histogram is contained on the ends, so in the case of measuring water temperature below freezing isn't a likely temp for water nor above boiling for the standard cases.  I would like to be able to consider the probability beyond one or both extreme bins to be 0.

Is there a standard algorithm which can be coded in C++/C# or something in pseudo code that I can convert to code for the above re-sampling / re-sizing?


texts: Is it good to remove outliers from the dataset?

texts: Suppose you have some training dataset that you want to use to train some ML models, where targets are comprised between let's say 1 and 100. However, from the 4000 samples, there are few of them (less than 10) which have values out of the previous range, much higher than 100, let's say 300. Is it reasonable to ignore these samples and remove them from the data set, or should they be kept ? I saw people react differently, some of them say that they may hurt the model, while others say no as these samples give additional information to the model.


texts: Linear Model: How to deal with predictors with a lot of missing/small values?

texts:
I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes.

All these predictors have the same unit, as they are all car usage percentage.

The problem comes from the fact that some zip codes have very 'inactive' car usage: the values is often 0, or much smaller compared with others.

When I use general linear models, such as LASSO or ridge, I usually standardize my data first. Through standardization, these 'inactive' predictors actually got amplified.

I am wondering if there is a systematic way to restrict the beta's of these inactive predictors.

Does anyone have any insight?


texts: How Hyper-linked library vs traditional library differs from each other as ML problem?

texts:
Traditional library can be understood as a system, that archives the collective information from the mediums produced by our society, by indexing them to shelves. It is assumed that libraries have single ontology by which the books can be categorised to a single shelf. Many machine learning research papers exist for this problem.

Hyper-linked library could be understood in such way, that instead of a medium having one place in the ontology, there could be multiple. Imagine a book as a symbolic link to the physical book.

This aspect changes the machine learning problem slightly as instead of forming a clear hierarchy and simple tree structure, the hyper-linked library is a multi-label classification problem instead of hierarchical multi-class classification problem.

What are the important aspects that one has to consider when moving from traditional library problem towards a hyper-linked library problem? Links to research papers more than welcome!


texts: Recommender system that matches similar customers with similar highly rated products?

texts:
I have a dataset of 1,000 customers that bought 20 distinct phones and rated them 1-5. I have several demographic attributes for these customers (gender, age). My website offers 100 distinct devices, each with several attributes (screen size, case material, operating system, price).

For new customers, I want to recommend a ranked list of 5 devices they might enjoy. This might be based on 1) the new customer having a similar demographic profile to the 1,000 customers that gave ratings and 2) devices that were highly rated or had similar attributes to the devices that were highly rated by a similar customer.

All the examples I’ve found show how to make recommendations to similar customers for products that were rated highly by customers of similar profiles. I want to do this without limiting recommendations to the products that were rated. Instead, I want to build a system that might recommend phones I have ratings for, and also phones that haven’t been rated (ex: the phone is new, I need to get rid of inventory, or I want them to try something they might enjoy more) based on their similarity to other highly rated products by similar customers.

What is a good approach to build a recommended system for this scenario? I’ve read about content-based, collaborative filtering, and hybrid recommenders, but I can’t find a good example of this scenario. Is there a name for the type of system I am describing here?


texts: I want to prepare meaningful charts and polarization chart with the data I have

texts:
I am a bachelor student that is preparing a project for my network science course. I have some data that I got via questionnaire. This data shows people that they prefer studying online or in presence, their age, and gender. I want to prepare network charts and a polarization chart but I am not very experienced with these so I am not sure about what kind of chart I can prepare with the data I have. I am familiar with python so I can create them via it or some other tools. An example from my data;
id,preference,gender,age
1,9,1,26
2,2,2,22
3,0,2,24
4,9,2,24
5,10,2,21
6,9,2,20
7,5,2,24
8,5,1,23
9,8,2,26
10,8,1,26

Preferences goes from 0 to 10. It is the level of prefrences. 0 means online, 5 means notr, 10 means in presence and between them are the power level of their preference like 8 means the person gives 5 point to in presence while he/she gives 2 point to online. 10 means person gives 5 point to in presence and 0 to online. I hope I am clear here. The most imporant thing I want to build is the polarization chart that shows their preference but I have no idea about how to do it. Here is an example of polarization chart;


texts: Why does the BERT NSP head linear layer have two outputs?

texts:
Here's the code in question.

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491

class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score

I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?


texts: what other metrics can i use to estimate quality of the model predicting income range - interval estimation task?

texts:
I trained a model that predicts customer's income given the features:

age, declared income
number of oustanding instalment, overdue total amount
active credit limit, total credit limit
total amount

The output is a prediction: lower-upper bound for a customer: e.g. [8756-9230]
Metrics used:

NIRDM - not in range distance mean - how far the value is from the
closest bound (on average) for values out of range(similar to true
negative)
in-interval - percent of tested values that actually
happen to be within the range(similar true positive)

e.g. NIRDM = 1.37, on avg values deviate by 0.37, less is better.
my question: what other metrics can be used to estimate the accuracy of the model.
I dont have deeper expertise on this 'range estimation task', maybe confidence interval?


texts: HOW TO: Deep Neural Network weight initialization

texts:
Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train.
 To ease many of the problems one might:

Normalize && handpick quality data
choose a different training algorithm (e.g RMSprop instead of Gradient Descent)
pick a steeper gradient Cost function (e.g Cross Entropy instead of MSE)
Use different network structure (e.g Convolution layers instead of Feedforward)

I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better:
Glorot and Bengio (2010)

for sigmoid units: sample a Uniform(-r, r) with $r =
   \sqrt{\frac{6}{N_{in} + N_{out}}}$
or hyperbolic tangent units: sample a Uniform(-r, r) with $r =4
   \sqrt{\frac{6}{N_{in} + N_{out}}}$

Is there any consistent way of initializing the weights better?


texts: Time of trainig vs time of prediction, which one is used during classification algorithms comparison?

texts:
I need to use many algorithms for making a binary classification, such as Logistic regression, SVM, XGBoost, CatBoost, ... I get an interesting improvement but All of those algorithms (except LR) take a very long time for training data.

So I need to know if I must indicate training time in my comparison or it is doesn't need to be indicated and predicting time is enough ??

And how many time tuning parameters and fitting training should be done in a real application? does it done each time we insert a new row on data?


texts: How to analyze CNN model summary and improve it?


texts: is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

texts:
I am learning SVD by following this MIT course

The lecturer is trying to normalize a vector

$${\begin{bmatrix}
-3\\
1\\
\end{bmatrix}}$$

to

$${\begin{bmatrix}
\dfrac{-3}{\sqrt{10}}\\
\dfrac{1}{\sqrt{10}}\\
\end{bmatrix}}$$

I tried this with Python NumPy

np.linalg.norm(v1,ord=2,axis=1,keepdims=True)

and got

array([[3.],
       [1.]])

I would like to get something like this

[[-0.9486833 ],
[ 0.31622777]]

is there a way with Python (for instance, NumPy) to do the job? any other 3rd party library is also appreciated.


texts: what is difference between fit and fit_transform in sklearn while applying feature scaling

texts:
I have seen few post related to this question but i am not quite clear about my confusions as mention bellow.
I have some confusion related to fit and fit_transform.
suppose, I have X_train and X_test data, and let my scaling function is standard scalar.
I am using following code for scaling,
sc_X =StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.fit_transform`
My question is, if i use same scalar on bot trainin and testing data, wont it creat problem of data leakage?
What if I use the code like below,
sc_X_train =StandardScaler()
sc_X_test =StandardScaler()
X_train = sc_X_train.fit_transform(X_train)
X_test = sc_X_test.fit_transform(X_test)
Does the both code give different results?


texts: Include label for each time step to allow gradient flow in LSTM

texts: Say I want to do a many-to-one classification task such as sentiment analysis with a sequence of ~100 elements using a LSTM. Might it help to train the network using a many-to-many setup instead to allow the gradients to flow more easily to the initial elements in the long sequence? Each $y_t$ would just be a replica of the true value $y$.


texts: How to Calculate semantic similarity between video captions?

texts:
I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences.

For example, the captions for one video are as follows:
These captions are for the same video only. However, reference sentences have been broken down with respect to different segments of a video.

Reference sentences (R):

A man is walking along while pushing his bicycle.
He tries to balance himself by taking support from a pole.
Then he falls on the sidewalk along with the pole and the bicycle with him.

Candidate Caption generated (C):

A person is trying to use a pole to push off his bike ride but ends up falling down.

I want to calculate a similarity score between each pair.
That is, (R1,C), (R2, C) and (R3, C)

What is the best method?

I tried using TF-IDF and then Cosine similarity. However, that only got the word matching. I want lexical and semantic accuracy between these sentences to estimate how accurately the sentence C has been written.

You can refer the code I have done till now here

I understand I need to tokenize, do word embedding, semantic analysis and then some similarity metric but not sure? In which order and which algorithm is best suited for which?


texts: Extracting tokens from a document: applying Deep Learning or Classification?

texts:
I have a legal document from Law. That document is 4-pages of evidence from the plaintiff. I want to identify the Dates, Addresses and Financial transactions in that document.

Can I apply deep learning, the data with me is very small, on just one 4-page document, or should I apply Text Classification to solve my problem?


texts: Problem in drawing lines with multiple colors while using subplots

texts:
I am trying to plot the track of cyclones over a map using cartopy. I am able to do the same while using only a single axis, but while I am trying to the same in subplots I am getting the error as "ValueError: Can not reset the axes.  You are probably trying to re-use an artist in more than one Axes which is not supported".I will attach my code. If anyone can help by saying where I am doing wrong or what I need to implement more to get the desired figure it will be much appreciated. I will attach the code that I have made.
import numpy as np
import matplotlib.pyplot as plt;
from matplotlib.collections import LineCollection
from matplotlib.colors import ListedColormap, BoundaryNorm
import pandas as pd;filename=r'E:\tracks.xlsx';import cartopy.crs as ccrs
import matplotlib.pyplot as plt
# =============================================================================
df=pd.read_excel(filename,sheet_name='kyarr')
x = df.lon
y = df.lat
dydx = df.wind
# =============================================================================
points = np.array([x, y]).T.reshape(-1, 1, 2)
segments = np.concatenate([points[:-1], points[1:]], axis=1)

#--- plotting
fig, ax = plt.subplots(ncols=2,figsize=(10, 5),subplot_kw={'projection': ccrs.PlateCarree()})
ax[0].coastlines() # draw coastlines
ax[0].set_extent([50, 75, 5, 25]) # zoom into India
ax[1].coastlines() # draw coastlines
ax[1].set_extent([50, 75, 5, 25]) # zoom into India
# =============================================================================
#levels=list(np.arange(20,140.1,10))

cmap = ListedColormap(['lightgray', 'black', 'yellow', 'orange', 'magenta', 'lightblue',
                       'red'])
norm = BoundaryNorm([17, 27, 33, 47, 63,89,119,120], cmap.N)
cmap.set_over('red')
lc = LineCollection(segments, transform=ccrs.PlateCarree(),
            cmap=cmap, norm=norm, linewidth=3)
lc.set_array(dydx)
line = ax[0].add_collection(lc)
line = ax[1].add_collection(lc)
cb=fig.colorbar(line, ax=ax,shrink=0.70,extend='max');cb.ax.tick_params(labelsize=10)


texts: when to use certain metrics for splitting decision trees?

texts:
So I just very recently learned about decision trees, and the different metrics for determining the best split when training the tree. I cannot seem to be able to find anything on which metric to use for certain situations, or which one is better to use for certain kinds of data?

The ones I'm comparing are:

Chi-Square
Gini Impurity
Gini Index (Gini Coefficient)
Variance
Information Gain
Information Gain Ratio

When should I use which ones, or is there an insignificant difference between all?


texts: Why my classification accuracy is high for both training and testing data?

texts:
I have a dataset with 10 features and 1 binary classification target. I tested this dataset with decision tree classifier. I did some basic check like missing values but the data looks clean. My classification accuracy for both training and testing data is really high and it's looks suspicious. I would like to know whether I am doing any mistake or is there any way to explain why the accuracy is too high?
Can anyone advise me here?
import pandas as pd
from sklearn.model_selection import KFold, StratifiedKFold, RepeatedKFold, RepeatedStratifiedKFold, cross_validate, train_test_split
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor

df = pd.read_csv('https://docs.google.com/spreadsheets/d/e/2PACX-1vSvUmbtHUh2e0iYj7nMDaP8Tf_pCnCa-HrWwAmaxrERxxvd2y_5qxuSP10t6db4RUSjTdOi9WshZhoR/pub?output=csv')

input_selected_features = df.drop(labels = 'Target', axis = 1)
target_selected_feature = df['Target']
X_train, X_test, y_train, y_test = train_test_split(input_selected_features, target_selected_feature, test_size = 0.2, train_size = 0.8, random_state = 101)

# k fold cross validation scores
kf = KFold(n_splits = 10, shuffle = True)
cv_results = cross_validate(estimator = DecisionTreeClassifier(), X = input_selected_features, y = target_selected_feature, cv = kf, scoring = ['accuracy', 'f1'], return_train_score = True)

# print(cv_results)
print('training accuracy - ',cv_results['train_accuracy'].mean())
print('testing accuracy - ',cv_results['test_accuracy'].mean())
print('training f1 score - ',cv_results['train_f1'].mean())
print('testing f1 score - ',cv_results['test_f1'].mean())
```


texts: Pre-trained model is unable to detect new license plates properly

texts: I have downloaded a license plate recognition project from GitHub. It is able to recognize images in the project repository. However, it does a poor job at detecting new images. Do I need to retrain the model?


texts:
I cannot install the Text add-on.

How can I fix this?