texts: Cosine Similarity: Works with TF-IDF Vectors OR with Probability Vectors?

texts:
Using Cosine Similarity is a common method to calculate Semantic Textual Similarity. And it is particularly useful when comparing Sentence Embeddings provided by the Universal Sentence Encoder.
However, the loss function used for training the USE model is based on Angular Difference, which can explain why Cosine Similiarity would be an effective alternative:

As shown Eq. 1, we first compute the cosine similarity of the two
sentence embeddings and then use arccos to convert the cosine
similarity into an angular distance.
(...)
We find that using a similarity based on angular distance
performs better on average than raw cosine similarity

Now, what about other Vectorial Representations of Text (not necessarily restricted to Sentence Embeddings), such as TF-IDF or even Probability Vectors (using the Probability Distributions as values for its dimensions)? Would Cosine Similarity be similarly effective to calculate Semantic Text Similarity across two vectors created using those mechanisms?


texts:
In slide 61 of the NLP text, to smooth out the n-gram probabilities, we need to find the lambdas the miximazies a probability to held-out set given in terms of M(λ1, λ2, ...λ_k). What does this notation mean please? Also, it says that "One way is to use the EM algorithm, an iterative learning algorithm that converges on locally optimal λs". Can someone refer me to a good example? Say the training text is "Sam I am Sam I do not eat."
and the held-out data is "I do like Sam I am sam"

What is the objective function please?


texts: What does the term "proportional to" mean in Bayes Equation?

texts:
I don't have a background in maths so I sometimes get confused by basic definitions. Let's for instance consider Bayes Theorem in Bayes Data Analysis:

$P(\theta|\textbf{y}) \propto P(\theta) P(\textbf{y}|\theta)$

How should this equation be understood? What does it mean that the RHS is proportional to LHS?


texts: First steps when analyzing a company's data

texts: I'm not sure if this question is appropriate for this forum, so excuse me if it's not (if not, any suggestions on where might be a better place would be very much appreciated). I'm currently an undergrad in a quantitative field, and for the summer, I've been given an opportunity to do a data project by the company I am working for. I'm not really sure where to start. I had a conversation with one of the business owners today, in order to get a better handle on how the business works, and what kind of data they have. We talked a little bit about what sorts of questions they have, and what sort of things would be nice to know. I guess that seems to be the main question: What questions to ask? My initial thoughts are to first just look at the data via traditional descriptive stats methods (histograms, scatter plots etc....), and maybe that creates some ideas. If anyone has some tips, or even some good links (yes, I have already Googled it quite a bit), I would be grateful. Thanks.


texts:
I have a 4 datasets with user features, item features, user-item rating and User-item link data. I'm trying to build a recommender system to recommend top 10 items to the user by maximizing NDCG as the metric.
So far I've used the user-item rating data with Implicit ALS using sparse matrix approach. I wanted to know how could the other data be trained and analyzed? Any good resources for similar problem?


texts: Machine learning solution approach to match loan repayments

texts:
I'm relatively new to the AI/ML space, but come from a programming background.

The problem: I have a dataset of users transactions who have taken short-term loans from a single loan provider and I need to match the loan repayments to a particular loan receipt.
The sample dataset is below:

transactionDate | amount | creditDebit

2017-09-01      | 500 | Credit
2017-09-08 | 250 | Debit
2017-09-17 | 294 | Debit
2017-12-11 | 300 | Credit
2017-12-15 | 150 | Debit
2018-01-07 | 200 | Credit
2018-01-12 | 398.8 | Debit
2018-01-19 | 200 | Debit
2018-02-02 | 150 | Debit

Here the 'Credit' values for creditDebit column indicate a received loan and 'Debit' indicates a repayment of some loan received previously.

As you can see the first 3 rows are pretty easy to match because there is 1 Credit (amount of 500) followed by 2 Debits (amounts 250 and 294).

From 4th row onward is where i cannot figure out how best to match.

2017-12-11 there is a Credit transaction for a loan
2017-12-15 is a partial repayment of this loan
Then is another Credit for a new loan
2018-01-12 is a partial repayment for the 2 loans previously
The final 2 Debits are partial repayments also

I have been able to manually determine which repayment makes most sense for which loan, but trying to write it out into an if-else function would be very hard in my opinion and not the best.
For example the hardest part is to determine the Debit on 2018-01-12 which is a combined repayment transaction for both of the previously received loans. I have manually calculated a reasonable split of this Debit for the loans at 170 for the loan received on 2017-12-11 and 228 for loan on 2018-01-11.

Is there some model/algorithm that I could use to match which debit transactions are most likely to be repayments of which credit transactions and if the repayment transaction looks like a compound repayment of multiple previous loans - then what would be a reasonable split of the debit to satisfy the credit transactions?

I don't know if this makes sense to you..

I tried thinking about how I could use the clustering algorithms, but i don't I could since the clusters would not indicate matching of credit and debit transactions, especially the compound repayment debits.

Any ideas on how to tackles this are much appreciated!


texts: Can the extent of variability within a dataset be reflected through clustering?

texts:
As an example: I need to compare the extent of variability amongst houses belonging to 4 different architectural eras - I want to see how different the houses are within each group and then compare such variability between groups (example of hypothetical conclusion: the houses from the baroque era had the most variability when compared to the other 3 groups).
Different descriptive variables on all houses are available (area, number of rooms, number of floors etc).
I wish to use a technique which can take into account all variables.
Would it make sense to perform clustering within each individual group, on the basis that the higher the optimal number of clusters, the larger the extent of variability between houses within each group? Would hierarchical clustering fit?
If not, what would you suggest please?


texts:
I was given the task as follows,

  Scrape articles appearing in Times of India since 2010 on HIV and AIDS. Classify them using training a neural network of your choice. Find patterns in those articles, analyze the data.

I have done the scraping part, But I have no clue what to do now with the data I have extracted. I searched for hours on internet but could not find any help.
I know theory part about the neural network but lack knowledge for implementing it on a text. It would be of great help if someone can suggest me any method or provide me some helpful link.


texts: My neural network 'learns' to estimate each class equally

texts:
I've implement a neural network. It's a fully connected back propogation network. I'm training it on this data set;

Iris Data Set

I'm using that data just as a test to check if my implimentation is working.

My neural network minimises it's cost function by giving an estimate of each class equally. In this case (Iris) there are three classes and after around 1000 iterations my network output is something like this;

0.3563204029871746
0.34553220619664271
0.36660774391053058

If I run it for 2000 iterations I get this;

0.33748041999368034
0.33555095097164173
0.33929502380142113

The network is almost certainly approaching 1/3 estimate for each class. This is obviously not what is desired, but I can't figure out where my algorithm is going wrong. I can post all my code/pseudocode if it would be useful, but I'm hoping someone has seen this before and can tell me where I'm missing something.

If it matters I'm using a logistic activation function.

Edit: Pseudocode

Ok here's my code;

**Sub DataInput**
Load in the iris text file
split file into lines
redim x(#lines)
redim y(#lines)
for each line
  datain = split line into columns (comma as separator)
  redim x(line #)(3)
  x(0) = datain(0)/7.9
  x(1) = datain(1)/4.4
  x(2).... etc.

  redim y(line #)(2)
  y(0) = 0
  y(1) = 0
  y(2) = 0

  select case datain(5) 'selects which type of flower
    case is flower1
      y(0) = 1
    case is flower2
      y(1) = 1
    case is flower3
      y(2) = 1
  end select
next line
**end sub**

**sub initialise**
'sets up array sizes for all required values
'sets all weights to random
**end sub**

**sub Train**
Do until cost function is below threshold
set DR = 0 for all l, i, j 'DR is the cumulative error value from each training example at each weight (l = layer, i = neuron in next layer, j = neuron in current layer)

for i = 0 to number of training examples
  call sub ForwardPropogate(i)
  call sub BackwardPropogate(i)
end for

for l = 0 to number of layers -1
  for i = 0 to number of neuron in next layer
    for j = 1 to number of neurons in this layer
       CapD(l)(i)(j) = 1/numExamples * DR(l)(i)(j) + lambda*theta(l)(i)(j)
    end for
    CapD(l)(i)(0) = 1/numExamples * DR(l)(i)(0) + lambda*theta(l)(i)(0)
  end for
end for

for l = 0 to number of layers -1
  theta(l) = theta(l) - alpha*CapD(l)
end for
loop

**end sub**

**Sub ForwardPropogate**(byval exNum as int) ' exNum is the training example being propagated

a(0) = x(exNum) ' a(n) is the activations of all neurons in layer n
a(0) = leadingBias(a(0)) ' leading bias adds a 1 to the start of the array for the bias neuron
for i = 1 to number of layers -1
  z(i) = theta(i-1) * a(i-1)
  a(i) = g(z(i)) ' g is the activation function
  a(i) = leadingBias(a(i))
end for
z(number of layers) = theta(number of layers -1) * a(number of layers -1)
a(number of layers) = g(z(number of layers))

**end sub**

**Sub BackwardPropogate**(byval exNum as int)
delta(number of layers) = a(number of layers) - y(exNum)
for i = number of layers - 1 to 1 step -1
  delta(i) = [transpose(theta(i)) * delta(i+1)] .*g'(a(i))    g' is the derivative of the activation function
end for
for i = 0 to number of layers -1
  DR(i) = DR(i) +delta(i+1) * transpose(a(i))
end for

**end sub**


texts: Next events prediction based on previous events

texts:
I have data set of sequences of user executed commands sorted in the order of its occurrence. The data looks like this.

user1 : cmd1->cmd2->cmd3->...->cmdn
user2 : cmd1->cmd2->cmd3->cmd5->...->cmdn
user3 : cmd6->cmd2->cmd4->cmd5->...->cmdn

So the idea is to predict the next user command ordered by probability based on the previous command or sequence of commands similar to the clickstream prediction.

I have tried 1st-order Markov Chain but it is only based on previous command, and higher order Markov Chain would not be efficient because the sequences could be really long, with many different numbers of command.

Is there any other way to solve this kind of problem?


texts: How to fix class imbalance in dialogue (text) time series data?

texts:
I have a dataset that looks like this:

df.head(5)

 data                                                     labels
0  [0.0009808844009380855, 0.0008974465127279559]             1
1  [0.0007158940267629654, 0.0008202958833774329]             3
2  [0.00040971929722210984, 0.000393972522972382]             3
3  [7.916243163372941e-05, 7.401835468434177e243]             3
4  [8.447556379936086e-05, 8.600626393842705e-05]             3

The 'data' column is my X and the labels is y. The df has 34890 rows. Each row contains 2 floats. The data represents a bunch of sequential text and each observation is a representation of a sentence. There are 5 classes.

I am training it on this LSTM code:

data = df.data.values
labels = pd.get_dummies(df['labels']).values

X_train, X_test, y_train, y_test = train_test_split(data,labels, test_size = 0.10, random_state = 42)

X_train = X_train.reshape((X_train.shape[0],1,X_train.shape[1])) # shape = (31401, 1, 5)
X_test = X_test.reshape((X_test.shape[0],1,X_test.shape[1])) # shape = (3489, 1, 5)
### y_train shape =  (31401, 5)
### y_test shape =  (3489, 5)

### Bi_LSTM
Bi_LSTM = Sequential()
Bi_LSTM.add(layers.Bidirectional(layers.LSTM(32)))
Bi_LSTM.add(layers.Dropout(.5))
# Bi_LSTM.add(layers.Flatten())
Bi_LSTM.add(Dense(11, activation='softmax'))

def compile_and_fit(history):

    history.compile(optimizer='rmsprop',
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])

    history = history.fit(X_train,
              y_train,
              epochs=30,
              batch_size=32,
              validation_data=(X_test, y_test))

    return history

LSTM_history = compile_and_fit(Bi_LSTM)

The model trains, but the val accuracy is fixed at 53% for every epoch. I am assuming this is because of my class imbalance problem (1 class takes up ~53% of the data, the other 4 are somewhat evenly distributed throughout the remaining 47%).

How do I balance my data? I am aware of typical over/under sampling techniques on non-time series data, but I can't over/under sample because that would  mess with the sequential time-series nature of the data. Any advice?


texts: Time-series classification using convolutional neural networks

texts: I am working with time-series data. How can we use convolutional neural networks (CNN) for time-series classificaiton?


texts: Loss function returns x whereas tensorflow shows validation loss as (x+0.0567)

texts:
I have a custom loss function. In order to experiement how the loss is calculated during valiation, I update the loss function as follows:

def custom_loss(y_true, y_pred):
    return 0 * tf.reduce_mean(input_tensor=-tf.reduce_sum(input_tensor=y_true_1 * tf.math.log(y_pred_1), axis=-1)) + 1

Multiply by 0, to make the data dependent loss value as always 0.
Add a constant value like 1 to it, so that the loss function always returns a value of 1.

Now when I run model.evaluate on my data, tensorflow shows me the loss value as 1.0567.

To further experiment, I updated the constant value being returned by loss function from 1 to other constant values. Below you can see the table:

constant 0 => loss 0.0567
constant 1 => loss 1.0567
constant 2 => loss 2.0567

I was expecting the final calculated loss to be same as the constant value I passed. But that's not the case. What is the reason for a constant difference of 0.0567?

I have verified it across three versions of tensorflow: 1.14.0, 1.15.0, and 2.1.0.


texts: How to write custom de-identification algorithm in Python?

texts:
I have tried a simple algorithm to anonymize my data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values. The data sample is available here

import pandas as pd
import uuid as u
import datetime as dt

# generate a pseudo-identifier sequesnce using python random number generator library uudi.

 def uudi_generator(length):
    uudi_list= list()
    i=0
    while i < length:
        uudi_list.append(u.uuid4())
    i+=1
    return uudi_list

#import original originaL dataset
dataset = pd.read_csv('bankcredit-data.csv')

# pseudo identifier
sLength = len(dataset['housing'])
dataset.insert(0, 'uuid', pd.Series(uudi_generator(sLength), index=dataset.index))

# Transaction record attached to the original
dataset.insert(0, 'transaction_date', pd.Series([dt.datetime.now]*sLength, index=dataset.index))

 #transcation record is attached to originaL data file
dataset.to_csv('bankcredit-data.csv')

#delete identifiabLe record from dataset
del dataset['firstnamme']
del dataset['lastname']

# export  de-identified dataset as csv to be shared with the user
dataset.to_csv('deidentified-data.csv')


texts: Keras model has a good validation accuracy but makes bad predictions

texts:
I have this model which takes 9000 images in a dataset containing 96 categories of traffic signs, each category has more or less the same number of images (about 50). This is the model I made but somehow the predictions are really bad even if the validation accuracy is really high (99%). I can't figure it out what's wrong. I read some possibilities are: overfitting, cnn is too big for the dataset I use, I train on the same data I use to validate the model. How can I understand where I am failing at?
JSON_PATH = "/Users/user/Documents/ML Projects/classname.json"
DATASET_PATH = "/Users/user/Documents/ML Projects/Dataset"

CLASSNAME_SIZE = 96
IMG_SIZE = 48

with open(JSON_PATH) as classnameJSON:
    CLASSNAME = json.loads(classnameJSON.read())

trainingData = []
X = []
Y = []
X_val = []
Y_val = []

def loadTrainingData():
    for instance in range(CLASSNAME_SIZE):
        joinedPath = os.path.join(DATASET_PATH, str(instance))
        label = str(instance)
        for img in os.listdir(joinedPath):
            try:
                img_array = cv2.imread(os.path.join(joinedPath,img), cv2.COLOR_BGR2RGB)
                new_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))
                trainingData.append([new_array, label])
            except Exception as err:
                pass

loadTrainingData()
print(len(trainingData))

def distributeTrainingData():
    for img, label in trainingData:
        X.append(img)
        X_val.append(img)
        Y.append(label)
        Y_val.append(label)

distributeTrainingData()
print("distributing data")

X = np.array(X, dtype='float32')
Y = np.array(Y, dtype='float32')
X_val = np.array(X_val, dtype='float32')
Y_val = np.array(Y_val, dtype='float32')
print(len(X))
print(len(Y))

def cnn_model():
    model = Sequential()
    model.add(Conv2D(32, (3, 3), padding='same',
                     activation='relu'))
    model.add(Conv2D(32, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(64, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(128, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(keras.layers.InputLayer(input_shape=(X.shape[1])))
    model.add(Dense(512, activation='relu'))
    model.add(Dropout(0.25))
    model.add(Dense(CLASSNAME_SIZE, activation='softmax'))
    return model

#model = cnn_model()
model = keras.models.load_model('traffic_signs.model')

model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X, Y, batch_size=32, epochs=30, validation_data=(X_val,Y_val), shuffle=True)
model.save('traffic_signs.model')```


texts: Why does vanilla transformer has fixed-length input?

texts:
I know that in the math on which the transformer is based there is no restriction on the length of input. But I still can’t understand why we should fix it in the frameworks (PyTorch). Because of this problem Transformer-XL has been created.

Can you explain to me where this problem is hiding, please?


texts: How to apply AdaBoost to more "complex" (non-binary) classifications/data fitting?

texts:
In this article, Chris McKinlay says he used AdaBoost to choose the proper "importances" of questions he answered on okcupid.

If you haven't read and don't want to read the article, or are unfamiliar with okcupid and the question system, here's the data and problem he had:
The goal is to "match" as highly as possible with as many users as possible, each of whom may have answered an arbitrary number of questions. These questions may have between 2 and 4 answers each, and for the sake of simplicity, let's pretend that the formula for a match% $\ M $ between you and another user is given by

$\ M = Q_a/Q_c $

Where $\ Q_c $ is the number of questions you and the other user have in common, and

$\ Q_a $ is the number of questions you both answered with the same value.

The real formula is slightly more complex, but the approach would be the same regarding "picking" a correct answer (he actually used boosting to find the ideal "importance" to place on a given question, rather than the right answer).
In any case, the point is you want to pick a certain value for each question, such that you maximize your match% with as many users as possible - something you might quantify by the sum of $\ M $ over all users.

Now I've watched the MIT course on AI up to and including the lecture on boosting, but I don't understand how you would apply it to a problem like this. Honestly I don't even know where to begin with choosing rules for the weak learners. I don't have any "rules" about what values to choose for each question (if the user is under 5'5, choose A, etc) - I'm just trying to fit the data I have.

Is this not the way boosting is supposed to be used? Is there likely some other optimization left out of how he figured this out?


texts: Visitor's probability to purchase on eCommerce site, based on aggregate historic data

texts:
On an eCommerce website we want to create some personalization for visitors who are more likely to make a purchase.

Let's assume we only have one single item for sale.

The likelihood should be based on data that we have, for example:

Price of the item when the customer visited (it may vary)
Average customer reviews rating of the item
If a discount coupon was offered or not
If certain words were present in the item title or description
etc.

Our plan is to create a DB that will contain each visitor's visit data (the above data), and indication if eventually he made a purchase or not.

Once we have at least a few purchases, we need to somehow look for a pattern among all these cases, and based on this pattern we can later forecast the likelihood.

For example, if at least in 50% of the cases that the item was purchased, its price was no more than $100, than we will make personalization only on cases where the item is being sold at a max price of $100.

Does this make sense?

How should we do it technically? From both DB structure concerns, and website speed concern -

Should we collect each variable in its own DB column, or is it better to create some kind of long string in the browser session like p:99-r:4,  (which means: price $99, rating: 4) and then later break this data down using another script?

Is there some kind of library that can analyze this data and show it maybe in graph, like some fancy fronted, or does this project need to be made from scratch?

What kind of technical skills does the person who do it need to have? Is this related to machine-learning?


texts:
I'm looking for a Python library that can compute the confusion matrix for multi-label classification.

FYI:

scikit-learn doesn't support multi-label for confusion matrix)
What is the difference between Multiclass and Multilabel Problem


texts: Creating a single number from a numpy array - Python

texts:
I am working on a gender classification project. I am extracting the pixels of an image using a Numpy array in Python, similar to the one below:

[[[129 155 191] [123 150 185] [120 149 183]]

How can I use these values to extract a single meaningful number to be used in a csv file for K-Nearest Neighbor Algorithm. For example single number like

0.34232

In summary, how can I turn [[[129 155 191] [123 150 185] [120 149 183]] into a single number like 0.34232


texts: Can reducing the number of classes in multi-label classification increase performance?

texts:
This is more of an open question with people which have experience in this.  I'm working on a multi-class multi-label classification for chest x-rays. I would like to know how much can reducing the number of classes increase performance for remaining classes before moving forward and making big changes to the training pipeline.
For the moment, there's about 50 classes with AUC performance varying between 0.7 and <~ .99. Final outputs used are some tweaked sigmoids and loss function is based on cross entropy as it's multi class multi label. However, not all 50 classes are equally important. One option would be to give class weights. But I'm also thinking about just dropping or merging some classes instead. Hence my question:
Has anyone experience a significant increase in performance for the remaining classes by merging / dropping-out some classes ?
Thanks !


texts:
I'm using Google Analytics on my mobile app to see how different users use the app. I draw a path based on the pages they move to. Given a list of paths for say a 100 users, how do I go about clustering the users. Which algorithm to use? By the way, I'm thinking of using sckit learn package for the implementation.

My dataset (in csv) would look like this :

DeviceID,Pageid,Time_spent_on_Page,Transition.

ABC,Page1, 3s, 1->2.

ABC,Page2, 2s, 2->4.

ABC,Page4,1s,4->1.

So the path, here is 1->2->4->1, where 1,2,4 are Pageids.


texts: What to report in the build model, asses model and evaluate results steps of CRISP-DM?

texts:
I would greatly appreciate if you could let me know what to report in the following steps of CRISP-DM?

Build Model: what should be reported for parameter settings, models and model description? I used grid search to tune hyperparameters.
Assess Model: what should be reported for model assessment and revised parameter settings?
Evaluate Results: what should be reported for assessment of data mining results?

In fact, I used just Logistic regression to do a classification task using a procedure like what is depicted below:

Could the final model evaluation on test data set be considered as a part of Evaluate Results or it should be done on new data?


texts: Looking for a 'CITY, STATE' within a body of text (from a CITY-STATE database)

texts:
I'm looking for an optimal way to search a large body of text for a combination of words that resemble any CITY, STATE combination I have in a separate CITY-STATE database.

My only idea would be to do a separate search against the body of text for each CITY, STATE in the database, but that would require a lot of time considering the amount of CITY, STATE combinations the database has in it. The desired result from this query would be to pull a single CITY, STATE for each body of text I am analyzing to tell the geographical side of the story for this data subset.

Anyone know of an optimal way/process to do such a query?


texts: How to measure uncertainty and prediction errors for a naive Bayesian Classifier?

texts:
I have a small dataset of 30 rows and 5 columns (4 features and 1 class). The classifier is used to give the likelihood of occurrence of an incident. thus, the class variable gives the probability of occurrence.

How can I measure the uncertainty and prediction errors of my classifier?

I have read a post here about measuring the uncertainty of prediction. However, it is tacking the issue of multiclass classification!


texts: Nested cross-validation generalization error for multiple models

texts:
I am referring to this question:

Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

In the answers it shows that nested cv can estimate the generalization error of hyperparameter optimization for different algorithms.
But in my opinion the choice between different algorithms is also an optimization process, which leads to generalization errors. Therefore, either the algorithm choice should be part of the inner cv or another third cv would have to be introduced to evaluate the error for the algorithm choice. Is this a correct assumption ?


texts: Does object detection do a better job at image classification than image classification

texts: I read in an article that object segmentation can do object detection better than object detection algorithms. I assume this is because there is more detailed information in the annotation images. I was wondering if this is true for object detection and multiclass image classification as well. If you just take the label and quantity of each label from object detection, does it perform better than multiclass image classification?


texts:
I would like to automate periodic report writing based on data. Given one/some data tables, the machine should output texts like Stock A rose by 10% this year and hit 5 year high on 2019-12-01, or we made a large profit in sector B.

I can find the subject automatic text summarization, but seems it's about reading text and shorten it to its key sentences. Not exactly data summarization.

Could someone recommend some book/paper/video/MOOC on text generation based on data?


texts: Time series data and ML - separating training/test data

texts:
I am using XGBoost to try to predict the direction of the stock market based on social media sentiment. Having read through some studies, I was planning to separate the training/test data by time period, e.g. use 2014-2016 data for training and 2016-2018 data for testing.
Does that make intuitive sense given the nature of the data I am using?
I am happy to provide any further details which would be helpful, thank you.


texts: tf.data.Dataset object and DataGenerator class with same output yield different training results

texts:
This week I converted my old datagenerator for audio to the new(ish) tf.data.Dataset api.  The dataset object seems to be working properly and when tested yields identical data, in the same order, as my previous generator.  But if I use the dataset for training the results are dramatically different.
When I use the tf.data.Dataset for training I get low validation accuracy.  There is no difference if I swap the data generator and the dataset for validation.
The two lines in the attached plot ending around 0.4 accuracy are using my old data generator for training, the two lines that diverge and have very low validation accuracy are using the dataset for training.

This is how I verified that the output of the Dataset, and the DataGenerator are identical:
    sd = SimpleDataset('../../input/birdclef-2021/train_short_audio',
                       name='train',
                       batch_size=BATCH_SIZE,
                       files_list=train_files,
                       is_test=True)
    ds = sd.get_dataset()
    # make it an iterator so I can get results
    ds_iter = ds.as_numpy_iterator()
    # create DataGenerator
    print('Creating DataGenerator...')
    from DataGenerator import DataGenerator
    train_generator = DataGenerator(wav_paths=train_files,
                                    labels=train_labels,
                                    sr=sr,
                                    dt=dt,
                                    n_classes=len(classes),
                                    batch_size=BATCH_SIZE,
                                    shuffle=False)
    # compare ouptuts
    matches = 0
    for step in range(int(steps)):
        audio, label = next(ds_iter)
        audiogen, labelgen = train_generator.__getitem__(step)
        if np.array_equal(audio, audiogen) and np.array_equal(label, labelgen):
            matches += 1
    print(f'{matches}/{steps} steps were equal')

The result was:
22712/22712.0 steps were equal

For training, I use the same code to fit the model:
history = model.fit(train_ds,
                    validation_data=val_ds,
                    epochs=3,
                    callbacks=callbacks)

or
history = model.fit(train_generator,
                    validation_data=val_ds,
                    epochs=3,
                    callbacks=callbacks)

What else could be going on?
I’m sending the same data in, but based on the tool used to send in the data I get consistently different results.  I must be missing something about the Dataset API.  From the low validation rate it seems clear that the Dataset is what is causing the problem.
Any help would be appreciated.


texts:
I drove myself into a corner with this, can someone please explain?

I feel I'm missing something obvious...

If, for LSTM, each layer is trained with inputs from t and t-1, than that'd mean that if I've got a training set of a 10 000 observations, the network is trained to get 10 000 observations and produce a result as a function of all of them. If I use it on a test set of say 1 000 observations, why would it work?

Or if I want to make a prediction, from a single observation, whz would that work at all?

Should, in the case of LSTMs, the train test (in the toy example above) be 10 000 observations long (i.e. 9 000 old 'train' observations and 1 000 new 'test' ones)?


texts: Are there any Python libraries for predicting the closest value to a correct label out of a variable-size list of possible label values?

texts:
Imagine I have the following dataset:

|---------------------|------------------|------------------|
|      Feature 1      |  Feature 2       | possible labels  |
|---------------------|------------------|------------------|
|          1          |         5        |    {1, 4}        |
|---------------------|------------------|------------------|
|          2          |         2        |     {2}          |
|---------------------|------------------|------------------|
|          4          |         3        |  {3, 5, 6}       |
|---------------------|------------------|------------------|

and my goal is to be able to predict a value that is the closest to the set of possible labels. That is, this is a regression problem.

(A sub-optimal solution I would also accept as an answer would be for my goal to be to predict a value that belongs to the set of possible labels, not taking into account the distance between the predicted value and the elements of the set.)

Does this have a name? Are there any Python libraries (for deep learning, if possible) that solve this?  Or should I define my custom loss function? If I have to define my custom loss function, could you suggest me a library/point me in the right direction? My main problem with that is that the set of possible labels has variable size! Thanks!


texts:
I am trying to estimate a parameter by repeated sampling (Monte Carlo simulation). Each time I sample, I have to run a particular operation on a data.table object. Here is an example:

visits.site <- site.visits.map[site.name %in% sample.sites,
                              .(length(unique(visits))),
                              by = week]

Since the number of samplings often run into millions, this is taking a lot of time. I have made the loop execute in parallel on the multiple cores of the CPU using the parApply function. This has appreciably improved the performance. (I ran it over 8 cores). Note: This is an embarrassingly parallel problem.

I want to improve the performance even further by running this on GPU. I have looked at gpuR but I am not sure if it has a function similar to parApply for GPUs. Are there any packages that have this functionality in R?

If there are some other ways in which I can improve the performance, please let me know.


texts: Why doesn't training RNNs use 100% of the GPU?

texts:
I wonder why training RNNs typically doesn't use 100% of the GPU.

For example, if I run this RNN benchmark on a Maxwell Titan X on Ubuntu 14.04.4 LTS x64, the GPU utilization is below 90%:

The benchmark was launched using the command:

python rnn.py -n 'fastlstm' -l 1024 -s 30 -b 128

How can I diagnose   what the bottleneck is?


texts: Data science and MapReduce programming model of Hadoop

texts: What are the different classes of data science problems that can be solved using mapreduce programming model?


texts: Is there an AI service that could be used to classify 30,000 different tools and parts?

texts:
I'm trying to build an image classifier where people can take a picture of a tool or part and have the image classified. Much like bixby or amazon's tool to do something similar, but with only 30,000 different items to classify instead of a huge shopping catalog.

I was hoping there would be an AI service that takes a lot of the work out as I am trying to avoid training my own CNN because the manpower necessary to take the amount of pictures I would need is not achievable for my project. I can definitely get the manpower to take some pictures, to the tune of maybe 300,000 to 1 million, but I don't believe that would be enough while training a brand-new CNN.


texts: bagging vs. pasting in ensemble learning

texts:
I am bit confused about two concepts.
From my understanding Bagging is when each data is replaced after each choice. so for example for each subset of data you pick one from population, replace it then pick one again, etc... and this is repeated for each subset of data.
But for pasting people say it is sampling without replacement however does that mean you can't have same data on any subset? I thought it picks one subset w/o replacement but when it replaces all data when picking another subset, no?


texts: Highly Imbalanced dataset fro classes more than 200

texts:
I have a text dataset where I need to train a classifier to classify the titles into categories. The dataset shape is more than 575000. There are 256 target classes here. The problem is the dataset is highly imbalanced. For target X1 it has 171793 records, X2 has 101575,........Xn-1 has 2, Xn has 2 records. Consider the target value counts are in decreasing order.

To handle with the imbalanced dataset, oversampling and undersampling works for multiclass say 3 classes. But in my case, there are 256 classes. How do I sample my dataset in this situation? How do I sample the dataset in a way so my model is stable for all the targets?

Do I have to remove the classes which have value counts 2 - 100 from this dataset? and apply undersampling/oversampling. Is there any approach to handle these type of situation?


texts: I can not find a pre-trained neural network model for object detection

texts:
A simple model is needed, but with good accuracy, mAP for detecting objects is preferably trained on COCO, preferably on ipynb (to run or train on google colab).

The models from the Tensorflow (Faster_rcnn, ssd, R-FCN, YOLO V2 ) will not fit, because need a custom model for example, because I need to develop my own model

Any help?


texts: Is it possible to use EinsumDense instead of multiple parallel Dense layers?

texts:
I have an input of shape (None, 20, 250), where 20 is my context window and 250 my embedding dimension.
I want to apply a different dense 250 -> 250 for each element in the context window.
The following code works and does what I want, but does not use Einsum:
x = tf.unstack(x, axis=1)
x = [layers.Dense(250)(i) for i in x]
x = tf.stack(x, axis=1)

If I understand Einsum correctly, the following code should do the exact same:
einsum = EinsumDense('abc,cbd->abd', output_shape=[20, 250], bias_axes='bd')
x = einsum(x)

The documentation for EinsumDense states that ab,bc->ac would be equivalent to a dense layer, so I figured this should work.
However I'm getting drastically different results when doing this.
What am I doing wrong here?


texts: What are the sparse and dense vector ? I cant undestand ,can you explain to me?please.Why do we use for?

texts: I am new to neural networks, embeddings, etc. I am struggling understanding things like sparse representation, embeddings, and especially sparse vectors. Could you explain these to me? Why do we need this vector? What is it?


texts: Text preprocessing on corpus in pipeline before Gensim word2vec training

texts:
I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this:
data = gensim.models.word2vec.LineSentence(corpus)

This creates an iterator over the lines of the corpus. The next step is training:
model = gensim.models.Word2Vec(data)

I'd like to lemmatize and POS-tag the corpus before training. I am planning to use NLTK WordNetLemmatizer and NLTK POS-tagger.
How should I do this in a pipeline?


texts: How to force a NN to ouput the same output given a reverse input?

texts:
I want to choose an architecture that can deal with an input symmetry.
As input, I have a sequence of zeros and ones, like [1, 1, 1, 0, 1, 0] and at the output layer I have N neurons that outputs a categorical distribution like [0.3, 0.4, 0.3].
How can I force a NN to ouput the same distribution when I feed its reverse copy, i.e [1, 1, 1, 0, 1, 0]?
A simple way just to learn twice:
feed straight [1, 1, 1, 0, 1, 0] -> [0.3, 0.4, 0.3]
feed reverse  [0, 1, 0, 1, 1, 1] -> [0.3, 0.4, 0.3]

Or maybe, there are more "elegant" ways? What type of architecture I should use or maybe I need to play with loss functuions?


texts: Create an Algorithm, that translates human language into code

texts:
I am an experienced programmer and want to get into ML. My final goal in ML is to create an algorithm, that converts written human language into a query language like SQL.

Example:

Input:
"Give me all clients, with an email"

Expected output:

SELECT * FROM Clients where email != null

Where should I look into, to achieve something like this sometime in the future?


texts: Where can I find dataset for word analogy task?

texts:
In the paper of Word2Vec by Thomas Mikolov and others, there is a accuracy report on the full Semantic-Syntactic data set. Where I can find this dataset or a related dataset for word analogy task?

Here is the paper link and I'm referring to table 5.
https://arxiv.org/abs/1301.3781


texts:
Let's say, I have the dataframe with numerical features A, B, C. I do not have the target variable but I extract the target variable Y from the features A, B, and C. E.g.
If values A<0.1*sum (A,B,C) and B 0.1*sum (A,B,C) and C<0.1*sum (A,B,C) then Y=1 or else Y=0.
Now I have features A,B,C and target variable Y for which I train model to predict Y. Is this data leakage? Is there anything wrong with this method?


texts: What should I do with the NaN values on this stock quote data?

texts:
I concatenated 3 stock quote data-frames all with date-time indexes.
However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent starting dates.

Should I just drop the rows with NaN and start the new data frame with the row where all have values or is there a way to fill them up? I'm planning on using the data to train a neural network that predicts future stock quotes.


texts: How does Doc2Vec treat numerical data which is a part of text data?

texts:
I have data containing both numbers and raw text related differently like:

The power of diesel generator is 15kva. I need a single phase generator.
Three phase generator required of 140 kva.
Need 70g/m^2 GSM a4 size copier paper.

I removed the numerical data and trained a classifier on vector obtained from Doc2Vec and got around 80% accuracy.

Is there a way to take in account the numbers (here 15,140,70) involved in text while creating a doc2vec vector? or if there is any way to take in account these numbers in order to improve on the accuracy?


texts:
What is the difference between SparseCategoricalCrosstentropy and sparse_categorical_crossentropy ?
SparseCategoricalCrossentropy: Computes the crossentropy loss between the labels and predictions.
sparse_categorical_crossentropy: Computes the sparse categorical crossentropy loss.

But I am still not sure. Any loss will always be calculates between labels and predictions. SO how are these two different ?


texts:
Given is a dataset that I need to use for a classification and I want to compare the performance of different classification models.
Let's assume, I want to look at logistic regression (with different cut-off-points) and KNN.
Is there anything problematic if I proceed as follows:

Split data in training and validation data (and a test set for the performance evaluation of the winning model).
Train a logistic regression model and a KNN classification model on the training set.
I consider for each cut-off point t between 0 and 1 the logistic regression model as a classification model - so the regression model leads to many classification models.
I now compare for a certain range of t (lets say 0.01 to 0.99) the classification performance of all my classification models (logistic regression for those t and KNN) on the validation data.
The one with the best performance (based on a certain metric) I'll choose.

I was discussing this with somebody else who argued that t needs to be considered as hyperparameter and this parameter needs to be tuned separately.
If this is true - why? And what's wrong with my arguments above?


texts:
I am working on this project in which I have to detect objects only when they are complete(not when objects are incomplete or partially visible).I have more than 20 classes of objects.I tried with yolo by annotating only complete objects, but that didn't work.

How can I approach this problem, any suggestion is useful.


texts: Which algorithm would you use for text categorisation using clustering?

texts: If you are asked to do text categorization using clustering. Which algorithm would you use and why?


texts: What is the correct way to use Active Learning on a huge and unbalanced text dataset?

texts:
I am working on a huge and unlabeled text dataset. It has over 30M lines (= sentences).
We are trying to detect illegal sentences by analyzing 20 different legal issues (e.g. racism, insults, etc.). So, this is a multi-label classification problem. As you can guess, most of the sentences are legal, making the dataset very unbalanced.
We decided to focus on 3 main classes (according to feedbacks) and started using Active Learning (AL) to get a training set and a validation set (the test set will be created later on) :

We extracted a first set of 1 500 sentences and labelled them. This dataset was created with half random sentences, and half judiciously selected sentences (with regex & embedding proximity). Thus, this dataset does not represent the true distribution.

We split this first set into a training set and a validation set (500 / 1 000)

From this, we trained a model on the training set and extracted 1 000 more samples from the pool set to be labelled (using an AL selection function). Once labeled, we injected those samples into the training set. We did this process several times.

At the end, we have 7 000 samples in our training set, and 1 000 in our validation set.

Once this process done, we trained a BI-LSTM and we were surprised to have a better f1-score on our validation set (~ 0.97) than on our training set (~ 0.90). Our understanding is that the training set has more complex cases (through the AL) than our validation set, in which case we are clearly unable to check if the model performs well on those "complex cases".
It also creates annoying side effects, such as triggering the early stopping too soon, or saving "the best model" according to the validation set whereas some epochs were clearly better at managing more varied cases.
We would like to add some of these new specific cases in our validation set. So, we think that, in our process, we should add a portion of newly labeled samples into our validation set (let's say 15%).
Pros:

More varied cases in our validation set, better metrics
No more annoying side effect during the training phase

Cons:

Less data in our training set
Risk of overfitting these corner cases ???

Do you think it is a good approach ?
Is it something common with Active Learning ?
Have we done something bad in our labeling process ?


texts: How to create a Document Categorization Classifier for different contexts of Documents

texts:
I have a doubt solving a test. The idea here is to demonstrate the NLP and Machine Translation abilities.
The Dataset is a multilingual, multi-context set of documents. The dataset is divided on context categories (Wikipedia, conference_papers, Amazon Reviews, etc.,) and on languages.
The objective is to create a document cartegorization classifier (in Python) for the different contexts of the documents. The classifier has to be done at context level, regardless of the language the documents are written in.
An important fact is that The dataset original has been modified and a document Never is repeated in 2 languages.
I have 2 ideas on mind to solve that:

Train on all the data creating a multilingual classifier
Doing language detection first and use monolingual models later.

What could be a reasonable approach to doing text classification for multiple languages?


texts: Create a model that can extract only specific data out of receipts or invoices?

texts:
I'm trying to build a model that is capable of identifying only some of the information on receipts and invoices.
All the documents having different structure in image format.
Sample Data :
Click here
I have used pypdf2 and pytesseract for text extraction from the receipt but the problem is that just returns all the text from a receipt. Tried to work with regex but as the varieties for documents are different everytime so it is not working in this case.
Looking on to build a model that returns only a certain fields such as total price, Date, Tax from a receipt.
I could parse the text to extract by hard coding things but it's not optimal I think. Is there any way to build model for this use case which can identify the required parameters and capture the values. I am looking for something to go on this project.


texts: I'm just getting started with Andrew Ng's Machine Learning wherein he explained the example of the cocktail party problem vs the gene clustering problem in order to explain the difference between clustering and non clustering problems within unsupervised learning itself. However I still don't understand the difference between the two. Can someone please help clarify.


texts: What is the Best NoSQL backend for a mobile game

texts: What is the best noSQL backend to use for a mobile game? Users can make a lot of servers requests, it needs also to retrieve users' historical records (like app purchasing) and analytics of usage behavior.


texts:
I read this paper, but I am having trouble understanding what Teacher Helping technique (page 3) is in context of RNN.

Can someone explain to me what it is? Please assume I don't have much experience in statistics. I'll be glad if this explanation would be as simple as possible (I'll be even more grateful for some example).


texts: Neural Network Learning Rate vs Q-Learning Learning Rate

texts:
I'm just getting into machine learning--mostly Reinforcement Learning--using a neural network trained on Q-values. However, in looking at the hyper-parameters, there are two that seem redundant: the learning rate for the neural network, $\eta$, and the learning rate for Q-learning, $\alpha$. They both seem to change the rate at which the neural net takes new conclusions over old ones.

So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $\eta$, or do they have ultimately different effects?


texts: using neural networks to predict set of charactertics

texts:
Say i have a matrix with m rows and n features where m is the number of people on say an dating website such as tinder. So n could be age, sex, location,job... etc these kinds of features.

My output 'Y' would be a features vector containing features of a person they have previously liked/matched. So it would have m rows and x features(columns) ( x can be age, sex, job, etc...). I would like to predict/suggest a possible match given this training data of X and Y.

Is it possible to use any machine learning/neural networks to predict a SET of features so Y being a matrix rather than a single columned vector? I have seen examples where given some data X we can predict Y but Y is normally a classification such as 0/1 or some sort of wine classification from (1-3) for example. What if it was a set of features? is it possible.


texts: Back Propagation Vs Learning rate in Neuralnet Optimisation

texts:
I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the weights of the neuron.
On the other hand, we have hyperparameter called 'learning-rate' used to update the weight of the neuron in each iteration by calculating the direction of the error.
These two cases look like working independently, I mean, while backpropagation algorithm itself finding the optimal weight, we do not need a learning rate parameter itself.
Is my understanding correct? Please correct me if I am wrong.


texts:
I am using google cloud ML for training data of 1000 images of size 200kb-300kb and using tensorflow to detect objects

Following are my specs:

**CPU: 2 core, 8GB RAM, GPU: 1 NVIDIA TESLA P100**

I want to calculate the price but i am unsure how long it will take to train the images by knowing the time i can estimate the price

any help will be greatly appreciated I dont have any source to ask except here


texts:
Most of the examples I have found online for LSTMs refer to "random" text generation.
One of the problems that I'm trying to solve is to generate a "summary" of many docs into 1 doc. For example:

News Article: Barack Obama life history (2K words)
Wikipedia: Barack Obama  (10K words)
Barack Obama biography (10K words)

Will generate an article about Obama in X words.

Is LSTM the right approach?


texts: Can we access HDFS file system and YARN scheduler in Apache Spark?

texts:
We can access HDFS file system and YARN scheduler In the Apache-Hadoop. But Spark has a higher level of coding. Is it possible to access HDFS and YARN in Apache-Spark too?

Thanks


texts: How to make it possible for a neural network to tune its own hyper parameters?

texts:
I am curious about what would happen to hyperparameters when they would be set by a neural network itself or by creating a neural network that encapsulates and influences the hyperparameters of the network it encapsulates.

The goal for me here is to experiment and get some more in-depth knowledge about neural networks. But I had a hard time finding information to be able to execute such an experiment. Which point in a direction that either it has never been done before or the idea is just really dumb.

Now what I would like to know is, does anyone of you know where I would be able to find information (e.g. books, web-articles, papers, et cetera) to conduct such an experiment?


texts:
I have a simple question.

Why only convolution is used in CNN? There are a lot of possible rules for combining a filter and an image.  Why is pixel-wise convolution the standard? For example, dropout or max-pooling are another typical layers. I want to do some research regarding special convolutions, for example, rotated convolutions or adding some "deformation parameters" to convolutions instead of data augmentation. This make sense? Is there literature regarding this?

Thanks a lot.


texts: Linear Model to generate probability of each possible output

texts:
This is related to Sports prediction (Cricket). I am new to Machine Learning and learning it through TensorFlow.

I am focusing only on one topic which is "How many runs Player A will score in today's match?"

My raw data includes all cricket player's score in all matches. It also includes more finer details like the ground the match happened, day/night match, opposition team etc., I need to use this data and predict four possible ranges for a player's score with all 4 range having equal probability.

I am planning to keep 4 simple FeatureColumn to start with:

Average of runs of the given player in the past 10 matches (relative to the match that is being trained)
Batting order (1 to 10) (in the match being trained)
Average of runs since debut (relative to the match that is being trained)
Average no. of balls faced in the past 10 matches (relative to the match that is being trained)

My actual output will be the runs scored by that player in the match being trained.

I can consider 100 recent matches of a player for this training.

This tutorial https://www.tensorflow.org/tutorials/wide from TensorFlow seems to match with what I am approaching. But the final output is a binary value (whether salary is >50K or not). But I need to generate output with probability of a given player getting a run. For eg.

Run     | Probability of getting this run
-----------------------------------------
0 runs  :  0.01
1 run   :  0.01
2 runs  :  0.02
....
13 runs :  0.5
14 runs :  0.04
....
56 runs :  0.08
57 runs :  0.03
....

Sum of Probability should add up to 1.
I'll then group all these values into 4 ranges with each range having an equal probability (0.25 for each range) like this:

Which approach can I use for solving this problem? I read about Linear regression and that seems to be the best fit for me, but I am not clear on how to generate probability of all possible outcomes.


texts:
With TF 1.x is there a way to stop training, when evaluation loss starts increasing?
Or is there another way to detect overlearning?
I know about the stop_if_no_decrease_hook, but it doesn't consider overlearning.
I also found this Github issue about plotting training and eval loss together, but it doesn't seem to have a simple solution: https://github.com/tensorflow/tensorflow/issues/18858


texts:
I want to get mutual information in iris dataset to select best feature but i confused about mutual information.

What is concept of mutual information for selecting feature?
Can anyone explain it in simple way?

  You do not really understand something unless you can explain it to
  your grandmother.

  Albert Einstein


texts: What are h(t-1) and c(t-1) for the first LSTM cell?

texts:
I know in a LSTM chain you should connect the h(t) of the previous cell to the h(t+1) of the next cell, and doing so for c(t). But what about the first cell? What does it get as h(t-1) and c(t-1)?

I also like to know, if we want to make a multi layer LSTM, What should we give to the first cell of the second layer as h and c? Also will we throw away the h and c of the last cell of the each layer?

And a little other question: will we give the y(output) of each cell to the x(input) of it's corresponding cell in the top layer?


texts: Selecting a boundary on a binary classifier to optimal precision and recall

texts:
I have a logistic regression classifier that shows differing levels of performance for precision and recall at different probability boundaries as follows:

The default threshold for the classifier to decide which class something belongs to is 0.5. However, am I right in understanding that in order to get the best performance trade-off I should set the decision boundary to be about 0.82 below? That can be done in Scikit-Learn, but I want to make sure that I am drawing the correct conclusions. Any advice would be appreciated.


texts: I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?


texts:
I am using an ANN for predicting high value customers based on their first 0, 5, 10 days.

I've made 3 variables: PurchasesOnDay0, PurchasesOnDay0to5, PurchasesOnDay0to10.

PurchasesOnDay0to5 will include PurchasesOnDay0 and PurchasesOnDay0to10 will include PurchasesOnDay0to10 and PurchasesOnDay0to5.

My question is, will the ANN pick up the correlation and adjust accordingly or will this be a problem when creating my model?


texts: Matlab: setting static iterations per epoch in a CNN

texts:
I'm building a convolutional neural network using Matlab's neural network toolbox. I have code designed to cross-train the network with different data sets, using the previous network's layers in training the new network (this provides the same functionality as saving and resuming training on a network).

However, after a few training sessions (generally session 3), Matlab begins increasing the iterations per epoch. I have set the iterations per epoch to be 1, with a maximum of 10 epochs. However, Matlab will increase this to 2 iterations per epoch, then 3, and so on until the full training process is finished.

Is there something I'm missing when I set up my network options? OR, is there a way to hard-set iterations per epoch to stop Matlab increasing the value?

Code for setting the network options (this is the same per training session, but the data inside the validation set changes):

options = trainingOptions('sgdm', ...
'InitialLearnRate', 0.03, ...
'MaxEpochs',10, ...
'ValidationData',set9, ...
'ValidationFrequency',5, ...
'Verbose',false, ...
'ValidationPatience',Inf,...
'Plots','training-progress');

Also included images of the network visualisation in stages 1 and 3 of training, to demonstrate difference:


texts: Is it better to have binary features rather than class ones

texts: I have a dataset containing several features, which have class values (e.g. DBF4, JUL23, ...). In a classification problem and when using decision trees, is it better to convert these values as new binary features: so DBF4 will become a feature, which value is either 0 or 1, or is it better to keep them as they are. Knowing that there are a lot of values (more than the number or rows actually). If this is the case, is there a reason why is it better ?


texts: Number of iterations for minimal cost complexity prunning?

texts:
I've been fiddling with weka's J48 decision tree implementation (C4.5). My goal is to implement cost complexity prunning using weakest link cut method. Basically my algorithm iteratively prunes the tree creating trees with fewer nodes from the previous ones ($ T_0 > T_1 > T_2 >...>T_N$).

My goal now is how do I find that $N$ from $T_N$ meaning how many times should I prune the tree. The algorithm seems to prune pretty aggressively, few iterations too much and it will be completely prunned.

For now I'm using the stop condition regarding the complexity factor and I stop the prunning when $\alpha \gt 0.001$ (from $R(t) = R(t) + \alpha|T|$) . This works pretty well for the two datasets I've tested so far but I wouldn't bet my money on it.


texts: Testing if a sample fits into an existing cluster

texts:
I have a sample of data I'd like to create a model from, which would create N clusters.
After the fitting to clusters, I'd like to test various samples against the existing clusters, seeing if the samples fit any of the clusters, and if not see it as an "anomaly".
I started working on it using DBScan (scikit), my data splits into 3 clusters but I can't find any function/method to test the specific sample against current results, without "refitting".
Is there any other model that would fit my needs? or am I missing something when using DBScan?

Thanks in advance


texts: Does standardization result in normal distribution?

texts:
I have a question about standardization (subtract mean, divide by standard deviation) of data consisting of different features with different ranges.
I read some information that seemed to be contradictory.

Standardization

does not alter the fundamental shape of the data
but it gives the data the properties of a normal distribution: mean of 0 and standard deviation of 1

So, does it change the normal distribution or not?


texts: What is the different between Fine-tuning and Transfer-learning?

texts:
Usually the neural network training has at least 2 steps:

first trained on a large set of some standard data (ImageNet, ...)
and then the resulting weights are trained on a small set of my data (in this step we can train all layers or only one last layer)

What is the same of 2-nd step, is it Fine-tuning or Transfer-learning? And what is the different between Fine-tuning and Transfer-learning?


texts: Applying ConvNets to classify motion/video data

texts:
How would someone go about using deep learning to classify sign language gestures? For example, suppose I had video files of many different gestures. For any given gesture, I might have many videos of it, and each video would be comprised of many frames. When trying to classify MNIST digits in images the dimensions of the input are comparatively simple: height, width, and RGB channels. How would gestures (multiple frames over time) be accounted for? Would time be a fourth dimension? What should the neural net's architecture look like so as not to overfit? Should I use something instead of a convoluted neural network?

Edit:

I know that there are probably clever ways to hand-code predictors for sign language, but I'm more interested in how to architect the neural net and take advantage of the time component of the data (where there is value in the transition of video frames over time). Classifying gestures is a simplification of the actual problem I'm trying to solve, so I'm looking for an approach that would be generalizable to other types of problems where it might be necessary to look at many frames of the video to predict the target variable.


texts: Tensorflow classification - maximize the accuracy of certain classes

texts:
I'm doing some experimentation and trying to train a forex trading model to classify based on three classes:

Buy
Sell
No action

Input rows are labeled as buy when the price at some later time is X% greater than the current price and labeled as sell when the future price is X% less than the current price.  All other rows are labeled as no action, which is the large majority of rows.

The problem I'm running into is that the no action items represent about 84% of the total dataset, however the model is predicting no action almost 99% of the time.  I suppose the reason for this is that the model is aiming for maximum accuracy and it doesn't realize that no action items are of no use.

So what I would like, if possible, is for the model to only care about buy and sell items in trying to maximize accuracy to hopefully increase the number of buy and sell predictions (to the extent that this helps the profitability of the model).

One way that I think I could do this is to increase the number of classes and have each class represent a roughly equal percentage of the total items, however, that would be pretty messy, so I'm curious if there's a way that directly affects the way accuracy is calculated.

EDIT:  I've added additional details here: Variability in CNN test results

But the focus is still in maximizing the accuracy of the buy/sell categories.  Looking through the Tensorflow metrics (https://www.tensorflow.org/api_docs/python/tf/metrics), I think I could use two separate binary classifiers and use the TruePositives metric, however, I'd prefer to keep things simple and get it all in one model.


texts: Beginner level: how to interpret LIME and classification result

texts:
I am new to the concept of model interpretability using LIME method. I am following the tutorial LIME for spectrogram classification. I am finding hard to understand the color coding -- before using LIME the important features were already visible. After applying LIME there is no way to see the colors that highlight the important features. The last set of images below the section "Compute LIME" shows the plot of spectrograms before and after LIME -- to me they look the same. Another tutorial imageLime shows how to apply LIME on natural images. This one is ok to understand since the overlay of LIME on the dog picture is clear. But for synthetic or non-natural images such as the spectrogram, I cannot understand the overlay part. Can somebody help in making me understand how to make sense of the color code and the overlay? Below is a spectrogram image X that I generated using some random data shown in Fig1. Fig2 is obtained by overlaying the LIME map `'image and Fig3 is obtained by applying segmentation. Fig4 is the result of overlay of superpixels. I cannot understand from Fig2 if the entire image pixels are important or not since the color seems to cover the whole image. Similarly, from Fig3 and Fig4 what does the segmentation tell? Also any suggestion on how to do segmentation or apply LIME in a better way would be very helpful.
figure
imagesc(X)
colormap gray %fig1

hold on  %fig2
label = categorical(Y(900)); %choosing 900the image as the query
scoreMap = imageLIME(net,X,label);
imagesc(scoreMap,'AlphaData',0.5)
colormap parula
colorbar

 colormap gray  %fig3
numSuperPixel=75;
[L,N] = superpixels(X,numSuperPixel);
figure
BW = boundarymask(L);
imshow(imoverlay(X,BW,'cyan'),'InitialMagnification',100)  %fig4


texts: Detecting if an image can be made BW/Greyscale/Colour

texts:
I am attempting to classify images of documents into Black&White, Greyscale and Colour. I originally get these images as colour images and in order to save space I want to change the bit depth where possible without a loss of 'content' from the original image.

I have hand labelled around 3000 documents (I could do more) based on what I though they should be changed to (BW/Greyscale/Remain as colour). Generally speaking I labelled them as the lowest possible bit depth without a loss of 'content'. It was acceptable to lose colour apart from in the case of things such as photographs where the amount of colour in the document was large.

I have tried a number of approaches such as taking the image histograms from the grey scaled and black documents and using them as a feature in a number of machine learning models such as a RandomForestClassifier. Creating an ensemble of classifiers, created a CNN LSTM where there were 3 input images. The colour image, the grey scaled image and the black a white image. I have tried doing some transfer learning in order to try an make up for my relatively small dataset and I have also tried a small conventional CNN, non of which where able to go anywhere above 83%.

The dataset has a rough split of BW = 1700, Greyscale = 800, Colour = 500.

The most confusion is between the BW and Greyscale class. I'm running out of ideas as how to tackle such a task as I have looked online and have never seen someone try and approach such a problem.

I am wondering if there are any particular model architectures anyone has used before or features I could extract from the images to help my classification accuracy.


texts: How to consume single piece of text for classification in a model?

texts:
Here's a high-level blueprint of my model -

Input data::

CATEGORY    SERVICE     TITLE                               DEPARTMENT

apple       fruits      i love eating fruits.               fruitshop
mango       fruits      mangoes are yellow in color.        fruitshop
cycle       vehicle     that cycle is really expensive.     motorshop

I convert this to a sparse matrix with Tf-Idf scores, where the columns are the title columns tokenized.

I am using LinearSVC and StratifiedKFold (n_splits = 10, random_state=777, shuffle=True) able to achieve a prediction accuracy of 73%.

Got 2 doubts -

1) What happens when a string like "fresh fish" is used for classification?
Because the words "fresh" and "fish" are never used before, which department will this be classified into?

2) How can I consume this string "fresh fish" in my model?

Currently, I have something like this -

...
# X.shape = (1181, 1930)
# y.shape = (1181,)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=777)
...
pr = mod.predict(X_test)
print(pd.DataFrame(confusion_matrix(y_test, pr)))
...

I tried giving input as print(mod.predict([["fresh fish"]])) but got an error -

ValueError: X has 1 features per sample; expecting 1930

Please advise. Thanks.


texts: How can my Pytorch based GAN output pure B&W with no grayscale?

texts:
My goal is to create simple geometric line drawings in pure black and white. I do not need gray tones. Something like this (example of training image):

But using that GAN it produces gray tone images. For example, here is some detail from a generated image.

I used this Pytorch based Vanilla GAN as the base for what I am trying to do. I suspect my GAN is doing far too much work calculating all those floats. I'm pretty sure it is normalized to use numbers between -1 and 1 inside the nn? I have read it is a bad idea to try to using 0 and 1 due to problems with tanh activation layer. So any other ideas? Here is the code for my discriminator and generator.

image_size=248
batch_size = 10
n_noise = 100

class Discriminator(nn.Module):
    """
        Simple Discriminator w/ MLP
    """
    def __init__(self, input_size=image_size ** 2, num_classes=1):
        super(Discriminator, self).__init__()
        self.layer = nn.Sequential(
            nn.Linear(input_size, 512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, num_classes),
            nn.Sigmoid(),
        )

    def forward(self, x):
        y_ = x.view(x.size(0), -1)
        y_ = self.layer(y_)
        return y_

Generator:

class Generator(nn.Module):
    """
        Simple Generator w/ MLP
    """
    def __init__(self, input_size=batch_size, num_classes=image_size ** 2):
        super(Generator, self).__init__()
        self.layer = nn.Sequential(
            nn.Linear(input_size, 128),
            nn.LeakyReLU(0.2),
            nn.Linear(128, 256),
            nn.BatchNorm1d(256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 1024),
            nn.BatchNorm1d(1024),
            nn.LeakyReLU(0.2),
            nn.Linear(1024, num_classes),
            nn.Tanh()
        )

    def forward(self, x):
        y_ = self.layer(x)
        y_ = y_.view(x.size(0), 1, image_size, image_size)
        return y_

What I have so far pretty much consumes all the available memory I have so simplifying it and / or speeding it up would both be a plus. My input images are 248px by 248px. If I go any smaller than that, they are no longer useful. So quite a bit larger than the MNIST digits (28x28) the original GAN was created over. I am also quite new to all of this so any other suggestions are also appreciated.

EDIT: What I have tried so far. I tried making the final output of the Generator B&W by making the output binary (-1 or 1) using this class:

class Binary(nn.Module):
    def __init__(self):
        super(Binary, self).__init__()

    def forward(self, x):
        x2 = x.clone()
        x2 = x2.sign()
        x2[x2==0] = -1.
        x = x2
        return x

And then I replaced nn.Tanh() with Binary(). It did generate black and white images. But no matter how many epochs, the output still looked random. Using grayscale and nn.Tanh() I do at least see good results.


texts: which scoring function for validation_curve (regression)?

texts:
Is there any thumb of rule which scoring function should be used for e.g. the validation_curve?

Atm I try to study the difference between several optimizers:

validation_curve(grid_best
                                          , X_train
                                          , y_train
                                          , param_name = 'Adam', 'SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adamax', 'Nadam']
                                          , param_range = param_range
                                          , cv=tscv
                                          , scoring="explained_variance"
                                          , verbose = 1
                                          , n_jobs = n_jobs
                )

I use the explained_variance but I think the function has to be minimized cause the values are mostly below zero. That's why I think the explained variance does not make sense here?

edit:

When I use r2 I get the following curve:

Is that normal?


texts: Individual differences in response time (RT) experiment - searching for the right test

texts:
Given a distribution of response times to different categorical variables, what's the best way to test for individual differences?

or more specifically:

There are 100 people pressing buttons in 10 colors, with more than 200k presses collected altogether.
I want to test for individual differences in response-time (RT) to the buttons, what's the best way to do this?

(remember there are differences in response time in-between the different colors).

So far I thought of:

ANOVA - calculate a general (weighted) mean RT and compare to individual people.
The distribution level: compare the mean distribution of RT to individual distributions using:

chi-squared for goodness of fit
Kullback–Leibler divergence test
Kolmogorov–Smirnov test

Find a cool bayesian way to do it (which?)
Use a mixed-effect regression model, and calculate the marginal contribution of the people (R square test).

please help me decide - what's the best practice here?
I'd love to learn about another way I didn't think of that would do the trick.


texts: Keras loading images in incorrect format

texts:
So I was working with the the vgg16 model for dogs vs cats classification and I noticed that keras is not loading images in correct color format.
The code is as follows:

import cv2
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from keras.preprocessing import image

path='data/dogscats/sample/train/dogs/dog.1402.jpg'

imgkeras=image.load_img(path)
imgkeras=image.img_to_array(imgkeras)

plt.imshow(imgkeras)
plt.show()

The output of the following code is

Where as the original image is

Can someone explain why is this happening? , also when the image is loaded through opencv and fed into vgg16 the predicted label is more accurate for this particular image than when it is loaded through keras as above,is the improper color format affecting that?


texts:
I'm trying to replicate the forgetting curve using the open sourced Duolingo data for fun.
The problem is that my finding doesn't make any sense, namely that the longer you wait the better recall value. Anyone have any pointers?

# make our plot outputs appear and be stored within the notebook.
%matplotlib inline

import matplotlib.pyplot as plt # import matplotlib for scatterplot, use the alias plt
import numpy as np # import the numpy package with alias np
import pandas as pd
from scipy.optimize import curve_fit

df = pd.read_csv('learning_traces.13m.csv') # Load Duolingo data
df = df[(df['history_seen'] == 1) & (df['session_seen'] == 1)] # Seen only once before
df = df.sort_values('delta') # sort by: time (in seconds) since the last lesson/practice that included this word/lexeme
df['delta'] = df['delta'] / 60.0 # Seconds to minutes
minInMonth = 44640

def func(x, a):
    return np.exp2(-x / a)

xdata = df['delta']
ydata = df['p_recall']

plt.scatter(xdata, ydata)

popt, pcov = curve_fit(func, xdata, ydata)
print(popt) # Show the result
plt.plot(xdata, func(xdata, *popt), 'r-', label='fit')

Expectation:

Reality

Rolling Mean


texts: How do we numerically update the decision boundary in case of logistic regression

texts:
In logistic regression classification, the initial decision boundary is defined as
−2.7a − 2.2b + 15.2 = 0

How do we apply gradient descent to the initial decision boundary, to calculate the new boundary?

So for this question, to first find z, I substituted, the values of a and b to the equation and then I found the value of Sigma(Zi) for all numbers by using the sigmoid function.

The next step I wanted to do was to use the weight update equation to update the weights for the same is.

My questions are:

I understand that x(i) here represents a feature. Since we have 2 features in the form of variable a and variable b, which feature should be used for xi?

Suppose we take one of the variables, like Var a, and apply it and compute the weight update, how exactly can we utilize this weight update value to find the new decision boundary equation? Like do we add/subtract it to the slope of the old line and find an equation? Or what are we supposed to do.


texts: Historical weather data with machine learning?

texts:
My company gave me a task to build some weather forecasting. I have now historical weather data for 10 years (temperature, precipitation in mm, humidity and etc. more than 30 features total). We need to know what probability for rainfall in South-Asian countries in particular time period (week, month)?

Now I'm able to calculate approx probability based only on monthly averages (like if for 10 years 1st of Dec was rainy 8 times, so the probability is 80%). Of course, it's not a scientific approach at all, but sometimes we need to know rainfall probability for 3 or more months period in advance.

I was thinking to try building a neural network model, but don't understand how it might work in production? What data to give to this model after training it? Does anybody have any ideas?


texts:
The Sizes of both the true label and predicted label are same still, the training accuracy is 0.0

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score

Data Preprocessing

train=pd.read_csv(r'C:\Users\yashd\Downloads\Datasets\titanic\train.csv')
train=train.dropna()
y_train=np.array(train['Survived'])
train=train.drop('Survived',axis=1)  #removing the label from the data

train=train.drop('PassengerId',axis=1) #removing irrelevant features from the training data
train=train.drop('Cabin',axis=1)
train=train.drop('Name',axis=1)
train=train.drop('Ticket',axis=1)

train['Sex']=np.where(train['Sex']=='male',1,0) #assigning a value of 1 to male and 0 to female
train['Embarked']=np.where(train['Embarked']=='S',1,np.where(train['Embarked']=='Q',2,3))
train['Fare']=(train['Fare']-train['Fare'].mean())/train['Fare'].var()
train['Age']=(train['Age']-train['Age'].mean())/train['Age'].var()

x_train=np.array(train)
x_train=x_train.T
y_train=y_train.reshape(1,-1)

Neural Network with 2 hidden layers ,128 neuorns in the first hidden layers and 64 in the second hidden layer. The output layers consists of single sigmoid Neuron

class FNN:
    def __init__(self):
        self.W1=None
        self.b1=None
        self.W2=None
        self.b2=None
        self.W3=None
        self.b3=None
    def sigmoid(self,x):
        return 1/(1+np.exp(-x))
    def forward_prop(self,x):
        self.Z1=np.dot(self.W1,x)+self.b1
        self.A1=np.tanh(self.Z1)
        self.Z2=np.dot(self.W2,self.A1)+self.b2
        self.A2=np.tanh(self.Z2)
        self.Z3=np.dot(self.W3,self.A2)+self.b3
        self.A3=self.sigmoid(self.Z3)
        return self.A3
    def back_prop(self,x,y):
        self.forward_prop(x)
        m=x.shape[1]
        self.dZ3=self.A3-y
        self.dW3=np.dot(self.dZ3,self.A2.T)/m
        self.db3=np.sum(self.dZ3,axis=1,keepdims=True)/m
        self.dZ2=np.dot(self.W3.T,self.dZ3)*(1-self.A2**2)
        self.dW2=np.dot(self.dZ2,self.A1.T)/m
        self.db2=np.sum(self.dZ2,axis=1,keepdims=True)/m
        self.dZ1=np.dot(self.W2.T,self.dZ2)*(1-self.A1**2)
        self.dW1=np.dot(self.dZ1,x.T)/m
        self.db1=np.sum(self.dZ1,keepdims=True)/m
    def fit(self,x,y,epochs=100,learning_rate=0.01,plot=True,disp_loss=False):
        np.random.seed(4)
        self.W1=np.random.rand(128,x.shape[0])
        self.b1=np.zeros((128,1))
        self.W2=np.random.randn(64,128)
        self.b2=np.zeros((64,1))
        self.W3=np.random.randn(1,64)
        self.b3=np.zeros((1,1))
        m=x.shape[1]
        loss=[]
        for i in range(epochs):
            self.back_prop(x,y)
            self.W1-=learning_rate*self.dW1
            self.b1-=learning_rate*self.db1
            self.W2-=learning_rate*self.dW2
            self.b2-=learning_rate*self.db2
            self.W3-=learning_rate*self.dW3
            self.b3-=learning_rate*self.db3
            logprobs=y*np.log(self.A3)+(1-y)*np.log(1-self.A3)
            cost=-(np.sum(logprobs))/m
            loss.append(cost)
        e=np.arange(1,epochs+1)
        if plot:
            plt.plot(e,loss)
            plt.show()
        if disp_loss:
            print(loss)
    def predict(self,x):
        y=np.where(self.forward_prop(x)>=0.5,1,0)
        return y

F=FNN()
F.fit(x_train,y_train)
y_pred=F.predict(x_train)
print('Predicted Label:',y_pred)
print('True Label:',y_train)
acc=accuracy_score(y_train,y_pred)
print(acc)

Output
Loss Plot


texts: Methods to reduce dimensionality within a feature?

texts:
Suppose that I am interested in predicting an outcome (say, the arrival delay [in seconds] of a flight) based upon a set of features.
One of these features is a nominal variable - carrier - that specifies the airline carrier of the flight. This feature has 16 different values. After investigation of how arrival delay is distributed across each carrier, it appears that some carriers could be collapsed into one value (e.g., "AS" and "HA" or "WN" and "B6").

install.packages("nycflights13")
library(nycflights)

boxplot(
    formula = arr_delay ~ with(flights, reorder(carrier, -arr_delay, median, na.rm = TRUE)),
    data = flights,
    horizontal = TRUE,
    las = 2,
    plot = TRUE
)

In general, are there well-known methods for reducing the dimensionality within a feature?


texts: greater reconstruction error with Autoencoder while training on Google Colab

texts: I am training an Autoencoder on Google Colab using the GPU. Every time I run the training afresh, the predicted result seems to have a greater image reconstruction error than the last time. The weights are initialised randomly, however this should not have a very significant impact on the predictions. And each new training is done as the training is done the first time. I want to know why does this happen. Is this problem related to the complexity of the model or a cache related problem?


texts: Autoencoder implementation using ImageDataGenerator

texts:
I'm using the concept demonstrated in this paper. Their training data consists of "GOOD" images and "BAD" images. They train the AE using "BAD" images (X) to make it produce "GOOD" image. "BAD" images in this case maybe very similar to "GOOD" image but with small dent or scratch.
I've had success using ImageDataGenerator on mnist number data but in such case it is trained by using X -> X or if not using ImageDataGenerator the model fitting code will be something like model.fit(x_train, x_train).
However, in this AE problem, we want to train using X_good_and_defect -> X_good or  model.fit(x_good_and_defect_train, x_good_train). Not sure how to achieve this using ImageDataGenerator.

I'm using keras's image data generator to load the images.
train_dir = r'chunks/training'
train_datagen = ImageDataGenerator(rescale=1 / 255)
train_generator = train_datagen.flow_from_directory(train_dir, target_size=(256,256),
                   color_mode='grayscale', class_mode='input', batch_size=256)
...
autoencoder.fit(train_generator,
                epochs=5,
                batch_size=128,
                shuffle=True,
                validation_data=(test_generator, train_generator),
                callbacks=[])


texts: Is it possible to get time period of a time series. Is there any library in python that can do this. I am aware of FFT and scipy.optimize.curve_fit() but I was meaning more about the time series that are fit using SARIMAX() functions. Is there a way to obtain time period of such time series ?


texts: Confidence score over xgboost logistic regression

texts:
The probabilities of logistic regression indicate how the certain the model is over predictions. if its 0.93 it means the model is 93% confident the label is 1 and 7% to be 0. or if the probability is 0.14 it means, 14% to be 1 and 86% to be zero.
In my case, I have 80% accuracy on the hold out data using 0.5 threshold over prediction. this value is calculated using AUC curve. However, if I set the threshold to be 0.9 to calculate the accuracy, the accuracy becomes 60%. Which is not so intuitive. I guess the probability over logistic regression are not necessarily the confidence or fitness of the model. I wonder if it is possible to get a confidence score over these values.
I have seen similar questions, where people have asked how to get confidence interval over prediction. This question is different, because the focus is on logistic regression.


texts:
Suppose I have five sets I'd like to cluster. I understand that the SimHashing technique described here:

https://moultano.wordpress.com/2010/01/21/simple-simhashing-3kbzhsxyg4467-6/

could yield three clusters ({A}, {B,C,D} and {E}), for instance, if its results were:

A -> h01
B -> h02
C -> h02
D -> h02
E -> h03

Similarly, the MinHashing technique described in the Chapter 3 of the MMDS book:

http://infolab.stanford.edu/~ullman/mmds/ch3.pdf

could also yield the same three clusters if its results were:

A -> h01 - h02 - h03

B -> h04 - h05 - h06
      |
C -> h04 - h07 - h08
                  |
D -> h09 - h10 - h08

E -> h11 - h12 - h13

(Each set corresponds to a MH signature composed of three "bands", and two sets are grouped if at least one of their signature bands is matching. More bands would mean more matching chances.)

However I have several questions related to these:

(1) Can SH be understood as a single band version of MH?

(2) Does MH necessarily imply the use of a data structure like Union-Find to build the clusters?

(3) Am I right in thinking that the clusters, in both techniques, are actually "pre-clusters", in the sense that they are just sets of "candidate pairs"?

(4) If (3) is true, does it imply that I still have to do an $O(n^2)$ search inside each "pre-cluster", to partition them further into "real" clusters? (which might be reasonable if I have a lot of small and fairly balanced pre-clusters, not so much otherwise)


texts: Machine Learning applied to database design

texts: Is it possible to apply machine learning to database design given a data source with (pseudo) relationships?


texts: Negative Latent Factors in Factorized Machines

texts:
I'm studing a specific implementation of a recommendation system leveraging on a factorization machine algorithm.
For each person_id and item_id combination, I have an implicit rating of 1 or 0 depending on if the user downloaded the content or not.
In the base model, I have just utilized as input variables the person_id and the item_id.
I selected a latent factor number equal to 5.
In the model output, some of the 5 the latent factors associated to some person_id and item_id are negative, and some predictions of the rating for the combination person_id/item_id are negative too.
I have searched for some theoretical explanations but not found much material, so here I am.

How a negative latent factor can be explained in this setting?
Being the training dataset provided with the target variable equal to 1 or 0, how the model end up with negative predictions for the implicit rating?

Many thanks for any hints or material
bests


texts: Where does spacy, nltk, prodigy, sklearn fit in the AI project?

texts: Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these:


texts:
I really like how this visualization represents the survey participants.

Is any tool for that?  (Or R/python library?)


texts: Is the hyperbolic tangent function a solution to the weight clipping problem of WGAN?

texts:
Instead of clipping to the range $[-c,c]$ in WGAN (Wasserstein generative adversarial network), why not smoothly map into the range $[-c,c]$ by using $c\times \mathrm{tanh}(w)$? This would guarantee the Lipschizt constant is no greater than $c$.
The problem I am talking about is mentioned in

https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html

"Sadly, Wasserstein GAN is not perfect. Even the authors of the original WGAN paper mentioned that “Weight clipping is a clearly terrible way to enforce a Lipschitz constraint” (Oops!). WGAN still suffers from unstable training, slow convergence after weight clipping (when clipping window is too large), and vanishing gradients (when clipping window is too small).
Some improvement, precisely replacing weight clipping with gradient penalty, has been discussed in Gulrajani et al. 2017. I will leave this to a future post."
but even in that 'improvements' paper, I do not see my idea about using tanh implemented.


texts: Data augmentation solutions for tabular/structured data

texts:
Are there any reliable libraries or methods for tabular/structured data (with numerical and categorical features) augmentation? Could you share some?

Basically I believe inventing/augmenting more data could help to improve model performance.


texts: Gaussian Naive Bayes (GaussianNB) classifier not working with large number of features

texts:
I'm trying to make a partial fitting with GuassianNB here's small snippet of my code

classfier = GaussianNB()
classfier.fit(X_init, y_init)
for i in range(X_train.shape[0]):
    if('some condition'):
        classfier = GaussianNB()
        classfier.partial_fit(X_train[i, :].reshape(1, -1), y_train[i].reshape(1, -1).ravel(), classes = np.unique(y_train))
    else:
        classfier.partial_fit(X_train[i, :].reshape(1, -1), y_train[i].reshape(1, -1).ravel())

Everything works correctly until the condition is true, it seems that the classifier for some reason stops learning and starts making arbitrary predictions (predicts all zeros)

tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
print(tn, fp, fn, tp)

>>> 1324 0 1031 0

I thought that maybe my classifier made predictions after reinitializing it (the condition is satisfied right before the loop ends) but after I checked it, it made at least 900 partial fittings before the loop ends with a both 0 and 1 labels.

I'm so confused, what's happening? Thanks

Edit: I figured out what's wrong

it turned out that the problem was that my dataset has so many features (around 40000 as it was originally a string of reviews and vectorized it with tfidf) And for some reason that made GaussianNB classifier break down. I'm trying to know why GaussianNB doesn't with a big number of features. If anyone knows, I would appreciate the help. Thanks


texts: How to find coherence between a large number of sentences

texts: I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a starting point, I'm using Word Movers Distance (WMD) and calculating the similarity between the sentences. But my list of sentences is too long, and doing a pairwise comparison for all the items within a list (document) would be computationally infeasible. What might be the best way to solve this?


texts: I am using a unsupervised pattern matching approach to create a trade strategy. I use the output of the pattern matched results to decide whether to enter a trade or not. For deciding the best pattern parameters I run several combinations over the entire data set and choose the best parameters that yield the best profits. My question is whether this would be considered overfitting. If so, how may I avoid the same? I looked at several posts on StackOverflow but did not find anything that relates to my particular use case.


texts:
I've setup a workflow in Orange3 and have tested some models. Now, I have the results in a Data table widget and I'm wondering if there is a way in which I can save the output of said widget?

I've tried using the Save data widget to no avail. Is there any other way?
I'm on a Mac, version 3.22.0, and I've experienced the same on a Windows machine.


texts: Fittting histograms with combination of an unknown number of Gaussians

texts:
I have many data in the form of one dimensional histogram, to give an example consider the data at  http://pastebin.com/embed_js.php?i=1mNRuEHZ

I expect that these data are obtained from a pdf composed of a sum of Gaussian distributions but with their actual number being unknown.

I want then to fit these data with a model given by a sum of Gaussians

$$
f_N(x) = \sum_{k=1}^N c_k \exp \left[ - \frac{(x-a_k)^2}{2 b_k^2} \right]
$$

where $a_k$, $b_k$, $c_k$ and $N$ are in principle parameters to be fitted.

The problem can be viewed as a problem of model selection between different $f_N$ for all the possible values of $N$.

My idea is then to make an usual fit by least squares at fixed $N$ and then compare the results for different $N$ via some statistical measure of the quality of the fit, as e.g. Akaike information criterion (AIC).
Is this an acceptable test?

For the data of the previous example (that I generated as a test from 3 gaussians) I will obtain

 N        AIC
 -------------
 1      +568.1
 2      +557.4
 3      -446.6
 4      -443.5
 5      -442.7

As you see, the AIC decreases quickly till $N=3$, where it starts to increase again very slowly (due to overfitting). So it is clear that $N=3$ is the best choice here.

I mention that in some cases I have more data histograms extracted from the same pdf, so I could do some cross validation. Although this always a good test, the availability of my data changes from sample to sample, so I'd prefer to have a criterion for the single histogram (if the quality of the test is not too much different).


texts: Sampling items from a population of subpopulations

texts:
I have a population of $n$ items to label and a budget to label only $m$ ($m << n$) of them before training. The population can be partitioned into subpopulations, recursively. In other words, the whole population can be represented as a tree of subpopulations, $x_1$ can be split into $x_2$ and $x_7$ subpopulations, $x_2$ into $x_3$ and $x_4$, etc. Some subpopulations are more diverse and have more subpopulations.

What algorithm should I use to sample $m$ items, so that they belong to the most diverse set of subpopulations? Is stratified sampling a way to go?
My scenario is to label 500 products from 27K products organized into a taxonomy of product categories. There are 17 root categories that split into 118 subcategories that split into 597 sub-subcategories, etc. E.g. root categories include Produce and Bakery, Produce involves Vegetables and Fruits subcategories, etc. You can see that not only the number of products but also the number of final subcategories (leaves) in the tree is much bigger than my budget $m$, i.e., the number of products I can select for labeling.


texts: Reasoning behind using Deep Learning on non-local data

texts:
I understand the using of deep learning for data that have "local" structure, for example, images/videos/texts, as the convolutional layers reduce the amount of dimensions.
However, I saw that some people use it on non-local data, as on databases for example, here or here on the titanic database.
My question is: as just one hidden layer with enough neurons within can theoretically creates as many dimensions as we want, why would one use several hidden layers/deep learning instead of just using a single bigger hidden layer?