texts: Cut-Off using Frequent Pattern Mining - Spark Mllib

texts:
I am using the Association Rules algorithm using this:

http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html

I'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?

Many thanks!


texts: Can I save only some VGG19's layers into a .H5 file?

texts:
I am training a deep-learning style transfer model with the pretrained-VGG19 CNN.

My aim is to use it in my Android app for personal purposes with Google Firebase Machine Learning Kit (which would host my .H5 model to make it usable by my Android app). The maximum .H5 model file's size allowed by Machine Learning Kit  is: 8MB. However when I save the whole VGG19 model, I end with 80MB... So I can't use it.

Since only some layers of the VGG19 network are used in my style transfer program, is it possible to reduce the .H5's size by saving only those layers' weights, or something like that? Is there any other solution to my problem?

To save my VGG19 network as a .H5 file, I use the following Python command:

model.save('./style_transfer/st.h5', include_optimizer=False) , where model = vgg19.VGG19(input_tensor=input_tensor, weights='imagenet', include_top=False).

As you can see, I already don't include the optimizer in order to reduce the saved .H5's size.


texts: Is there a possible intersection of mathematical logic and machine learning?

texts:
Why there is so comparatively little work in machine learning using  mathematical logic? Most of the research uses methods in (Bayesian) statistics, linear algebra, numerical analysis and so on. However, almost no use of, for instance, relevant logics is made.

My background is in philosophy, not computer science. But given that logic is in the business of analyzing  the notion of inference I was rather surprised to learn that machine learning and other branches of AI research do not rely on mathematical logic as much as I thought they would. What about non-monotonic logics in drawing inferences or modal logic? It seems to me that it is mostly philosophers and some, dare I say, older people in cs departments that work in logic?

Are the reasons practical (easier implementation of methods from other areas), sociological (machine learning/AI was started by non-logicians), or theoretical (the bulk of machine learning is mostly about categorising and analyzing data, because that is the hard part, and less drawing inferences).

As a closer: Is there any chance that mathematical logic will have greater influence on machine learning in the future?

All the best!


texts: Is reward accumulated during a play iteration when performing SARSA?

texts:
I've been having issue with getting my DQN to converge to a good solution for snake. Regardless of the different types of reward functions I've tried, it seems that the snake is indefinitely going around in circles. I have not tried exploring more states yet because I am confused about how to properly assign reward.

Currently, I am using a 2D-Gaussian distribution to assign reward where $f(x=x_{food},y=y_{food}) = 1$. Terminal states like hitting the wall or itself result in a reward value of -1.

My reason for using the Gaussian was because of the relatively sparse rewards in this game and the ability to easily clip rewards between [1,-1] in meaning full way.

I have two questions.

Is this an appropriate way to define the reward function?
Currently during training, I do no accumulate the reward during each play iteration. So each reward for transitions are independent of the reward values before it. Right now I am doing $\big[S_1,A_1,R_1,S_2,A_2,R_2,S_3\big]$. I've looked at other code where people have accumulate the reward like $\big[S_1,A_1,R_1,S_2,A_2,R_2 = R_1+r,S_3\big]$. Where $r$ is given by the reward function. The thing is, I can't find a paper that defines why you should do this. So my question is, which way is the appropriate way to assign reward?


texts:
I am just starting off with TensorFlow and trying to implement an RNN for a dataset which consists of 11 Features (all numeric). These features will be used to predict the output of another column.

I am currently lost on where to start and tho I am able to understand how a RNN functions all the tutorials I could find were mainly related to image and text datasets.


texts: Using class weights with validation data in Keras

texts:
I'm using validation data for hyperparameter optimization and am trying to use class weights. For model.fit(), there is an option to pass class weights (the argument is class_weight) to the function for test data, but I'd like to do the same to validation data.

Another user had the same question a while back but no satisfactory answers were given:

Class weighting during validation in Keras

When passing validation data as an argument to model.fit(), the validation data can take the form of a tuple: (x_val, y_val, val_sample_weights) but in the Keras documentation I'm not seeing any way to use class weights instead of sample weights.

Does anyone have any idea on how to use class weights with validation data in Keras?

Here is some info that might be helpful:

From the Keras documentation, description of the 'class_weight' argument: "Dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to "pay more attention" to samples from an under-represented class."

Here is a notebook from Francois Chollet which shows an example of using class_weight: https://colab.research.google.com/drive/1xL2jSdY-MGlN60gGuSH_L30P7kxxwUfM#scrollTo=REP1KlrSEx-m

Thanks for any help!


texts:
I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade.

How can I return only the details of the student that have positive grade (make the join) but not using SQL Context.

I've this code:

val all_students = sc.textFile("/user/cloudera/Data");

case class Students(Customer_ID:String,Name:String,Age:String);

def MyClass(line: String) = {
val split = line.split(',');
Students(split(0),split(1),split(2))
}

val df = all_students.map(MyClass).toDF("Customer_ID","Name", "Age").select("Customer_ID","Name", "Age");

val students_positive_grande = sc.textFile("/user/cloudera/Data");

How can I make the join between this datasets? I want to join the "Customer_ID" with the first column of the second column...


texts: How to evaluate the performance of a model in production when labeling data is costly?

texts:
I have come to a problem for which I can't find a solution.

Let's talk about a hypothetical binary classification problem in which you have some years of (human) labeled data. The final objective is to remove the need for a human to label the data. You train the model with, for example, the first 3 years of data, and then evaluate the model with the last year or last two years. A different but valid approach that tries to mimic a real-life situation is testing every month with all the data available up to that month.

Let's assume that the choice of the train and evaluation methods explained above are not relevant. You get everything done and deploy the model in production, now the human that had to label manually 1000 samples a day is not needed but on the other hand, you stop receiving the "ground truth" thus preventing you from evaluating the model in production!!

Instead, you could reduce the work hours of the human and label just some % of the data, and here arises my question. Which % would be representative enough to assert with confidence that the model is performing well/poorly in production? How do you even calculate what "representative" means? I suppose that it is dependent of the variability of the data, but how do you measure it? I did a lot of googling but haven't found any relevant information I would love to hear how do you measure a model performance in production when "creating" new ground truth is costly


texts: How can I make big confusion matrices easier to read?

texts:
I have recently published a dataset (link) with 369 classes. I ran a couple of experiments on them to get a feeling for how difficult the classification task is. Usually, I like it if there are confusion matrices to see the type of error being made. However, a $369 \times 369$ matrix is not practical.

Is there a way to give the important information of big confusion matrices? For example, usually there are a lot of 0s which are not so interesting. Is it possible to sort the classes so that most non-zero entries are around the diagonal in order to allow showing multiple matrices which are part of the complete confusion matrix?

Here is an example for a big confusion matrix.

Examples in the Wild

Figure 6 of EMNIST looks nice:

It is easy to see where many cases are. However, those are only $26$ classes. If the whole page was used instead of only one column this could probably be 3x as many, but that would still only be $3 \cdot 26 = 78$ classes. Not even close to 369 classes of HASY or 1000 of ImageNet.

See also

My similar question on CS.stackexchange


texts: Can I install Tensorflow and Keras on Cloud?

texts:
I will like to install Tensorflow and Keras on my PC. I use 32 bits OS. I learnt Tensorflow is not compatible with 32 bits. I cannot upgrade my OS to 64 bits since my hardware does not support it. I have learnt one or two things about cloud computing but I only know about how I can use it for storage purposes.
My question is, can I install Tensorflow and keras on cloud and somehow use them on my PC? Or is there a way round this as I cannot afford to get another PC for now and I cannot afford to stop learning as well?
NOTE: I use Windows 7 32bits
NOTE: I read about how one could install Tensorflow from source but the writer discourages it as he says it won't be worth the headache in the long run as the software would not be able to implement many programs because of space.
NOTE: I have heard about Pytorch but I don't think it is supported by 32 bits. In addition, the materials I am using for learning machine learning only focus on Tensorflow.
Thanks.


texts:
What is the Number of capsules in the Primary Capsule Layer of Capsule networks?

In many articles, it is written that the number of Capsules is 32 but in the paper, by Hinton - Dynamic Routing between capsules it is written that
"In total PrimaryCapsules has [32 × 6 × 6]
capsule outputs (each output is an 8D vector) and each capsule in the [6 × 6] grid is sharing their
weights with each other."

which implies that the layer has 36 capsules.


texts:
I am trying to write a program in python to calculate growing degree-day for a specific insect out of average hourly temperature. The way you calculate growing degree day from hourly temperature is by creating a curve/function out of these discrete values (one value for each hour) usually using interpolation. Then you use min/max temperature threshold where the insects are active to grow and find where they intersect with the created curve/function (the values I am using are threshold_min=11.7C and threshold_max=30C). I am using scipy intepolation package to create a curve out of these discreet values as follow:

    x = numpy.linspace(0, 23, 24)
    y_temp = df_temp.loc[pd.datetime(2018, 7, 15)].values

    cubic_spl_temp = scipy.interpolate.CubicSpline(x_spl,y_temp)

then I plot this curve with matplotlib, here is what I get:

    plt.plot(x, cubic_spl_temp(x_spl))
    plt.axhline(11.7, color='green', lw=2, alpha=0.5)
    plt.axhline(30, color='red', lw=2, alpha=0.5)

image_link:  https://i.stack.imgur.com/iJ8YG.png

what I need to do is to calculate the area under the curve where the curve and min/max threshold intersect (area between the curve and the two min/max threshold lines).

Is there a scipy function or any other python package that can help me with this? also, is there a way to do it in r which would be easier to do and solve the same problem?


texts:
To compute anomaly score we use whole dataset to test or only test dataset? Can someone please help.
And with streaming data we doesn't train so in that we are using whole dataset to find accuracy. So how would we compare both without streaming and with streaming.

Thank you.


texts: Data Visualization with multiple dimension, and linear separability

texts: I have a dataset of two classes with several features, how can I visualise such data using Matlab to decide if it is linear separable or not?


texts: Why ELMo's word embedding can represent the word better than glove?

texts:
I have read the code of ELMo.
Based on my understanding, ELMo first init an word embedding matrix A for all the word and then add LSTM B, at end use the LSTM B's outputs to predict each word's next word.
I am wondering why we can input each word in the vocab and get the final word representation from the word embedding matrix A after training.
It seems that we lost the information of LSTM B.
Why the embedding can contains the information we want in the language model.
Why the training process can inject the information for a good word representation into the word embedding matrix A?


texts: PCA shows overlapping boundaries, then why SVM performs best

texts:
I am trying to understand which model might work for a given problem before trying the models, I find this case against my knowledge. Please guide what I am missing. I am new to Data Science.

Here is the graph which I got through PCA :

Now you can see the boundaries are very much overlapping.
The theory for SVM says that this model might work best with overlapping non linear data, which does not seems to be this case.

But still its able to identify all data in test set. So can you provide some clarity on why SVM performing good in this.

So my final results it is below order:

Logistic Regression and SVM are same (Accuracy Score : 1.0)
Random Forest (Accuracy Score : 0.9680851063829787)
KNN (Accuracy Score : 0.925531914893617)

other details :

feature set : 40
sample data : around 500


texts:
The problem: I have a huge dataset which is more than 200GBs in size. The dataset contains around 200 columns (predictors). The task at hand is to predict which product to promote to the customer so that he/she ultimately buys it and thus maximizes the revenue of the company.

It is a cross selling situation but I need to look at product recommendation as well as revenue maximization.

From what I understand, revenue maximization would require a regression model. (Correct me if I'm wrong)

I am not sure how to recommend products since there are more than 2000 unique products. (Dummy coding would require tremendous amount of time and resources I feel).

Due to the sheer size of the data, I am planning to use Python for handling the data. (Suggestions about R also welcome)

PS: Forgive me if the problem seems too basic, but I have just started learning

[UPDATE]:

The data is in long(narrow) format
I can also use R in order to tackle this
Products are identified by their unique product ID (2000+ unique product IDs)
Headers:
date | time | pid | cust_id | ... | amount | tax | net_revenue
Net revenue: Continuous variable
Product ID: Continuous but to be treated as nominal
Cust_id: Continuous but to be treated as nominal


texts:
I have a dataset with the target variable having 3 classes.  Value counts of Target variable are as follows:

Class A :26000
Class B: 5000
Class C: 500
Class C  prediction is most important from a business standpoint.

To date I have mostly done binary class (mostly 1:0.12) distribution of target class,wherein at the end, deciling of prediction probabilities is done, in decreasing order of probabilities starting with D1 as the highest probability and D2 the lowest.

Now with influx of a new class that too with so less population,I am perplexed on three aspects.

What algorithms are best?
What metric should be used?
How to get a deciling of probabilities in this case, if applicable?

Any leads/kernel/blog post would be really helpful.


texts:
I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:

Network Architecture: [67 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~45%
Observation: Both training accuracy and validation accuracy stops improving.

Exp 2:
Repeated Exp 1 with batchnorm layer

Validation Accuracy: ~50%
Observation: Got 5% increase in accuracy.

Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network

Network Architecture: [67 40 40 40 40 40 40 5]
Optimizer: Adam
Learning Rate: exponential decay of base learning rate
Validation Accuracy: ~55%

Thoughts on what might be happening?


texts: Scaling the data iteratively one by one vs batch scaling

texts:
I have 2000 signals in a dataset of shape (2000, 400000) where each signal is recorded within the range -127, 128. I want to downscale each signal from (-127, 128) to (-1,1) to save memory space and also for better visualization. There are two approaches:
Approach 1:
Iteratively apply minmax_scale individually at each signal something like the following:
from sklearn.preprocessing import minmax_scale
data = read_dataset(...)
for i in range(data):
    minmax_scale(i, feature_range=(-1, 1))

Approach 2:
Fit the whole dataset using MinMaxScaler something like the following:
from sklearn.preprocessing import MinMaxScaler
data = read_dataset(...)
scalar = MinMaxScaler(feature_range=(-1,1))
scaled_data = scalar.fit_transform(data)

I use the first approach because the dataset does not fit the memory but I am worried if it is incorrect. I want to make sure if my choice is sound, in theory.
Thank you very much.


texts: Clustering cartesian coordinates associated with 1 categorical feature

texts:
I have a series of 2D coordinates X = {x, y}. Each are associated with one categorical variable W that can take 7 different values.

E.g:

coord  W
X1     3
X2     5
X3     7
X4     3
X5     2
X6     3
X7     2
...
X2000  5
...

I would like to get all the clusters that belong to a given set of my categorical variable within 2 pixels of each other. Say, for the triplet of W values (2, 5, 7), or a tuplet of all values (1,2,3,4,5,6,7), I want to have all the sets (if any) of coordinates that are within 2 pixels of each other. What would be the most appropriate methods?

Also, there are no singletons, all these coordinates have at least another nearest neighbour with a different W value. I only know how to do this to only find the sets of pairs of coordinates for 2 different W values (using pair-wise euclidian distance matrix), but to get clusters of more elements for more than 2 of my W values, I am confused about what clustering method to use (and if that actually falls in the realm of clustering at all...), as that seems rather basic and I keep reading about rather sophisticated approaches that look overkill (KNN, HDBSCAN, ...).


texts: Which probability distribution will you use to model outliers?

texts:
I was asked this question in a recent interview for the position of a Data Scientist:  Which probability distribution will you use to model outliers ?
I told him outliers are like rare events which can be modelled by a Poisson distribution. I pretty sure I'm wrong and the interviewer seemed to think the same. But I don't know the answer to this.
Please advise.


texts: Clustering with Replicator Neural Network

texts:
I'm trying to cluster an unknown set of data with a replicator neural network.
The number of clusters is determined by the number of neuron units in the middle layer, multiplied by the number of steps in our step function responsible for the middle layer activation.
The only information provided, is that the feature size vector will be 10. However there are no information about the number of possible clusters.
The final goal is to detect anomalies with the neural network.

How do I approach this task?
I'm not sure how much neurons and steps I should be using. I believe I can limit the neurons to about 4, as it should be usually less than half the input neurons. However I'm not sure how the network is going to perform on a lot of steps.
Perhaps to many neurons won't hurt, as for my understand neural networks cluster, compared to k-means, independent from the number of clusters, while a number to small might damage the anomaly detection by limitation?


texts: Losses of keras CNN model is not decreasing

texts:
I am working on Street view house numbers dataset using CNN in Keras on tensorflow backend. I have queries regarding why loss of network is not decreasing, I have doubt whether I am using correct loss function or not. First I preprocess dataset so  my train and test dataset shapes are:

('Training set and labels', (26721,32,32,1), (26721, 6) )
('Validation set and labels ', (6680,32,32,1), (6680,6) )
('Test set and labels', (13068,32,32,1), (13068,6) )

Reason why labels array has 6 columns, because maximum digits in one image is 6. For suppose, if image has 2 digits "1 and 2", then labels array is considered as [1,2,10,10,10,10] , where 10 represents no digits.

Now this is my model definition:

def modelCNN(input_shape, num_classes):

  model = Sequential()

  model.add(Convolution2D(16, 7, 4, border_mode='same',
                    input_shape=input_shape))
  model.add(PReLU())
  model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2), border_mode='same'))
  model.add(BatchNormalization())

  model.add(Convolution2D(32, 5, 3, border_mode='same'))
  model.add(PReLU())
  model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2), border_mode='same'))
  model.add(BatchNormalization())

  model.add(Convolution2D(64, 3, 3, border_mode='same'))
  model.add(PReLU())
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), border_mode='same'))

  model.add(Flatten())
  model.add(Dense(512))
  model.add(PReLU())
  model.add(Dropout(0.5))
  model.add(Dense(num_classes))
  model.add(Activation('softmax'))

return model

I tried with different customs model, but nothing seems working. I do compilation, loss and optimization like this:

model.compile(optimizer='adam', loss='categorical_crossentropy',
                                    metrics=['accuracy'])

print(model.summary())
csv_logger = CSVLogger('training.log')
early_stop = EarlyStopping('val_acc', patience=200, verbose=1)
model_checkpoint = ModelCheckpoint(model_save_path,
                                    'val_acc', verbose=0,
                                    save_best_only=True)

model_callbacks = [early_stop, model_checkpoint, csv_logger]
K.get_session().run(tf.global_variables_initializer())

model.fit_generator(train,
          samples_per_epoch=np.ceil(len(train_dataset)/batch_size),
          epochs=num_epochs,
          verbose=1,
          validation_data=valid,
      validation_steps=batch_size,
          callbacks=model_callbacks)

Now, What I think that categorical_crossentropy loss needs one hot vector representation, but I am not sure how should i convert my labels array. Because unlike Mnist dataset, I have multiple labels in each image.

Is there any loss function that I can use? or Any suggestion to make things work?

Note: I am using CNN just for classification only, not for detection.

Thanks very much in advance


texts: Where are the 60 million params of AlexNet?

texts:
On the abstract of the AlexNet paper, they claimed to have 60 million parameters:

The neural network, which has 60 million parameters and 650,000 neurons, consists
of five convolutional layers, some of which are followed by max-pooling layers,
and three fully-connected layers with a final 1000-way softmax.

When I implement the model with Keras, I get ~25 million params.
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(96, 11, strides=4, activation="relu", input_shape=[227,227,3]),
    tf.keras.layers.MaxPooling2D(pool_size=(3,3), strides=(2,2)),
    tf.keras.layers.Conv2D(256, 5, activation="relu", padding="SAME"),
    tf.keras.layers.MaxPooling2D(pool_size=(3,3), strides=(2,2)),
    tf.keras.layers.Conv2D(384, 3, activation="relu", padding="SAME"),
    tf.keras.layers.Conv2D(384, 3, activation="relu", padding="SAME"),
    tf.keras.layers.Conv2D(256, 3, activation="relu", padding="SAME"),
    tf.keras.layers.Dense(4096, activation="relu"),
    tf.keras.layers.Dense(4096, activation="relu"),
    tf.keras.layers.Dense(1000, activation="softmax"),
])

Note that I removed the normalization and set an input of 227*227 instead of 224*224. See this question for details.
Here is the summary from Keras:
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 55, 55, 96)        34944
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 27, 27, 96)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 27, 27, 256)       614656
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 256)       0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 13, 13, 384)       885120
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 13, 13, 384)       1327488
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 13, 13, 256)       884992
_________________________________________________________________
dense (Dense)                (None, 13, 13, 4096)      1052672
_________________________________________________________________
dense_1 (Dense)              (None, 13, 13, 4096)      16781312
_________________________________________________________________
dense_2 (Dense)              (None, 13, 13, 1000)      4097000
=================================================================
Total params: 25,678,184
Trainable params: 25,678,184
Non-trainable params: 0
_________________________________________________________________

I'm really far from 60 million. So, how did they sum 60 million params?
For reference, here is the architecture of the model as described in Sec. 3.5 of the paper:

The first convolutional layer filters the 224x224x3 input image with 96 kernels of size 11x11x3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5x5x48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3x3x256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3x3x192, and the fifth convolutional layer has 256 kernels of size 3x3x192. The fully-connected layers have 4096 neurons each.


texts: Merging (intersecting) more than two posting list in linear time

texts:
The intersecting algorithm for two posting lists implemented below:

def intersection(A, B):
    a, b = len(A), len(B)
    i , j = 0, 0
    result = []
    while i < a and j < b:
        if A[i] == B[j]:
            result.append(A[i])
            i += 1
            j += 1
        elif A[i] < B[j]:
            i += 1
        else:
            j += 1
    return result

A = [1, 2, 4, 11, 31, 45, 173, 174]
B = [2, 31, 54, 101, 174]
intersection(A, B) #returns [2, 31, 174]

Brief description of the algorithm:

For the given posting lists, it starts from the first index of both array A and B. Then it compares the documents:

If A[i] equals to B[j]: Add A[i] (or B[j]) to result and then increment both i and j by one.
If A[i] is greater than B[j]: Increment j by one.
If A[i] is lower than B[j]: Increment i by one.

If we consider the length m, n for A and B the time complexity of this algorithm would be $O(m + n)$.

Problem Description:

Suppose that we have more than two arrays. Is it possible two do this is linear time?

P.S: A good approach is that we start with increasing length order of arrays. but the actual question is about the time complexity.


texts: What are good toy problems for testing Transformer architectures?

texts: I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer (or alternative thereof) is working at all? I am looking for simple problems that can preferably be synthetically created and can be trained with really small setups (few layers, small embedding sizes etc.) in a short time. Ideally this should be problems that play to the strengths of transformers but would be hard to solve for, say, a fully connected feed-forward network. Tasks that can be applied to just an Attention Layer would be useful, too.


texts:
A confusion matrix is a table that is often used to describe the performance of a classification model (or "classifier") on a set of test data for which the true values are known. The confusion matrix itself is relatively simple to understand, but the related terminology can be confusing.

What is sensitivity in confusion matrix?


texts: doubt in improvement of performances given by a layer in a deep neural network

texts:
Today I was discussing about neural network with a friend, and he said that the more layers we add, the less increase in accuracy each layer gives individually. Is this fact true?

I know that it is better to go deep then wide when building a neural network, but it is true that if we have a lot of layers, individually they give a smaller contribution to the performances? Thanks in advance.


texts:
I tried topic modeling (LDA, NMF) to extract insights from the data.

I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context?

(Aside) Are there any methods to show similarity of a topic or documents from topic?


texts:
Given a set of lines, is there a way to train to extract geometric shapes. For example, the picture on left has some blue lines (with red endpoints). How can I train to extract shapes like on the right side? The output must be different shapes with each having a set of points.

I searched for related research papers. I could not find any paper with this exact problem, though some of them came close to this with RNN.


texts:
I have implemented NER system with the use of CRF algorithm with my handcrafted features that gave quite good results. The thing is that I used lots of different features including POS tags and lemmas.

Now I want to make the same NER for different language. The problem here is that I can't use POS tags and lemmas. I started reading articles about deep learning and unsupervised feature learning.

My question is:

Is it possible to use methods for unsupervised feature learning with CRF algorithm? Did anyone try this and got any good result? Is there any article or tutorial about this matter?

I still don't completely understand this way of feature creation so I don't want to spend to much time for something that won't work. So any information would be really helpful. To create whole NER system based on deep learning is a bit to much for now.


texts: Effects that Empty cells have in Unsupervised Machine Learning

texts:
I have a dataframe in a csv file that I would like to perform different unsupervised machine learning algorithms on.

The file itself has some cells that are not filled - they are just empty:

As shown in the picture above, some of the cells are empty. So the question follows - will these empty cells effect the outlier detection process which I would like to use?

Or should I instead work on a code that fills these empty cells with 0's

I have been told that that filling the empty cells has to be done so we don't have data sparsity, even though my csv files is thousands of rows long

Any advice regarding this is more than welcome.


texts:
I am having an HTML string and want to find out if a word I supply is relevant in that string.

Relevancy could be measured based on frequency in the text.

An example to illustrate my problem:

this is an awesome bike store
bikes can be purchased online.
the bikes we own rock.
check out our bike store now

Now I want to test a few other words:

bike repairs
dog poo

bike repairs should be marked as relevant whereas dog poo should not be marked as relevant.

Questions:

How could this be done?
How to I filter out ambiguous words like in or or

Thanks for your ideas!

I guess it's something Google does to figure out what keywords are relevant to a website. I am basically trying to reproduce their on-page rankings.


texts: Bayesian optimization with Keras tuner for time series

texts:
Goal: trying to use walk-forward validation strategy with keras tuner for time series when training a neural network (mainly LSTM and/or CNN).

Did anyone find a direct way of doing this?

One possible way I can think of is:

implementing a custom 'objective' function, e.g. 'mean squared error' using walk-forward validations trategy
this custom function could be passed via the 'objective' input parameter in the tuner (be it RandomSearch, BayesianOptimization, Hyperband...)
when calling tuner.search, we pass the train & validation data, to be used in the walk-forward custom function

Thanks in advance


texts: Reinforcement Learning - What's the formula for the value function

texts:
I'm trying to implement a value-iteration algorithm to solve a grid-world problem (I'm new to the field). The usual formula that I encounter about the value function V(s) is:
$$V(s) = R(s) + max_{a \in A} \sum_{s' \in S} T(s, a, s') V(s')$$
where $S$ is the set of states, $A$ the set of actions, $T$ the transition model

$$T(s, a, s') = P(s_{t+1} = s' | s_t = s, a_t = a)$$
and $R$ the reward function.

Since I'm working on a model-based problem, $T$ and $R$ should be known; the problem is that I don't know how to define (or compute knowing the details of the problem) $R$. If I'm in a state $s$ and take an action $a$ that would make the agent hit a boundary of the grid world, the new state $s'$ will be $s$, and the reward, as defined by the problem, is -5.

On the other hand, if I had arrived in state $s$ through another state, the reward will be -1. So basically, $R(s)$ depends on the previous state and the previous action. How do I represent that in the $V(s)$ formula?


texts: Reason behind the sum of rate factors for calculating cost function derivative

texts:
Suppose we have a network of neurons like below:

We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from many paths

now we just pick one of that paths from our first choosed weight to -> it's activation's neuron to -> one of it's weights on the next layer to -> it's activation and so on...

we can imagine an equation (using chain rule) like below for this only one path and we call it a rate factor:

but it is only for one path of changing the cost as we changing that weight, so we need calculate the rate of cost changing by all possible related paths (all the rate factors) in the famous book of michael nielsen wrote that we calculating the rate of cost changing by doing sum all over the rate factors like below:

but really i don't understand exactly how can we calculate that slope of the cost with respect to that weight by summing all that rate factors how can relate that sum to our equation

a little later on that book we had a proof for an equation for the error in layer 'l' in terms of the error in the next layer, for that we have:

and here we do a sum over all related neurons errors too.
can someone give me a proof or an explain for that sum
how that rate of cost changing can relate to sum over all that related rate factors, why we use sum?

of course the neurons's activations are related to each other by calculating sum of all neurons in last layer using weights and biases

but using this for calculating that derivative is not so clear and tangible for me but any explanation about that sum over the derivatives can be very helpful for me.

thank you.


texts: In the context of machine learning, what is the relationship between "Normalization" and "norm"

texts:
"Normalization" and "norm" are used a lot in machine learning

In statistics and applications of statistics, normalization can have a range of meanings.

  In the simplest cases, normalization of ratings means adjusting values
  measured on different scales to a notionally common scale, often prior
  to averaging. In more complicated cases, normalization may refer to
  more sophisticated adjustments where the intention is to bring the
  entire probability distributions of adjusted values into alignment. In
  the case of normalization of scores in educational assessment, there
  may be an intention to align distributions to a normal distribution. A
  different approach to normalization of probability distributions is
  quantile normalization, where the quantiles of the different measures
  are brought into alignment.

In linear algebra, functional analysis, and related areas of mathematics, a norm

  is a function that assigns a strictly positive length or size to each
  vector in a vector space—except for the zero vector, which is assigned
  a length of zero. A seminorm, on the other hand, is allowed to assign
  zero length to some non-zero vectors (in addition to the zero vector).
  A norm must also satisfy certain properties pertaining to scalability
  and additivity which are given in the formal definition below.

in the context of machine learning, what is the relationship between "Normalization" and "norm"


texts: Filter phrases based on correspondent POS tags

texts:
I have two lists:
The first one includes sentences and the second one includes parts of speech (POS) tags. I want to use POS tags as a filter and find phrases according to a pattern such as 'JJ JJ NNS' (adjective+adjective+noun)

How can I do that?

import re

key_list= [' techniques from nonlinear analysis and partial differential equations form the basis for these studies .',
 ' in this case , center-manifold theory can be applied to study the ordinary differential equations which replace the nonlinear \
partial differential equations for the wave motion .',
' differential equations are cool .']

value_list = ['NNS IN JJ NN CC JJ JJ NNS VBP DT NN IN DT NNS .',
 ' IN DT NN , JJ NN MD VB VBN TO VB DT JJ NN NNS WDT VB DT JJ JJ JJ NNS IN DT NN NN .',
 ' JJ NNS VBP JJ .']

pattern= re.compile('JJ JJ NNS',re.M|re.DOTALL)

for value in value_list:
    value_output = pattern.findall(value)
    print(value_output)

# I want  partial differential equations and ordinary differential equations


texts: How to use r2-score as a loss function in LightGBM?

texts: I am trying to implement a custom loss function in LightGBM for a regression problem. The intrinsic metrics do not help me much, because they penalise for outliers... Is there any way to use r2_score from sklearn as a loss function for LightGBM?


texts: Keras: Provide One-Hot-Encoded input values to neural network

texts:
I have a dataframe which has two columns of interest: A and B with string values. I am trying to build a prediction model which takes in a set of values in A as input and predicts the corresponding B values. I am trying to one-hot encode the string values before giving it to the neural network. This is what I have done:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder
from keras.models import Sequential
from keras.layers import Dense

def load_dataset(filename):
    # load the dataset as a pandas DataFrame
    data = pd.read_csv(filename, header=None)
    dataset = data.values
    X = dataset[:, :-1]
    y = dataset[:,-1]
    # reshape target to be a 2d array
    y = y.reshape((len(y), 1))
    return X, y

def prepare_inputs(X_train, X_test):
    ohe = OneHotEncoder()
    ohe.fit(X_train)
    X_train_enc = ohe.transform(X_train)
    X_test_enc = ohe.transform(X_test)
    return X_train_enc, X_test_enc

def prepare_targets(y_train, y_test):
    ohe = OneHotEncoder()
    ohe.fit(y_train)
    y_train_enc = ohe.transform(y_train)
    y_test_enc = ohe.transform(y_test)
    return y_train_enc, y_test_enc

# load the dataset
X, y = load_dataset('rdf.csv')

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)

# prepare input data
X_train_enc, X_test_enc = prepare_inputs(X_train, X_test)

# prepare output data
y_train_enc, y_test_enc = prepare_targets(y_train, y_test)

# define the  model
model = Sequential()
model.add(Dense(32, input_dim=X_train_enc.shape[1], activation='relu', kernel_initializer='he_normal'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# compile the keras model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# fit the keras model on the dataset
model.fit(X_train_enc, y_train_enc, epochs=100, batch_size=16, verbose=2)

# evaluate the keras model
_, accuracy = model.evaluate(X_test_enc, y_test_enc, verbose=0)

print('Accuracy: %.2f' % (accuracy*100))

But I am getting this error: ValueError: Found unknown categories [...]. Is there any efficient way to solve this usecase?


texts: Xgboost (classification problem) feature importance per input not for the model

texts:
I have trained a xgboost model for a classification problem. I'm able to get the feature importance for the model as below.

http://machinelearningmastery.com/feature-importance-and-feature-selection-with-xgboost-in-python/

But I would like to get feature importance per input. Basically, one input may give me a output probability 0.9, another one input may give me a output probability 0.1. I want to know because which features (with values) give me a probability 0.9? which features (with values) give me a probability 0.1?

How can I approach that? Is there a package for this?


texts: Labels are not given for multiclass classification problem

texts:
I have probably a weird question.
If you are dealing with a multiclass classification problem, do you always have already determined target output/labels?

I have e.g. a huge data set with a lot of features about different city areas (population, population density, number of services, banks and so on). I want to classify objects (houses, buildings) in these city areas based on these features, whether they are near the city center or not, let's say I want to have 3-5 labels at the end. But I don't know myself yet, how should I determine these labels.
Is there a specific approach to solve this? Had anyone a similar problem? Please advise

P.S. Earlier I calculated the distance between some object (e.g. house) and city center point (based on latitude&longitude). And based on distances I generated labels. But this approach is not universal when we have different cities of different sizes.

P.P.S. Do I have to follow probably unsupervised learning methods? Do clustering and find the clusters. Then analyze the clusters to give meanings to those identified clusters. And then solve the problem as a multiclass classification problem?


texts:
I want to do a text classification problem for which I want to train use Adaboost Classifier from sklearn using a Keras estimator. I do know how to use the Keras wrappers for using sklearn functions. So that is not the problem. I have a 3D input data (number_sentences, number_words, features) and when I checked the source code of Adaboost, it shows that X can be only 1D or 2D.

Why is that so?
Is there any other way for me to use Adaboost for 3D    input data?


texts: problems with installing tensorflow using anaconda

texts:
I am trying to use anaconda 4.5.12 to install tensorflow on an Ubuntu 18.04.1 LTS system with python 3.7.1 and pycharm 2018.3.3.

I issue the following terminal command:

conda install tensorflow

And this gives me:

Solving environment: /\-

(It keeps switching between /-). This just continues indefinitely. I waited for like 20 minutes at one point. Why is this happening?

Also, I am confused by the explanation on this anaconda site, which gives the following terminal command:

conda create -n tensorflow_env tensorflow
conda activate tensorflow_env

What does this command do, and why would we use this rather than "conda install tensorflow"? I assume it creates a new environment but why do we need a new environment to install tensorflow? Can't we just install tensorflow for all environments? This seems messy, because I think it also means that I can not create an environment in PyCharm directly.


texts: RNN-based Predictions of Sine Waves with Frequency Different From Training Data

texts:
I am wondering if I can generate a sine wave with a frequency different from training data using RNN. For example,

Using two training data of two time series, say 0[sec] ~ 10[sec] each:
sin(t) and sin(2t)

If we feed these two data to RNN, can we possibly predict a sine wave of sin(1.5t) with some amount of initial data?

I would be very appreciated if you could tell me the possibility of this, and perhaps some examples of implementation.

Thank you in advance


texts: What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?

texts: I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small.I have 12K sentences and only 10% of them are from positive classes. Does anyone here have any experience on finetuning bert in small datasets? Do you have any suggestions for learning rate, batch size, epoch number warmup steps etc.?


texts: Best way to evaluate interlaced recommendation system results while reducing bias

texts:
I already asked this question but I worded it in such a way that it was a completely different question to the one I want to ask. I have not deleted the old question in case someone finds it useful.
Given two recommendation systems, system A and system B. Users will listen to a number of songs from both systems, and rate each song out of 6. I will then group the ratings based on if they are system A or B, and find the average rating to evaluate which system was better.
I would like to know which is the best system of interlacing the recommendations by A and B, such that bias is minimised.
I have currently implemented 2 methods. The first is that each user only ever listens to the recommendations by one system, so the songs are all AAAAA or BBBB. This leads to the issue that if A is objectively better than B, then if the user rated B a 3, for example, if they had also listened to A, the rating for B might have been lower since it is relatively better. The same applies if the user only listens to A, and the average rating is 3, for example. If they had listened to B, which is worse, it might have received a higher rating since people are not completely objective. Thus, this method introduces a bias against the better system.
The second method is that the users listens to the songs with a 50/50 chance of them being from either system. AABBABABBAA etc. This system eliminates the first system's bias issue, however it introduces another one. If the good system is making good recommendations, and the bad system makes a bad recommendation, the user might allow a better rating for it since they were 'put into a good mood' by the good recommendations. The same applies if the bad system keeps on making bad recommendations, the good system's ratings will be weighted down since the bad system will 'put the user into a bad mood'. Thus, this method also introduces a bias against the better system.
Am I wrong in my reasoning, and if not, is there a better way to interlace the results from both systems while minimising bias?
Thanks


texts: Solve for the set of coordinates that reduces the average distances between request and server in half

texts:
I generate a DataFrame with coordinates and distances to 3 servers.
import pandas as pd
import numpy as np
import random
from math import sin, cos, sqrt, atan2, radians

def haversine(lat1, lon1, lat2, lon2):
    R = 6373.0
    lat1 = radians(lat1)
    lon1 = radians(lon1)
    lat2 = radians(lat2)
    lon2 = radians(lon2)

    dlon = lon2 - lon1
    dlat = lat2 - lat1

    a = sin(dlat / 2) ** 2 + cos(lat1) * cos(lat2) * sin(dlon / 2) ** 2
    c = 2 * atan2(sqrt(a), sqrt(1 - a))
    distance = R * c
    return distance

def nearest_server(distances):
    nearest = np.argmax(distances)
    return distances[nearest]

def main():
    df = [(random.uniform(-90.0000, 90.0000), random.uniform(-180.0000, 80.0000)) for i in range(0, 10000)]
    df = pd.DataFrame(df, columns=['lat', 'lon'])

    # Calculate the haversine distances from request to each server
    df['server_1_km'] = df.apply(lambda x: haversine(x['lat'], x['lon'], 26.91, 75.78), axis=1)
    df['server_2_km'] = df.apply(lambda x: haversine(x['lat'], x['lon'], 43.54, -116.56), axis=1)
    df['server_3_km'] = df.apply(lambda x: haversine(x['lat'], x['lon'], 18.51, 73.84), axis=1)

    #determine the nearest distance in kilometers
    df['nearest_distance_km'] = df.apply(lambda x: nearest_server([x['server_1_km'], x['server_2_km'], x['server_3_km']]), axis=1)

if __name__ == '__main__':
    main()

The resulting dataframe:
df.head()
         lat        lon   server_1_km  server_2_km   server_3_km  nearest_distance_km
0  46.669639  39.533947   3853.364128  9715.302979   4425.403201          9715.302979
1  64.845647 -75.829235   9524.431080  3449.905531  10396.715639         10396.715639
2 -12.997567 -85.863194  17561.869774  7018.701526  17765.524191         17765.524191
3  58.701055 -95.549241  10465.032112  2215.936083  11378.201340         11378.201340
4  26.392046 -61.901963  12560.106384  5229.352101  13109.087621         13109.087621

The average nearest_distance_km from request to server:
df['nearest_distance_km'].mean()
12581.12885335224

I need to solve for any set of server coordinates of a "minimum" cardinality that halves that distance (12581/2 == 6290.5).
Looking for interesting solutions/pointers to solving this problem.
Note:
It doesn't matter if the new coordinates are in the middle of the ocean. Although, restricting the solution to land-only coordinates is nice-to-have.


texts: I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?


texts: Common deep learning practices in NLP for text classification

texts:
Are there any articles about best modern techniques in text classification (not only for English texts, but in general)? In particular, I'm interested:

what kind of text preprocessing techniques are usually useful? Can you share with some good open-source text preprocessors?
what kind of neural networks architectures one should try first on a new text classification problem? What kind of architectures should be solid baselines?
what are the "rules of thumb" for choosing the size of the dictionary? maximum number of words to be left (in sequence padding)? word embeddings dimension?

Would be great if you could share articles, or (even better!) the must-read research papers, or anything else you find useful about this topic. If you could share with your own experience in the comments, that would be greatly appreciated, too!


texts: Does it make sense to repeat calculating AUC in logistic regression?

texts:
I have a question regarding logistic regression models and testing its skill.
I am not quite sure if I understand correctly how the ROC Curve is established.
When calculating the ROC curve, is a train test split happening and then the skill of a model based on the training split is tested on the test split? or is a model based on the ENTIRE data just tested on the ENTIRE data?
If the first is the case, would it make sense to do repeated random train test splits and average out the Area under the curve? would that bring about any additional certainty about the model's skill?
Thank you.


texts: Help Interpreting Machine Learning Results

texts:
I am trying to implement a machine learning model for a regression task in bioinformatics. It is a task analogous to scoring images. The goal is to get the predicted label for an input as close to its true label as possible. Ideally, one would feed the model with raw data and have the model learn features during the training process. Unfortunately this is not possible due to the comparatively small dataset size. I have tried the following approaches with both raw sequence data and pre-featurized data.
Model 1: Pre-Featurized Data $\Longrightarrow$ CNN Block $\Longrightarrow$ Dense Network $\Longrightarrow$ Output
Model 2: Pre-Featurized Data $\Longrightarrow$ Dense Network $\Longrightarrow$ Output
Model 3: Raw Sequence Data $\Longrightarrow$ CNN Block $\Longrightarrow$ Dense Network $\Longrightarrow$ Output
Model 4: Raw Sequence Data $\Longrightarrow$ Embedding Layer* $\Longrightarrow$ CNN Block $\Longrightarrow$ Dense Network $\Longrightarrow$ Output
*Description of embedding layer linked.
Models 1, 3, and 4 perform similarly and eventually settle at a validation loss of about 0.6. Model 2, however, has an extremely high validation loss at more than 10. This is very confusing to me.

If raw data is insufficiently featurized, then models 3 and 4 should not have performed well.
If the pre-featurized data was not capturing features that were meaningful to the task, model 1 should not have performed well.

I am at an absolute loss here. I would be very appreciative of anyone who can make sense of these results.


texts:
Could someone please explain how the EasyEnsemble algorithm works? Im using it for a prediction model for imbalanced minority class.
Please don't refer me to this paper, as it makes no sense to me.
EasyEnsemble and Feature Selection for Imbalance Data Sets
Im using the algorithm in Pandas, with the UnbalancedDataset library which is on GitHub  UnbalancedDataset
I get an array of matrices as O/P, I don't know how to use this in the end, to train with random forests.
Thanks


texts: For which real world data sets does DBSCAN surpass K-means.?

texts: For clustering, DBSCAN surpass k-means in terms of handling arbitrary  shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data sets that have special features (for example, the moon-shaped data sets). Can someone point me to some real world datasets where DBSCAN outperforms K-means.


texts: Does keras categorical_cross_entropy loss take incorrect classification into account

texts:
I was looking at keras source here which calculates cross entropy loss using:

output /= tf.reduce_sum(output,
                        reduction_indices=len(output.get_shape()) - 1,
                        keep_dims=True)
# manual computation of crossentropy
epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
output = tf.clip_by_value(output, epsilon, 1. - epsilon)
return - tf.reduce_sum(target * tf.log(output),
                       reduction_indices=len(output.get_shape()) - 1)

target is the truth data, which is 0 or 1, and output is the output of the neural net.

So it looks like the loss is of the form

$$J_{y'} (y) = - \sum_{i} y_{i}' \log (y_i)$$

where $y_i$ is the model output for class $i$, and $y_i'$ is the truth data.

Does this mean the errors for $y_i' = 0$ do not contribute to the loss?  Why isn't the formula

$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

used?


texts:
sometimes I see the kernel with GP programming.
but without explanation, they put some random number with unknown equation
Here is some part of GP programming

gp <-function(data)
{
  p<-0.200000*tanh(((((31.006277) * ((((((data$Embarked) + (data$Cabin))/2.0)) + (((((sin((tanh((data$Parch))))) * (3.0))) - (data$Pclass))))))) * 2.0)) +
    0.200000*tanh(((31.006277) * (((((data$Age) * (data$Cabin))) + (((((0.318310) - (data$Pclass))) + (pmin((2.0), (((data$Parch) * 2.0)))))))))) +
    0.200000*tanh(ifelse(rep(1.,NROW(data))*(ifelse(rep(1.,NROW(data))*(data$SibSp>0.),data$Cabin,sin((data$Parch)))>0.),(7.90205097198486328),(((((((data$Cabin) + (data$Fare))/2.0)) - (9.869604))) - (31.006277)))) +
    0.200000*tanh(((((((((((tanh(((((0.636620) < (data$Parch))*1.)))) * 2.0)) - (data$Pclass))) + (((data$Embarked) + (sin((data$Pclass))))))) * 2.0)) * 2.0)) +

full code is here https://www.kaggle.com/scirpus/my-first-gp-in-r

can someone explain what is this GP mean and
how this number and equation come up with?

My guess is same as Neural network right?


texts: openai gym - what is an agent I can use with a multi-discrete action space?

texts:
I have a custom environment with a multi-discrete action space.

The action and observation spaces are as follows:

Action:

MultiDiscrete([  3 121 121 121   3 121 121 121   3 121 121 121   3 121 121 121   3 121
 121 121   3 121 121 121   3 121 121 121   3 121 121 121   3 121 121 121
   3 121 121 121   3 121 121 121   3 121 121 121   3 121 121 121   3 121
 121 121   3 121 121 121   3 121 121 121   3 121 121 121])

Observation:

MultiDiscrete([100   3   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121   2 121
 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121
 121 121 121 121 121 121 121 121 121 121 121 121 121 121 121])

I am having an extremely tough time finding an agent (for example in keras-rl) that is capable of handling these spaces.

This issue: https://github.com/keras-rl/keras-rl/issues/224 indicates that the keras-rl DDPG agent is capable of handling a multi-discrete action space, but the model has a float output that I cannot use as an action for the step() function, which expects an integer output!

Most other agents seem to use a tanh activation layer, or some layer that produces a binary output. I need an output in the same shape as my action space.

How can this be handled?


texts:
Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good performance. Of course, big data always carry associated terms, like scalability and efficiency, but what exactly defines a problem as a big data problem?

Does the computation have to be related to some set of specific purposes, like data mining/information retrieval, or could an algorithm for general graph problems be labeled big data if the dataset was big enough? Also, how big is big enough (if this is possible to define)?


texts: Training and cross validation error curves

texts:
I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve.

It is observed that the accuracy of training dataset decreases but the accuracy of validation dataset increases. I am not able to justify this behavior.

Normally, as training dataset increases, the training accuracy is supposed to increase right?
Also, assuming that the dataset is very noisy and hence training accuracy is decreasing as dataset size increases. But this doesn't explain why validation accuracy increases because noise is supposed to affect that too.


texts: How should I configure my Neural Network to accept a column of numpy.ndarrays as input?

texts:
I have a dataframe with 10k rows and the following columns:
      array                                     target
[1,5,6,1,3,etc...]                                5
[3,3,1,0,5,etc...]                                10
[0,0,1,1,7,etc...]                                3
        .                                         .
        .                                         .
        .                                         .

Each array has 33222 elements, so I have 10,000 rows each with a 33,222 long numpy.array that I want to input into a Neural Network to predict the target variable.
Here is how the NN is configured:
x = df.loc[:, 'array']
y = df.loc[:, 'target']

model = Sequential()
model.add(Dense(12, activation='linear'))
model.add(Dense(1, activation='linear'))

model.compile(loss='mse',
                optimizer='adam',
                metrics=['accuracy', 'mse', 'mae'])

model.fit(x, y, epochs=10, batch_size=1, verbose=1)

I'm getting
"ValueError: Please provide as model inputs either a single array or a list of arrays"

I haven't attempted to use arrays as input for a NN before, so I would also appreciate any advice on the optimal layer choice and configuration for this kind of problem.


texts: When to use k-medoids over k-means and vice versa?

texts:
I had someone ask me about k-medoids at work and don't know about the performance of this algorithm over other clustering algorithms (namely k-means as it is most similar to it). In this case, it was recommended for use on taxonomic data (ie bacterial/viral species/strains), but I do not know why this is better.

time complexity of k medoids is  $O(k * (n-k)^2)$.

is time complexity of a comparable k-means algorithm the same?
When would you use one or the other?
What qualities does one look for to use k-medoids?
What are the differences in the output?


texts: Neural Network - Sparsity of collaborative based filtering and modelling the prediction problem

texts:
I'm fairly new to machine learning and for that matter, neural networks, but for the past couple of days I decided to take a stab at a fairly classical and practical problem of neural networks/machine learning which is recommendation systems.

Apologies if this is an unnecessarily broad question, but I found it hard to read up on resources answering this particular question. My main question is, how do you even model the problem (or what directions/advice is there on how to model it)?

Let $M$ denote the set of all movies you can recommend, each of which have an associated $id$ to them. What exactly would the input to the model be (and the output) and how would you segregate the training data and the observed results? For example, if I have a row of training data (row of size $|M|$ and each entry is a number between $1$ and $10$ that denotes the user's rating, or $0$ if the user has not seen it yet), would I just remove, say half of the user's ratings that are between $1$ and $10$ (so are meaningful) and replace them with $0$ in the training row and then in my observed row (for testing) it would be the entire row with no missing ratings like illustrated below?

$$\underbrace{[9 \ 0 \ 0 \ 5 \ 7 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Information} \rightarrow \underbrace{[0 \ 0 \ 0 \ 0 \ 0 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Training} \rightarrow \underbrace{[9 \ 0 \ 0 \ 5 \ 7 \ 0 \ ... \ 8 \ 0 \ 10 \ 0]}_{Observed} $$

A few naive ways that are obvious are:

Input size = $|M|$, output size = $|M|$

Each input neuron is given the associated user rating for the $i^{th}$ movie. Assume some number of intermediate layers and then we get to the output size, which is $|M|$ output neurons each of which attempt to predict the predicted rating of the user for that movie. This seems weird, because basically almost all of the input neurons will be given $0$ and almost all of the output neurons will try to output $0$ (since it's likely a user has only seen an insignificant portion of all $|M|$ movies). This doesn't seem like the neural network will learn anything useful. And what do I do for movies that the user hasn't seen? Do I just consider this a $0$ loss? So the loss function for neuron $i$ is just:
$$\text{Loss}(y^{(i)}) = 0 \ \text{if the user hasn't seen ith movie, otherwise: } \ \phi{(y^{(i)}, t^{(i)})}$$

Where $\phi$ is some loss function of the neuron predicted output $y^{(i)}$ and the observed value (the actual user rating) $t^{(i)}$. I find it difficult to believe that this will yield any useful ratings though..since the loss has no effect on the nonsensical outputs of my neural network (it doesn't affect the $i^th$ neurons' values if the user hasn't seen it yet). This also poses the problem of how a prediction will be made then. Do I just take all the highest predicted values from the neural network and return the respective movies?

Because of the above problems, I tried to express the model differently, but I don't think there is anyway to overcome the sparsity of the data and also the fact that the output size must be $|M|$ as I can't have it try to predict discrete integer movie ids. Could someone please enlighten me on this topic or provide some possible insight to obvious mistakes I am making?


texts: Recommending college degrees based on high school subjects

texts:
I have a list of our college students' high school courses.

I want to recommend college degrees to current high school students based on their courses - that is, predict a class based on a vector. For instance someone who studied algebra, calculus and statistics could be recommended software engineering, accountancy or mathematics.

There seem to be a few approaches I could take: market basket analysis, collaborative filtering, clustering or even neural networks.

I can structure my data in a sparse matrix of courses, with each row having a class representing the student's eventual degree, e.g.:

DEGREE       English    Calc    Algebra    Geography    History ... etc
Soft.Eng.    0          1       1          0            0
Comms        1          0       0          1            1
Mech.Eng.    1          1       1          0            0

How should I approach this?


texts:
I am learning about social media analysis. I am aware that we can extract the data from twitter using hashtags and API. Ex; If I use #covid19, I will get all tweets that contain this hashtag for the duration that I specify.
Similarly, let's say I visit a public Facebook page https://www.facebook.com/A123
Will I be able to extract only the posts which contain the term covid in their text (from all of their posts) using my API credentials?
Is it possible? I don't like to extract all the posts from the page A123. However, I only wish to extract only the posts which contain the term covid in their text (from all of their posts)
Can you help me on how can it be done?
Any tutorial/tools available to extract this info?


texts:
I have a trained KNN, I created with
https://github.com/kevinzakka/blog-code/blob/master/knn/knn.py

I want to add more data points to the KNN but I am on a raspberry pi so limited by RAM and therefore the number of data points I can add at a time to the model.

I have 20k images, I can add 2k at a time. Because of the way scikitlearn works, I cant use the model trained on a Windows PC on the pi (different architectures).

Can I create a model, and then add more data points to it somehow?


texts: Time Series data: How to convert it in a streaming data?

texts:
I have a time series data which is available in offline csv format. I am using this data to create anomaly detection model. Although I could create this model to predict anomalies in this dataset, I need to use this model when data is real time. Every second data point will be coming and then if point is anomalous I need model to raise an anomaly.

Any ideas how should I implement this?

Edit -
Data is updated regularly in computer, file is updated as data is streaming in this file only. Predictions are required for the realtime anomaly detection as decision will have to be made to stop the process when such prediction for detection of anomaly is made.


texts: Trade-off between variable stability and discriminatory model quality

texts:
It is very normal for data scientists and modeling professionals to be concerned with the stability of the model. It basically means that if a variable is important today, it cannot change its importance over time because this instability would be a demonstration of the model's weakness.
It is difficult to argue that stability is good. But in the area of credit modeling (which produces credit scores, for example) people are cracked by stability, to the point of preferring models with less discrimination to a more stable one.
My question is what is the true utility of stability and what is the correct way to make this trade-off with the discriminatory metric of the model (KS, Gini, AUC, etc.).
I would guess (personal guess) that stability somehow ties the model to a less than ideal metric because it does not allow behaviors to change over time. If they don't really change, there will be no problems, but if the effects of the variables are changing over time, it is normal to assume that this is reflected in less stability and analysts work around this by placing simpler models or removing the variables.


texts:
I need to scrape a table off of a webpage and put it into a pandas data frame. But I am not being able to do it. Let me first give you a hint of how the table is encoded into html document.

United States Total**
15,069.0
14,575.0
100.0

Arizona

Pinal Energy, LLC
Maricopa, AZ
50.0
50.0
NA
2012-07-01
2014-03

Arizona Total
50.0
50.0
NA

The body of the table begins with
....
. Each
....
 is a row of the table.Within each row, that is within each pair of
....
, each column is given by 50.0	.

Here are my questions:

1) How do I scrape it ? I am using BeautifulSoup and requests for this purpose as well as pandas module. I tried the following:

r = requests.get(url)
bs = BeautifulSoup(r.text)
info = bs.findALL('tr','td')
  ....
  ....

But it is giving me this error:

TypeError                                 Traceback (most recent call last)
<ipython-input-24-32d9483e2c59> in <module>()
      1 bs = BeautifulSoup(r.text)
----> 2 info = bs.findALL('tr','td')
      3 #print bs

TypeError: 'NoneType' object is not callable

2) I need to skip some of the rows based on the text in it. For example I don't want to read in the row in which the word 'Total' appears (as inUnited States Total**	). How do I do that ? Although, it is not extremely important as I can get rid off it later, but skipping these rows while reading the data is ideally what I need.

I know it is a long post, but if someone can help me with it, i would greatly appreciate it. Please let me know if more information is needed.

Thanks much.


texts: Machine learning model (neural network or SVM) for unequal feature matrices size

texts:
I have feature matrices obtained from visual bags of words model for various dictionary sizes. Example, Nx5, Nx10, …., Nx15000. Where N is the number of samples and 5, 10, …15000 are the visual vocabulary or dictionary size or feature vectors size. There are classification labels for each samples. There are 13 methods (SIFT, SURF, BRISK and others) to extract the feature descriptors which are eventually used in visual bags of words to encode the images to obtain the feature vector. Also, there are 13 different dictionary sizes 5, 10 to 15000.
If I had only one method and one visual vocabulary, I could have created one multi-class classifier. But, here I have 13 methods and for each of them, there are 13 visual vocabulary sizes. Thus, making 169 classifiers if I want to try to train machine learning models individually. This makes things cumbersome to present.
So I would like to ask you if there is some method where I can combine these unequal Nx5, Nx10, …., Nx15000 feature matrices to have one classifier model for each 13 different feature descriptors, instead of having 169 classifiers.
Please let me know if you have any suggestions.


texts: What is the point of getting rid of overfitting?

texts: I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too sparse then your model may start to memorize data and not perform well on new data. However, are there any cases in which adding dropout, regularization, etc would increase accuracy on the validation set? For instance, if my training acc is 95% and val accuracy is 70%, would removing overfitting simply bring the training accuracy down lower to the val accuracy? Or is there a way to actually improve training accuracy? I assume there is but some intuition on this would be very much appreciated!


texts:
Any idea to what $\bar{X}$ refers to in association rules? Is it "not-$X$"?

It's in the context of interestingness measure for rule $X \rightarrow Y$.


texts: Correlation matrix with a dataset with many missing value

texts:
I am a data-science rookie and I would like to use Python/ R to create a correlation matrix (something like this: http://www.marketcalls.in/python/quick-start-guide-compute-correlation-matrix-using-nsepy-pandas-python.html) and build a machine learning model. However, I have some questions and would really appreciate some guidance.

Question 1: Although the data file is pretty big and have more than 350,000 entries, some columns missed many values (i.e., 60%/ 70% of the values are missing). I am wondering should I abandon those columns/ delete those rows/ any other great recommendations? And what is a good threshold, is it okay to proceed with columns that 20%? 30%? 40% values are missing.

Thank you very much. Greatly appreciated your help!!


texts:
I have a set of data composed of time series (8 points) with about 40 dimensions (so each time series is 8 by 40). The corresponding ouput (the possible outcomes for the categories ) is eitheir 0 or 1.

What would be the best approach to design a classifier for time series with multiple dimensions ?

My initial strategy was to extract features from those time series : mean, std, maximum variation for each dimension. I obtained a dataset which I used to train a RandomTreeForest. Being aware of the total naivety of this, and after obtaining poor results, I am now looking for a more improved model.

My leads are the following :
classify the series for each dimension (using KNN algorithm and DWT), reduce the dimensionality with PCA and use a final classifier along the multidimensions categories.
Being relatively new to ML, I don't know if I am totally wrong.


texts: Association between continuous input and categorical output

texts:
I have two independent continuous variables like Age, Price and an outcome variable like purchased or not purchased
I know if we have continuous variables on input and output, we can use pearson correlation or chi-square test if both input and output are categorical.
Now, which test should I use to test the association of continuous input variable with categorical outcome ?
So, can someone please help me?


texts:
I want to take the derivate of the jacobian using pytorch, but it seems like I am doing the wrong thing. Here is part of my code:

x_1 = np.arange(1,4,1)
x_1 = torch.from_numpy(x_1).reshape(len(x_1),1)
x_1 = x_1.float()
x_2 = np.arange(1,4,1)
x_2 = torch.from_numpy(x_2).reshape(len(x_2),1)
x_2 = x_2.float()
x = torch.cat((x_1,x_2),1)

x.requires_grad = True

w1 = nn.Linear(2, 2, bias=False)

y = w1(x)

def jacobian(inputs, outputs):
    return torch.stack([torch.autograd.grad([outputs[:, i].sum()], [inputs], create_graph=True)[0]
                        for i in range(outputs.size(1))], dim=-1)
jac = jacobian(x,y)

So at this point jac contains the jacobian matrix of my system, which works just fine. But I want to take the derivative of jac with respect to x (basically second order partial derivatives). But when I do this: d_jac = jacobian(x,jac) I get this error: RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior. I am not sure why, as obviously jac is derived from y, so it should still depend on x. Can someone help me? Thank you!


texts: How do I feed three-dimensional input layer into a Neural Network?

texts:
I have training data where each example is a 12x3 two-dimensional array. Let's say I have 1000 of these training examples so my input matrix is a three-dimensional 1000x12x3 matrix. I am trying to feed this into a Neural Network (similar to the one outlined in the diagram below). However the first layer of the neural net is expecting the input matrix to be two dimensional since the hidden layer with which it will be multiplied is also two-dimensional.

How do I resolve this? Should I force each input example to be single dimensional, by flattening it out into a 1x36 array? Or should the hidden layers themselves be three-dimensional in order to match the input?


texts:
I have a random vector with known pdf x and i'm trying to predict some behaviour y. I created a dataset containing (X,y), where X is a vector of D realizations of the VA X and y = (y1,y2). That is, for each instance I have a vector D of input and the vector y for output. I tried:

 from tensorflow import keras
 model = keras.models.Sequential([
 keras.layers.Dense(300, input_shape=(D,), activation="relu"),
 keras.layers.Dense(300, input_shape=(D,), activation="relu"),
 keras.layers.Dense(100, input_shape=(D,), activation="relu"),
 #keras.layers.Dense(10, activation="relu")
 keras.layers.Dense(2)
 ])

 model.summary()
 model.compile(loss='mean_squared_error', optimizer="adam")
 history = model.fit(X_train, y_train, epochs=30, validation_data=     (X_val, y_val))

And in training:

 Epoch 2/252 \\
 252/252 [==============================] - 0s 965us/sample - loss: 67222078.7937 - val_loss: 99721252.0000
 Epoch 2/252
 252/252 [==============================] - 0s 325us/sample - loss: 66736292.6984 - val_loss: 99545232.0000
 Epoch 3/252
 252/252 [==============================] - 0s 246us/sample - loss: 64592091.8095 - val_loss: 99156788.0000
 ...
 Epoch 252/252
 252/252 [==============================] - 0s 305us/sample - loss: 23.6345 - val_loss: 92329540.0000
 80/80 [==============================] - 0s 143us/sample - loss: 29423635.9000

Loss and val_loss is too bad. Can anyone give me an insight?


texts: Is logistic regression actually a regression algorithm?

texts:
The usual definition of regression (as far as I am aware) is predicting a continuous output variable from a given set of input variables.

Logistic regression is a binary classification algorithm, so it produces a categorical output.

Is it really a regression algorithm? If so, why?


texts:
I have a text classification task that consists of classifying text into classes (literary genres).

I have computed the average word length and sentence length. Also, some POS relative frequency so that I can use these as features to train my model. I was thinking of using Decision Trees or another rule-based classifier. This is the first time that I created my own features for classification. Before, I relied on n-grams, vectorization, and BoW.

My question is: How can I use shallow features in text classification using python? I would like to know how the training set should look (in terms of features tabs in a csv file). Also, how can I instruct python to pick these features?

This a bit of a procedure question. Thanks.


texts: Possibility of working on KDDCup data in local system

texts:
I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R
http://www.kddcup2012.org/c/kddcup2012-track2

It seems not possible to work with this 10GB training data on my local system with 4GB RAM.
Can anyone work on this data using this kind of a local system ? Or is using a cluster the norm ?
It would be great if anyone could provide me with any guidance on how to get started with working on a cluster and the normally used type of cluster for such tasks


texts: How to build a classification pipeline that will pass to another model?

texts: Not sure if the title explained it, but I am trying to build a pipeline where it's like a decision tree, but also not. Say for example, I had a picture. The model classified the picture, but now I want to send it to another CNN. How could I create something like this? Sorry if it's not detailed, still figuring it out!


texts: Machine learning anomaly detection for continues data

texts:
I need to find anomalies in logs created by network monitor tool Bro. Right now I am referring to the example given by Bro Analyze Tool. But the example only uses recent logs.

So I need to make it work with full log meaning to train on past logs and perform anomaly detection in real time on each new log. In the example, as far as I understand, unsupervised machine learning with IsolationForest and KMeans in scikit-learn python are used.

So I have the following question.

For unsupervised machine learning is it possible to store the model with trained data which is already created on previous run?

If possible can I add new data with existing model and then perform anomaly detection in real time?


texts:
I have past data of a large number of people who applied for a loan and their movement through 8 different stages, from start of application to loan being paid out.

I am trying to build a model that would predict how many people will get their loan completed (paid out) in the next 5 days (a working week).
I'm using LightGBM and it would learn based on past data, and then use the model on people currently at various different stages and predict how many of them will likely be completed in the next 5 days.
Those 8 stages can only happen one after another like a pipeline, so stages nearer to the last stage 'Completion' have a higher probability of being complete.

The model gives a good accuracy, but a poor precision score, there are too many false positives.

I primarily used the Mean Squared Error, any recommendations for another evaluation metric?
Since the 8 stages follow a pipeline in which they can only go one after another, can I somehow define them differently than I did?


texts: Regression vs Classification to rank by profit

texts:
Say that I have a population of 10k customers, for which I expect 100 responders in my next campaign and have budget to send a letter to 5k.

I have past data of how much those who respond spent, and I'm interested in getting the most $ out of my 5k letters (i.e. ideally I find the 100 responders, but if not, I want to find those who leave more money).

I have created a regression model and a classification model (1 if they spent money -i.e. reponded in the past - and 0 otherwise). After running bot models, I find that in a test sample of 10k, if I select the first 5k ranked by probability (for the classification model) and by the expected amount in the regression, the regression model performs better (not evaluating how many responders they found, but how much they found out of those selected).

Does it make sense based on the numbers of my context? I had expected the classification model to perform better, but can't figure out why the regression model is coming on top in my example.

Thank you very much!!


texts: What is Mixture of Experts and how it works?

texts: I've seen this subject in Haykin's "Neural Networks" book (2nd ed.) but it was removed in 3rd edition. His explanation is not very intuitive and i didn't find "friendly materials" about this model. Any recommendation of material will be appreciated.


texts: Calculate regression coefficients for individuals (low sample size regression)?

texts:
Is there a way to calculate the regression coefficients for individuals instead of just a group resp. calculating regression coefficients for a very small sample size?

Background

My goal is to test different ways to cluster/segment a group of people. In the domain in question the best variables to use for segmentation would be variables that best express the importance an individual places on certain attributes.

However I do not have direct measurement of that importance. What I do have is an indirect measurement for the whole sample based on a shapley value regression of attribute ratings and final respondent choice.

Clearly put, I asked a couple of people how they would rate the attributes of a given brand and which brand they would choose in the end.

Now I wonder whether it is possible to obtain this indirect importance measurement also on an individual level which would mean either:

Extracting individual coefficient estimators from the overall regression model ( I don't believe that is possible)
Calculating a regression for each individual (Don't know how to do that hence the question)

Problem and what I tried

I work in R and I do know how to fit a grouped linear model. As all my respondents rated two brands I do have n = 2 per group.

However fitting a normal grouped linear model results in unusable results where almost all coefficients are NA and one or two equal 1. Additionally I would prefer to fit a relative importance or shapley value regression but the relaimpo package throws an error complaining about too few observations.

What other avenues could I pursue?

Update:

I have been using this code so far:

df %>%
  group_by(i_TAN) %>%
  do(model = lm(formula = Consideration ~ ., data = .))


texts: Predicting global horizontal irradiance using satellite images

texts:
I have the aim to build a model to predict global horizontal irradiance (ghi) using satellite images and other features namely the day of the year and time of the day.

For extracting the satellite images features, I aim at using one of the available network in keras (e.g. Xception). Then I can concatenate the features output from the CNN to the other features (I already made the modifications to keras to handle those requirements.

Now, essentially, ghi is also a time-series problem so I assumed using time-series techniques could help. But I don't know where to look for those architectures (CNN + Time series + other features) so :

given my problem and desires, do you have ressources to help me tackle this issue?
one thing that is not yet clear to me is if I have don't freeze Xception internal layers, are the fiters parameters automatically learned by the model? I read here and there that filters are optimized also but I never found in keras a clear statement about that.


texts: What’s more appropriate for outlier detection? Classification or Regression?

texts:
I'm using PyTorch and i’m working on a data set where I only care about “outliers” from the norm. The data is comprised of long time series with gaussian distribution, while once in a while there is a large outlier. I would like to classify this into: “normal”, “large positive outlier” and “large negative outlier”.

As far as I recognize this problem that can be solved either:

(1) via regression, using MSELoss, and classifying the prediction “manually” at the end into one of the three classes. The target in this case would be the outlier value.

(2) via classification in 3 classes, using CrossEntropyLoss. The target in this case would be the class number directly.

What do you think is more appropriate here? Which one is faster?
Both seem to converge to just predicting “normal”, because that’s correct in 99% of the cases. Which model should I use and how can I force the model to make predictions on outlier cases as well?


texts: Automated bug finder through machine learning?

texts:
Assume that I have a large software application, this application is event driven i.e. when a user configures something by pressing on a button a code flow executes.

Along with this application I also have a system which can emulate a user and can perform all permutation of steps which a user might perform. Hence it is safe to say that I can bring my software application to all the possible states. I have already done the automation for this, I just need a brain which can say that the current state is buggy (undesired) or not.

Now the question is:

  Is it possible to determine if my system has gone into an unexpected state (Bug) automatically via machine learning?

At first I thought of using Rete Net to solve this. Idea was to make a knowledge base which will emulate a tester and can tell if the system is in an inconsistent state or not, but for this I need to constantly feed the knowledge base with new rules whenever a new feature is added to this software application.


texts:
I have a dataset of posts for sentiment analysis that are labelled with -1 (negative), 1 (positive) or 0 (neutral).
So I wonder how should I deal with that. These are my ideas:

make a multiclass classifier : I tried with a random forest, and the results are pretty correct; however, I have a certain amount of negatives in positives and vice versa; I would've preferred the error to rather be in neutral.
make a binary classifier, but when predicting, if the probabilities are too balanced, return neutral. However, it seems to me that I don't use the neutral data - isn't it a waste of data? Maybe using a OneVsAll will be better?
make a perceptron/neural network with an tanh neuron at the output; but I don't know what good loss function could be used here.

Do you know if there is any of those that is theoretically/practically better?


texts:
I have two questions.

1- I have weather data of 10 turbines and I know their collective production(Power).I also know maximum power a turbine can make. How can I forecast collective production if I know individual turbine weather condition?

Power Data

    Time                       Power Produced(MW)
    2010-11-11 00:00:00+00:00     248.367512

Weather Data

time    Temperature     RelativeHumidity    Wind_U  Wind_V    Pressure  CloudCover  WindGustSpeed   latitude    longitude
2010-11-11  80.060944    0.959463         4.791473  3.756961    99201.50    0.999969    9.061006    55.003349   9.983611

2- Moreover, I have more than one value for one hour weather like 2010-11-11 (100F 30.3m/s) ,and there is another set of values for same date like 2010-11-11 (99.8F 30.1m/s). How can I use those multiple values for single date?


texts: conceptual question about triplet loss embeddings

texts:
A question on the paper Schroff, Kalenichenko and Philbin "FaceNet: A Unified Embedding for Face Recognition and Clustering", 2015.

It is my understanding that the input to the CNN is a triplet (hence the name) of three pictures or thumbnails - one picture is the anchor, another one is a picture of the same person as the anchor picture, and the third one is a picture of a different person as the anchor.

The output is one 128 to 256 embedding for the anchor picture, so that embeddings of pictures of the same person are closer to such output as are embeddings of pictures of different people.

So the input for training, as I understood, is three pictures and careful selection of the embeddings is crucial.

But what about testing? I thought the goal is to get the embedding of any one picture - or are three pictures (anchor, same person, different person) also necessary for testing?


texts: Assistance needed on what machine learning approach to use

texts:
I'm currently writing my Master's Thesis on Subjective tagging of sounds and I feel that I've been stuck with the same problem for quite a time now and need assistance to progress. I'll, in short, describe my goal and what I've accomplished so far.

I'm writing a program that lets users manually tag audio files with a single label according to their own, i.e. subjective, perception. The tags are always adjectives, like happy, spooky or spacy. The goal is to have the program automatically tag similar sounds with the correct label by finding related sounds using machine learning. The program currently looks like this:

Usage of the program goes like this:

The user adds sounds to be labeled.
The program analyses the files and searches for related sounds using Algorithm 1.
The user tags a sound and the program automatically tags related sounds using Algorithm 2.
The user can verify tags given by the program and choose to accept them or give the sound a new tag.
Repeat from step 1 or step 3.

By looking at the program flow above I've been able to implement step 1 using AudioCommons Timbral Models to convert sounds to a usable data format, and step 2 with Algorithm 1 using Mean shift to find groups of related sounds. The idea is to use the groups found by Algorithm 1 as a starting point when no tag has yet been given to a group. When one of the sounds in the group is tagged, all the sounds receive the same tag.

My problem, however, lies in step 3 with Algorithm 2 and step 4. I've tried a few different approaches, but they all seem a bit off and not quite fit for the job.

My goal is to have the program continuously or recursively learn from user validation, hence improving over-usage. My thought process is to have a verified flag on each sound, which the user can use to say if a tag given to a sound by the program is correct or not. Algorithm 2 should (re)learn from verified sounds in step 4 so that when step 3 is repeated, the automatic tagging of sounds done by the program is more accurate according to the user's choices.

So my question is: What kind of algorithm(s) is suitable for step 3 and 4 in place of Algorithm 2?

I've tried to make myself as clear as possible, but I'm quite new to machine learning, so feel free to point out confusing sections or errors.

In hope of assistance,

Andreas

Notes:

I've used tag and label interchangeably throughout the text; they refer to the same thing.
I want all machine learning to occur in isolation for a user, i.e. I don't want to learn from other user's input (like Collaborative filtering).


texts: ML/NN as Function Evaluator for further Optimization (maximization) - Practical Example

texts:
I am working on a production optimization problem; a very similar idea to what is described by Vegard Flovik How to use machine learning for production optimization. The following image, taken from the referred post, summarizes it very well:

First step is obvious, and I do have a model in the form of machine learning or neural networks model. How would I go about the second step? How can I use the trained model as the function evaluator for further multi-dimensional nonlinear optimization (e.g. maximization) either via Scipy, Bayesian Optimization etc.?
I cannot seem to find a practical example. Having a closed-form analytical function as the objective of an optimization problem is well-established. The article Optimization with SciPy and application ideas to machine learning by Tirthajyoti Sarkar gives a few examples using Scipy, & introduces packages that do  optimizations with bound constrains and more. Yet examples are quite simple (a closed-form mathematical function) and he only glosses over the extension of such idea to use NN as the objective function, I'm quoting:

You are free to choose an analytical function, a deep learning network
(perhaps as a regression model), or even a complicated simulation
model, and throw them all together into the pit of optimization.

Any leads/hints/links are appreciated!

[Appendix]
In order to have a concrete example, let's imagine we have a dummy data set with a set of feature and a imaginary ProductionYield that is a nonlinear combination of input variables:
import numpy as np
import pandas as pd

df = pd.DataFrame(columns=['Pressure','Temprerature','Speed','ProductionYield'])

df['Pressure'] = np.random.randint(low= 2, high=10, size=2000)
df['Temprerature'] = np.random.randint(10, 30, size=2000)
df['Speed'] = np.random.weibull(2, size=2000)
df['ProductionYield'] = (df['Pressure'])**2 + df['Temprerature'] * df['Speed'] + 10
df['ProductionYield']= df['ProductionYield'].clip(0, 100)

   Pressure  Temprerature     Speed  ProductionYield
0         7            20  1.810557        95.211139
1         2            29  0.674221        33.552409
2         8            17  0.537533        83.138065
3         3            24  1.945914        65.701938
4         6            23  0.514679        57.837610

1.Predictive Algorithm (a simple Neural Network):
## Train/Test Split
from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(df[['Pressure','Temprerature','Speed']].values, df['ProductionYield'].values, test_size=0.33, random_state=42)

## Build NN Model
import tensorflow as tf
from tensorflow.keras import layers

def build_model():

    # create model
    model = tf.keras.Sequential()
    model.add(layers.Dense(64, input_dim=3, kernel_initializer='normal', activation='relu'))
    model.add(layers.Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(layers.Dense(1, kernel_initializer='normal'))

    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')

    return model

model = build_model()
model.fit(x_train, y_train,
          validation_split=0.2,
          verbose=0, epochs=1000)

2.Otimization [Core of the Problem]:
Problem lies herein, when a ML/NN is trained, I do not get to see (export as I wish) the mathematical form of the function (here in this example NN) and its variables (which should be my feature variables) to do the optimization as we do with closed-form explicit mathematical functions.
[UPDATE 15.01.2021]:
Following Valentin's great answer, I've put pieces together in a practical example showcasing how one can use a ML/NN model as an input function for further optimization (herein via scipy.optimize) using the dummy data set shown in the Appendix. Please see this notebook for more details.


texts: In Reinforcement Learning can I randomly assign next_states from the state space to my agent while creating transition set?

texts:
In Reinforcement Learning, while creating transition samples (state, action, next_state, reward), where:

Agent: The learning agent
Environment: The trainer

The environment gives two feedback to the agent: reward and next state.
Can I as the environment, randomly assign next_states from the total state space to my agent. ?
How can I decide what are the allowed next_state(s) from a given state?


texts: Combining Datasets with Different Features

texts:
I have multiple datasets, with slightly differing features. What tools can I use to make this a homogeneous dataset?

Dataset1:

featureA,featureB,featureC
1,7,3
4,8,4

Dataset2:

featureA,featureC,featureD,featureE
3,4,5,6
9,8,4,6

Homogeneous Dataset

featureA,featureB,featureC,featureD,featureE
1,7,3,,
4,8,4,,
3,,4,5,6
9,,8,4,6